A Cross-Task Visuo-Tactile Representation Using Point Clouds

Combining visual and tactile cues has proven effective for object recognition, grasping, and manipulation tasks. However, integrating these modalities is challenging as tactile and visual data convey distinct information and differ structurally. Researchers have addressed this problem by proposing approaches that either do not consider mechanical properties conveyed by tactile sensors or cannot be deployed directly for diverse tasks. In this paper, we propose a cross-task visuo-tactile representation that encodes both the geometrical and mechanical properties of objects in a point cloud (PC) data structure. By physically exploring different areas of a given item, we collect tactile information to estimate the local compliance of the surface, encoding it as the color information of the PC in the probed areas. This color information is extended to the entire object assuming that neighboring points share the same mechanical properties. We apply the proposed PC to 6 real-world objects showing that it can be effectively used to encode their shape along with their information on the local compliance. Further, we show that the augmented PC can be used for different tasks by exploiting this in three robotic tasks-a visuo-tactile object classification, a path following and a reaching in clutter. Corresponding author(s) Email:    [email protected]