The 3rdHand project consortium publicly provides data-sets featuring single-human, human-human, and human-robot collaborative demonstrations of assembly tasks. The data was collected and shared between the partners as a development and benchmarking utility for the developed methods and algorithms.

Individual and Collaborative Assembly of Objects

The data-sets contain tracking of objects and human hands during single-human and collaborative assembly of composite objects and furniture like toolboxes and chairs:

As well as kinaesthetic data during the execution of human-robot joint cooperative motions:

The repository contains tracking data data and software tools for visualization. The repository only contains the tracking data for each demonstration; the corresponding videos are set up at a different location: [[ | video archive]].

For more details on the format and precise content of the data-sets, as well as instructions on how to use the visualization software, please refer to the README files in the appropriate folders of the repository.

Also provided in the public repository, is a set of software utilities which allow to visualize the data-sets directly, and to run some of the models developed during the first year.

Innsbruck Pointing Dataset

Deictic gestures – pointing at things in human-human collaborative tasks – constitute a pervasive, non-verbal way of communication, used e.g. to direct attention towards objects of interest. In a human-robot interactive scenario, in order to delegate tasks from a human to a robot, one of the key requirements is to recognize and estimate the pose of the pointing gesture.

More info and download

Innsbruck Multi-View Hand Gestures (IMHG) Dataset

Hand gestures are one of the natural forms of communication in human-robot interaction scenarios. They can be used to delegate tasks from a human to a robot. To facilitate human-like interaction with robots, a major requirement for advancing in this direction is the availability of a hand gesture dataset for judging the performance of the algorithms.

More info and downloads


Relational Activity Processes

An implementation of Relational Activity Processes and basic Monte-Carlo on top. This uses an unusual and minimalistic implementation of first order logic that represents a KB as a graph and offers efficient methods to compute feasible substitutions. The representation is flexible enough to represent stochastic (decision) rules (as in STRIPS or NDRs), regression trees, aggregate literals, etc.

source code

Temporal Segmentation of Pair-Wise Interaction Phases in Sequential Manipulation Demonstrations

We present a linear-chain Conditional Random Field model to detect the pair-wise interaction phases and extract the geometric constraints that are established in the environment, which represent a high-level task oriented description of the demonstrated manipulation. We test our system on single- and multi-agent demonstrations of assembly tasks, respectively of a wooden toolbox and a plastic chair.

Scientific Publication source code

Server broadcasting Kinect 2 features over the network

C# Application connecting to a Kinect 2 sensor to broadcast the following features available in the manufacturer's SDK: 25-joints skeleton tracking, real-time gesture recognition, face and mood tracking, multilingual text-to-speech and speech recognition via a grammar, and access to raw RGBD images. A Python client is included and works for any OS, including Linux with or without ROS.

source code

Logic-Geometric Programming

Software implementation of Logic-Geometric Programming, where joint motions of multiple agents are optimized to solve cooperative sequential manipulation tasks which require planning both at the symbolic and motion level.

Scientific Publication source code

zum Seitenanfang