MOVEON project at the Viva Technology fair in Paris

The project MOVEON has been presented by Dr. Alain Pagani (DFKI) and Romain Boisseau (INRIA) at the 6th edition of the Viva Technology fair in Paris on June 17th, 2022.

Viva Technology, or VivaTech, is an annual technology conference, dedicated to innovation and startups, held in Paris, France. This year, France and Germany have been showcasing their future European digital champions on a single stand: the French-German Tech Lab. This Lab, organized by twelve French and German partners, highlights the most promising startups in their ecosystems, as well as the concrete academic and economic cooperation between the two countries.

MOVEON is a common project between DFKI and INRIA, and is aiming to develop a new generation of visual positioning algorithms that will enable geometric reasoning to be carried out on high-level primitives taken from learning.

More information:

Contact: Dr. Alain Pagani

First consortial meeting of INFINITY in Vienna

The EU project INFINITY has concluded its first in-person project meeting in Vienna on April 13th, 2022. This meeting was the occasion for the end user partners to try out the updated VR demos.

INFINITY aims at delivering an integrated solution for data-driven criminal investigations, synthesising the latest innovations in virtual and augmented reality, artificial intelligence and machine learning with big data and visual analytics. The result of the project will be a platform for collaborative work between Law Enforcement Agencies, where teamwork will be facilitated in virtual spaces and supported by AI-based data analytics.

More information:

Contact: Dr. Alain Pagani

HITACHI interviews Prof. Dr. Didier Stricker

Artificial intelligence (AI) is serving an increasingly significant role in many workplaces — one that continues to grow as new developments in AI and surrounding technologies become available. This leads us to think about how such technologies can be and should be used to empower humans. In this article, Tomokazu Murakami from the Research & Development Group of Hitachi, Ltd. sat down with Didier Stricker, Head of the Augmented Reality Research Department at the German Research Center for Artificial Intelligence (DFKI) to discuss the use of AI in the workplace, its benefits and challenges, human-AI collaboration and its impact on workers.

Check out the full interview “Insights from AI/Analytics: Human-AI collaboration in the industrial sector: what does the future hold?”.

René Schuster successfully finishes his PhD
René Schuster and Prof. Dr. Didier Stricker moments after the oral defense.

On March 18th, 2022, René Schuster successfully defended his dissertation entitled “Data-driven and Sparse-to-Dense Concepts in Scene Flow Estimation for Automotive Applications”. The reviewers were Prof. Dr. Didier Stricker (Technical University of Kaiserslautern) and Prof. Dr. Andrés Bruhn (University of Stuttgart). Mr. Schuster received his doctorate from the Department of Computer Science at the Technical University of Kaiserslautern.

In his thesis, Mr. Schuster worked on three-dimensional motion estimation of the dynamic environment of vehicles. The focus was on machine learning methods, and the interpolation of individual estimates into a dense motion field. A particular challenge was the scarcity of annotated data for this problem and use case.

René Schuster received an M. Sc. in computational engineering from Darmstadt University of Technology in 2017. He then moved to DFKI to join the augmented reality group of Prof. Stricker. Much of his research was done in collaborative projects with BMW.

René Schuster at the celebration of his newly earned title.
Our BIONIC Technology demonstrated to Germany’s Chancellor
Markus Miezal explains the core elements of the BIONIC system to Germany`s Chancellor Olaf Scholz

On March 18th, 2022, we had the great privilege to demonstrate the BIONIC system to the German chancelor Olaf Scholz during his visit to DFKI GmbH in Kaiserslautern. Dr. Markus Miezal was wearing the BIONIC system on a workstation and presented the real-time ergonomic assessment.

The system consists of work pants and a shirt, which carry inertial measurement units similar to those in a smartphone. Using sensor fusion methods, the body posture of the worker can be extracted to monitor the health of the worker and allows to give feedback, either direct, if the working posture is hazardous or via statistics gathered throughout a workday. Since privacy is a main topic of this project, the data is processed on a small wearable device, similar to a smartphone, so that the personal data literally stays on the person in the first place. Under premise of the workers approval, the data might be shared with his doctor, so that he can make recommendations on exercises according to this phyiscal stress during work. In a further step, the worker might even donate anonymized data to his employer, which allows him to analyze the general ergonomy of the workstation itself.

The chancellor was interested on how long and when to wear the system. Indeed, the system is not meant for every-day use but can be beneficial during training on a new workstation, where new work processes are learnt in an ergonomically sound way. Meanwhile the project has finished, however, spin-offs, e.g. the sci-track GmbH, will continue to provide the technology to the market.

For more information, please check out the following links:

Official BIONIC website

DFKI BIONIC Project Page

DFKI – BIONIC Video on YouTube (3 minutes, 8 seconds)

DFKI + TUK – Tech Demonstrator (1 minute, 47 seconds)


Presenting the online risk alerting and realtime kinematics and ergonomics capabilities of the BIONIC system in the Smartfactory KL

CVPR 2022: Two papers accepted

We are happy to announce that the Augmented Vision group will present two papers in the upcoming CVPR 2022 Conference from June 19th-23rd in New Orleans, USA. The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) is the premier annual computer vision event internationally. Homepage:

The two accepted papers are:

Summary: ZebraPose sets a new paradigm on model-based 6DoF object pose estimation by using a binary object surface encoding to train a neural network to predict the locations of model vertices in a coarse to fine manner. ZebraPose shows a major improvement over the state-of-the-art on several datasets of the BOP Object Pose Estimation benchmark.


Contact: Yongzhi Su, Dr. Jason Rambach

Summary: We propose a novel Multi-Sphere Image representation called Soft Occlusion MSI (SOMSI) and efficient rendering technique that produces accurate spherical novel-views from a sparse spherical light-field. SOMSI models appearance features in a smaller set (e.g. 3) of occlusion levels instead of larger number (e.g. 64) of MSI spheres. Experiments on both synthetic and real-world spherical light-fields demonstrate that using SOMSI can provide a good balance between accuracy and run-time. SOMSI view synthesis quality is on-par with state-of-the-art models like NeRF, while being 2 orders of magnitude faster.

For more information, please visit the project page at

Contact: Tewodros A Habtegebrial

One of our projects has been awarded with the Nvidia Academic Hardware Grant

We are happy to announce that our project DECODE has been accepted for the Nvidia Academic Hardware Grant. Nvidia will support our research in the field of human motion estimation and semantic reconstruction by donating a Nvidia A100 GPU for data centers. We will use the new hardware to accelerate our experiments for continual learning.

Paper accepted for publication in IEEE PAMI

We are very happy to announce that our paper “HandVoxNet++: 3D Hand Shape and Pose Estimation using Voxel-Based Neural Networks” has been accepted for publication in the renowned journal “IEEE Transactions on Pattern Analysis and Machine Intelligence“.

We introduce a new HandVoxNet++ method for 3D hand shape and pose reconstruction from a single depth map, which establishes an effective inter-link between hand pose and shape estimations using 3D and graph convolutions. The input to our network (i.e., HandVoxNet++) is a 3D voxelized depth map based on the truncated signed distance function (TSDF), and it relies two hand shape representations. The first one is the 3D voxelized grid of hand shape, which does not preserve the mesh topology and which is the most accurate representation. The second representation is the hand surface that preserves the mesh topology. We combine the advantages of both representations by aligning the hand surface to the voxelized hand shape either with a new neural Graph-Convolutions-based Mesh Registration (GCN-MeshReg) or classical segment-wise Non-Rigid Gravitational Approach (NRGA++) which does not rely on training data.  In this journal extension of our previous approach presented at CVPR 2020, we gain 41.09% and 13.7% higher shape alignment accuracy on SynHand5M and HANDS19 datasets, respectively. Our results indicate that the one-to-one mapping between voxelized depth map, voxelized shape and 3D heatmaps of joints is essential for an accurate hand shape and pose recovery.

2 Papers accepted at BMVC 2021 Conference

We are happy to announce that the Augmented Vision group will present 2 papers in the upcoming BMVC 2021 Conference, 22-25 November, 2021:

The British Machine Vision Conference (BMVC) is the British Machine Vision Association (BMVA) annual conference on machine vision, image processing, and pattern recognition. It is one of the major international conferences on computer vision and related areas held in the UK. With increasing popularity and quality, it has established itself as a prestigious event on the vision calendar. Homepage:  

The 2 accepted papers are:

1.  TICaM: A Time-of-flight In-car Cabin Monitoring Dataset
Authors: Jigyasa Singh Katrolia, Ahmed Elsherif, Hartmut Feld, Bruno Mirbach, Jason Raphael Rambach, Didier Stricker

Summary: TICaM is a Time-of-flight In-car Cabin Monitoring dataset for vehicle interior monitoring using a single wide-angle depth camera. The dataset goes beyond currently available in-car cabin datasets in terms of the ambit of labeled classes, recorded scenarios and annotations provided;  all at the same time. The dataset is available here:



Contact: Jason Rambach

2. PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image
Authors: Yaxu Xie, Fangwen Shu, Jason Raphael Rambach, Alain Pagani, Didier Stricker

Summary: Piece-wise 3D planar reconstruction provides holistic scene understanding of man-made environments, especially for indoor scenarios. Different from other existing approaches, we start from enforcing cross-task consistency for our multi-task convolutional neural network, PlaneRecNet, which integrates a single-stage instance segmentation network for piece-wise planar segmentation and a depth decoder to reconstruct the scene from a single RGB image.


Contact: Alain Pagani

Paper accepted at the CSCS 2021!

We are happy to announce that our paper “Multi-scale Iterative Residuals for Fast and Scalable Stereo Matching” has been accepted to the CSCS 2021!

The Computer Science in Cars Symposium (CSCS) is ACM’s flagship event in the field of Car IT. The goal is to bring together scientists, engineers, business representatives, and anyone who shares a passion for solving the myriad of complex problems in vehicle technology and their application to automation, driver and vehicle safety, and driving system safety.

In our work, we place stereo matching in a coarse-to-fine estimation framework to improve runtime and memory requirements while maintaining accuracy. This multiscale framework is tested for two state-of-the-art stereo networks and shows significant improvements in runtime, computational complexity, and memory requirements.

Link to preprint:

Title: Multi-scale Iterative Residuals for Fast and Scalable Stereo Matching

Authors: Kumail Raza, René Schuster, Didier Stricker