You in 3D

Contact person: Onorina Kovalenko


Tel: +49 (0) 631 20575 3607

Real-time Motion capture of multiple persons in community videos

Tracking multiple persons in 3D with high accuracy and temporal stability in real-time with monocular RGB camera is a challenging task which has a lot of practical applications like 3D human character animation, motion analysis in sports, modeling human body movements and many others. The optical human tracking methods often require usage of multi-view video recordings or depth cameras. Systems which work with monocular RGB cameras are mostly not in real-time, track single person and require additional data like initial human pose to be given. All this implies a lot of practical limitations and is one of the major reasons why optical motion capture systems have not yet seen more widespread use in commercial products. The DFKI research department Augmented Vision presents a novel fully automatic multi-person motion tracking system. The presented system works in real-time with monocular RGB video and tracks multiple people in 3D. It does not require any manual work or a specific human pose to start the tracking process. The system automatically estimates a personalized 3D skeleton and an initial 3D location of each person. The system is tested for tracking multiple persons in outdoor scenes, community videos and low quality videos captured with mobile-phone cameras.


Contact person: Jason Raphael Rambach
Funding programm: German BMBF
Begin: 01.07.2016
End: 30.06.2019

Comprehensible, interactive experiments: practice and theory in the MINT study

Be-greifenThe project is funded by the Federal Ministry of Education and Research (BMBF). Combine tangible, manipulatable objects (“tangibles”) with advanced technologies (“Augmented Reality”) to develop new, intuitive user interfaces. Through interactive experiments, it will be possible to actively support the learning process during the MINT study and to provide the learner with more theoretical information about physics.

In the project interfaces of Smartphones, Smartwatches or Smartglasses are used. For example, a data gadget that allows you to view content through a combination of subtle head movements, eyebrows, and voice commands, and view them on a display attached above the eye. Through this casual information processing, the students are not distracted in the execution of the experiment and can access the objects and manipulate them.

A research project developed as a preliminary study demonstrates the developments. For this purpose, scientists at the DFKI and at the Technical University Kaiserslautern have developed an app that supports students and students in the determination of the relationship between the fill level of a glass and the height of the sound. The gPhysics application captures the amount of water, measures the sound frequency and transfers the results into a diagram. The app can be operated only by gestures of the head and without manual interaction. In gPhysics, the water quantity is recorded with a camera and the value determined is corrected by means of head gestures or voice commands, if required. The microphone of the Google Glass measures the sound frequency. Both information is displayed in a graph that is continuously updated on the display of Google Glass. In this way, the learners can follow the frequency curve in relation to the water level directly when filling the glass. Since the generation of the curve is comparatively fast, the learners have the opportunity to test different hypotheses directly during the interaction process by varying various parameters of the experiment.

In the project, further experiments on the physical basis of mechanics and thermodynamics are constructed. In addition, the consortium develops technologies that enable learners to discuss video and sensor recordings as well as analyze their experiments in a cloud and to exchange ideas with fellow students or to compare results.

The DFKI is a co-ordinator of five other partners in research and practice: the Technical University of Kaiserslautern, studio klv GmbH & Co. KG Berlin, University of Stuttgart, Con Partners GmbH from Bremen and Embedded Systems Academy GmbH from Barsinghausen.


Contact person: Jason Raphael Rambach
Funding by: BMBF
Grant agreement no.: 16KIA0243K
Begin: 01.02.2015
End: 31.01.2018

Professional Wireless Industrial LAN

Due to the rising requirements of the industry for a flexible and cost-efficient production, secure and robust wireless solutions are steadily gaining much interest. The BMBF project “Professional Wireless Industrial LAN – proWiLAN” comprises a number of experts from a consortium of eight German organizations developing the next generation of wireless radio technology, which will meet the rapidly growing requirements of the future industrial applications.

The aim of the project is to improve robustness, bandwidth and latency of wireless solutions so that even sophisticated or safety-critical applications such as augmented reality or radio-based emergency stop button can be efficiently and user-friendly supported. The common wireless technologies allow stable execution of cooperative Augmented Reality applications only to a limited extent. Besides, in inaccessible environments where assembly and maintenance work should be performed, present-day wireless technologies cannot satisfy the growing requirements.

Necessary and planned innovations include, among others, a multi-band-capable radio interface, which is not sensitive to interferences in each single band and thus is always immediately available. This makes very fast response times of applications possible. It is important to be able to ensure short response times of the systems in order to e.g. achieve the guaranteed shut down time of a machine in case of an emergency stop. Another key innovation of proWiLAN is the integration of a powerful 60-GHz module, which brings a significant increase of transmission data rates. Furthermore, a localization method for industrial environments should be integrated, so that mobile devices should be capable of determining their location and orientation in space. In order to get a high customer acceptance, novel Plug & Trust process developed in proWiLAN, which allows a quick and easy commissioning, retrofitting and security, is of key importance.

proWiLAN is funded by the research program ICT 2020 — Research for Innovations by the Federal Ministry of Education and Research (BMBF) with a total of 4.6 million euro. The project started in February 2015 and runs until the beginning of 2018. In addition to the DFKI as project coordinator, the consortium includes ABB AG, IHP – Leibniz Institute for Innovative Microelectronics, IMST GmbH, NXP Semiconductors Germany GmbH, Bosch Rexroth AG, Robert Bosch GmbH and the Technical University Dresden.


Contact person: Dr. Gerd Reis
Funding by: BMBF
Grant agreement no.: 03VP00293
Funding programm: VIP+
Begin: 01.10.2016
End: 30.09.2019

© S. Siegesmund

The virgin stone marble has been used as preferred material for representative buildings and sculptures. Yet, due to its chemical composition and its porosity marble is prone to natural deterioration in outdoor environments, with an accelerating rate since the beginning of industrialization, mainly due to increasing pollution. A basic requirement for a successful restoration and conservation is a regularly repeated assessment of the current object condition and knowledge about prior restoration actions. Ideally the assessment is non-destructive. This requirement is fulfilled for both the optical digitization of objects shape and appearance, and the ultrasound examination used to acquire properties with respect to material quality.

Goal of the joint research project Marmorbild of the University Kaiserslautern, the Fraunhofer Institute (IBMT), and the Georg-August-University Göttingen is the validation of modern ultrasound technologies and digital reconstruction methods with respect to non-destructive testing of facades, constructions and sculptures built from marble. The proof of concept has been provided with prior research.

The planned portable assessment system holds a high potential for innovation. In the future, more objects can be examined cost-effectively in short time periods. Damage can be identified at an early stage allowing for a target-oriented investment of efforts and financial resources.

Dresdner Knabe

Contact person: Dr. Kiran Varanasi
Funding by: BMBF
Grant agreement no.: 01IW15003
Begin: 01.08.2015
End: 31.07.2018

The objective of DYNAMICS is to develop a new methodology for 4D reconstruction of real world scenes with a small number of cameras, as well as to learn statistical models from the captured data sets. A 4D reconstruction refers to a sequence of accurate 3D reconstructions (including geometry, topology and surface properties) of a dynamic (evolving in time) real-world scene. We aim to build a robust lightweight capture system that can be easily installed and used (e.g. in the living room of a house, in outdoor environments, and broadly under various spatial and temporal constraints).

We are developing a novel interactive software system for motion estimation capitalizing on our experience from the predecessor project DENSITY and exploring new directions (new hardware and machine learning methods).

Specifically, the project DYNAMICS can be subdivided into several work packages according to the target scenarios and concerned areas of computer vision:

1) Software for an interactive monocular 4D reconstruction of non-rigid scenes. The main components are modules dealing with non-rigid structure from motion
(NRSfM) pipeline and non-rigid registration. Underlying technology will allow to reconstruct non-rigidly deforming scenes with a minimal number of assumptions from a single RGB camera.
Target scenarios include endoscopy, capture of facial expressions, small motion and post-factum reconstructions.

2) Software for robust 4D reconstruction from multiple views incorporating optical flow and scene flow with additional assumptions. We plan to assemble a capture studio with five Emergent HT-4000C high-speed cameras (a multi-view setting). Here, we aim at the highest precision and richness of detail in the reconstructions.

3) 3D shape templates with attributes derived from real data using deep learning techniques. The main objective of this work package is to provide statistical models as a prior knowledge in order to increase the robustness and accuracy of reconstructions. Furthermore, the shape templates will allow for more accurate reconstructions of articulated motion (e.g. skeleton poses) from uncalibrated
multi-view settings.

DYNAMICS is a BMBF project with an emphasis on development of core technologies applicable in other ongoing and forthcoming projects in the Augmented Vision Lab.


Contact person: Manthan Pancholi
Funding by: BMWi
Grant agreement no.: 01MD16003F
Begin: 01.01.2016
End: 30.06.2018

Project Partners:

            The aim of this research project is to create an open platform and associated digital infrastructure in the form of interfaces and architectures that allow market participants to achieve simple, secure and fair participation at different levels of value creation. This platform will be designed for the recording, analysis and aggregation of data acquired from everyday used sensors (Smart Objects), as well as the conversion of these data into digital services (Smart Services), both technically and with regard to the underlying business and business model. Furthermore, for an initial technical product area, sports shoes will be considered, the possibility is exerted to extend these to cyberphysical systems which go beyond the existing possibilities of data collection (pure sensors). This is to create a broadly communicable demonstrator object that can bring the possibilities of smart services and intelligent digitization to the general public. In addition, the technical prerequisites as well as suitable business models and processes are created in order to collect the data under strict observance of the legal framework as well as the consumer and data protection for the optimization of product development, production and logistics chains. Here, too, an open structure will allow market participants to work with these data at different levels of added value.

During the project period, national standardization and work for the necessary internationalization can be started in parallel, and all preparatory work should be done by the end of the project. Standardization is the foundation for trust to motivate companies to participate in the platform. On the basis of the defined standards a certification can take place. This is necessary to ensure data exchange (encryption) and the security of the networks (against cybercrime). This also includes clarifying questions about the right to data in the cloud.


Contact person: Stephan Krauß
Funding by: Stiftung RLP Innovation

The goal of the SwarmTrack project is the research and development of a novel method for accurate automatic tracking of objects moving in groups.

A common problem in object tracking are occlusions, where moving objects or static parts of the scene obstruct the view on the target objects. This may lead to tracking loss or wrong assignment of identity labels. Another important issue is confusion where nearby objects have a similar or identical appearance. This may lead to errors in the identity assignment, where labels are switched between objects.

SwarmTrack investigates possibilities for exploiting the group structure in the spatial and temporal domain to derive clues and constraints for correct identity assignment after occlusions and in the presence of similar objects. Such clues include the spatial arrangement of the tracked objects and its changes over time as well as motion continuity and coherency constraints. Furthermore, SwarmTrack investigates methods for the automatic creation of coherent groups as well as updating them, when objects leave the group or new objects join.

The resulting multi-target tracking approach has applications in traffic monitoring and analysis as well as other fields where objects move coherently in groups.

Body Analyzer

Contact person: Oliver Wasenmüller
Funding by: BMBF
Grant agreement no.: 01IS12050
Begin: 01.03.2015
End:  31.05.2016

Reconstruction and analyze of 3D human body models

Individual anthropometric measurements build the basis for a wide range of applications such as custom clothing or biometric identity verification. The possibility to extract these data automatically from 3D body models is therefore from high importance. Within the ‘Body Analyzer’ project a body scanner has been developed that is not only precise but also robust and manageable.

Extracted anthropometric measurements for five exemplary human body scans (Wasenmüller et al., 3DBST 2015)

Body analyzer: precise, robust, manageable 

In this project, a body scanner has been developed, which is distinguished from its competition by its precision, robustness and manageability as well as the addition of an analysis component as a body analyzer. The required hardware for capturing the data is thereby reduced to a single camera.


Joint use of color and depth information

A visual body scanner reconstructs a closed 3D model of a person from different camera views. These views can be obtained on the basis of a camera by rotating the person in front of this camera around its own axis, or by moving the camera around the person. If a depth camera is used, recent filter and global non-rigid registration methods are used to transform the recorded data into a consistent body model.

Generic template point clouds (left) and overlay of the registered template (middle) with the human body scan (Wasenmüller et al., 3DBST 2015)

Such systems have already been developed and published in the past. However, these systems have some limitations in precision, detail, and robustness that have made practical use difficult to date.

In this project, the limitations were largely eliminated by the joint use of color and depth information, and the system was expanded to include a cost-effective, non-technical body analyzer by supplementing an analytical component for extracting anthropometric measurements. Therefore, the project proposes a new method to define anthropometric measures once on a generic template using landmarks. After the initial definition the template can be registered against an individual body scan and the landmarks can be transferred to the scan using a new proposed algorithm. Exemplary extracted measurements are displayed in the figure above.


Further detail regarding RGB-D reconstruction and non-rigid movements (Link) and Precise and Automatic Anthropometric Measurement Extraction using Template Registration (Link).


Contact person: Oliver Wasenmüller
Funding by: BMBF
Grant agreement no.: 01IM13001J
Begin: 01.09.2013
End: 31.10.2016

3D Discrepancy Check for virtual product verification

The department Augmented Vision at DFKI deals in this project with a 3D Discrepancy Check for virtual product verification. Thereby a real object is captured in real-time with a RGB-D camera and compared to a reference model (e.g. CAD) of the object. Though algorithms for high precision reconstruction of objects with small or medium size using depth cameras are investigated and developed.

Capture of objects in real-time with RGB-D cameras

Discrepancy check is a well-known task in industrial application. Within this project, the department Augmented Vision at DFKI presents a new approach for Augmented Reality 3D Discrepancy Check consisting of three main contributions. First, a new two-step depth mapping algorithm for RGB-D cameras is proposed, which fuses depth images into consistent 3D models with an accuracy of around 0.01m outperforming state-of-the-art algorithms. Second, a semi-automatic alignment algorithm is proposed, which rapidly aligns a reference model to the reconstruction and third, an algorithm for 3D discrepancy check based on pre-computed distances is proposed.

The results of the project were summarized and published in the following paper: 3D discrepancy check

The new AR discrepancy check, which is able to capture scene geometry in real-time using a RGB-D camera (Wasenmüller et al., IEEE ISMAR 2016)
The new AR discrepancy check, which is able to capture scene geometry in real-time using a RGB-D camera (Wasenmüller et al., IEEE ISMAR 2016)

ARVIDA – Service oriented reference architecture for virtual technologies (VT)

The research on Discrepancy Check was performed within the ARVIDA project. This project is funded by the German Federal Ministry for Education and Research (BMBF) with actually 24 partners from research and industry. The main goal of the project ARVIDA is the creation of a service oriented reference architecture for virtual technologies (VT). The service orientation and the usage or rather adaption of available internet and VT-standards ensure interoperability between different modules and VT applications. A broad cross-company evaluation of the reference architecture in selected industrial scenarios guarantees that the results can be used as a future standard. As one exemplary application the 3D Discrepancy Check was developed.

Eyes Of Things

Contact person: Dr.-Ing. Alain Pagani
Funding by: EU
Grant agreement no.: 643924
Funding programm: H2020
Begin: 01.01.2015
End: 30.06.2018

The aim of the European Eyes of Things it to build a generic Vision device for the Internet of Things.

The device will include a miniature camera and a specific Vision Processing Unit in order to perform all necessary processing tasks directly on the device without the need of transferring the entire images to a distant server. The envisioned applications will enable smart systems to perceive their environment longer and more interactively.

The technology will be demonstrated with applications such as Augmented Reality, Wearable Computing and Ambient Assisted Living.