Contact person: Dr. Gerd Reis
Funding by: BMBF
Grant agreement no.: 03VP00293
Funding programm: VIP+
Begin: 01.10.2016
End: 30.09.2019

Contact person: Oliver Wasenmüller
Funding by: BMBF
Grant agreement no.: 13N14318
Funding programm: Photonik Forschung Deutschland – Digitale Optik
Begin: March 2017
End: February 2020

Design and application of an ultra-compact, energy-efficient and reconfigurable camera matrix for spatial analysis

Within the DAKARA project an ultra-compact, energy-efficient and reconfigurable camera matrix is developed. In addition to standard color images, it provides accurate depth information in real-time, forming the basis for various applications in the automotive (autonomous driving), production (Industry 4.0) and many more.

Real-time depth image calculations with camera matrix

The ultra-compact camera matrix is composed of 4×4 single cameras on a wafer and is equipped with a wafer-level optics, resulting in an extremely compact design not bigger than a cent-coin. This is made possible by the innovative camera technology of the AMS Sensors Germany GmbH.

The configuration as a camera matrix captures the scene from sixteen slightly displaced perspectives and thus allows the scene geometry (a depth image) to be calculated from these by means of the light field principle. Because such calculations are very high-intensity, close integration of the camera matrix with an efficient, embedded processor is required to enable real-time applications. The depth image calculations, which are researched and developed by DFKI (Department Augmented Vision), can be carried out in real-time in the electronic functional level of the camera system in a manner that is resource-conserving and real-time. Potential applications benefit significantly from the fact that the depth information is made available to them in addition to the color information without further calculations on the user side. Thanks to the ultra-compact design, it is possible to integrate the new camera into very small and / or filigree components and use it as a non-contact sensor. The structure of the camera matrix is reconfigurable so that a more specific layout can be used depending on the application. In addition, the depth image computation can also be reconfigured and thus respond to certain requirements for the depth information.

Ultra-compact, reconfigurable, low energy consumption
The innovation of the DAKARA project represents the overall system that provides both color and depth images. Similar systems, which are also found in the product application, are generally active systems that emit light and thus calculate the depth. Major disadvantages of such systems are large designs, high energy consumption and high costs. There are passive systems that have much lower energy consumption, but are still in the research stage and generally have large designs and low image rates.

For the first time, DAKARA offers a passive camera, which convinces with an ultra-compact design, high image rates, reconfigurable properties and low energy consumption, leaving the research stage and entering the market with well-known users from different domains.

Three-Dimensional Geometry for Automotive and Industry

In order to demonstrate the power and innovative power of the DAKARA concept, the new camera is used in two different application scenarios. These include an intelligent rear-view camera in the automotive field and the workplace assistant in manual production.

The planned intelligent rear-view camera of the partner ADASENS Automotive GmbH is capable of interpreting the rear vehicle environment spatially, metrically and semantically compared to currently used systems consisting of ultrasonic sensors and a mono color camera. As a result, even finer structures such as curbsides or poles can be recognized and taken into account during automated parking operations. In addition, the system is able to detect people semantically and to trigger warning signals in the event of an emergency. The DAKARA camera provides a significant contribution to increasing the safety of autonomous or semi-automated driving.

A manual assembly process at the Bosch Rexroth AG and DFKI (Department Innovative Factory Systems) is shown in the case of the workplace assistant. The aim is to support and assure the operator of his tasks. For this purpose, the new camera matrix is fixed over the workplace and both objects and hands are detected spatially and in time by the algorithms of the partner CanControls GmbH. A particular challenge is that objects such as tools or workpieces that are held in the hand are very difficult to separate from these. This separation is made possible by the additional provision of depth information by the DAKARA camera. In this scenario, a gripping path analysis, a removal and level control, the interaction with a dialog system and the tool position detection are implemented. The camera is designed to replace a large number of sensors, which are currently being used in various manual production systems by the project partner Bosch Rexroth, thus achieving a new quality and cost level.

In the next three years, the new camera matrix will be designed, developed and extensively tested in the mentioned scenarios. A first prototype will be realized by late summer 2018. The project “DAKARA” is funded by the Federal Ministry of Education and Research (BMBF) within the framework of the “Photonics Research Germany – Digital Optics” program. The project volume totals 3.8 million euros; almost half of it being provided by the industry partners involved.

Partners:

• AMS Sensors Germany GmbH, Nürnberg (Konsortialführung)
• Deutsches Forschungszentrum für Künstliche Intelligenz GmbH (DFKI), Kaiserslautern (Technische Konsortialführung)
• ADASENS Automotive GmbH, Lindau
• Bosch Rexroth AG, Stuttgart
• CanControls, Aachen

Contact person: Dr. Gerd Reis
Funding by: BMWi
Grant agreement no.: 03ET1416A-F
Funding programm: EnOB
Begin: 01.11.2016
End: 31.10.2019

Light and solar management using active and model-predictively controlled Components

Project partners: Universität Kaiserslautern, DFKI GmbH Kaiserslautern, ebök Planung und Entwicklung GmbH, Dresden Elektronik Ingenieurtechnik GmbH, Agentilo GmbH, Herbert Waldmann GmbH & Co. KG

Contact person: Dr. Kiran Varanasi
Funding by: BMBF
Grant agreement no.: 01IW15003
Begin: 01.08.2015
End: 31.07.2018

The objective of DYNAMICS is to develop a new methodology for 4D reconstruction of real world scenes with a small number of cameras, as well as to learn statistical models from the captured data sets. A 4D reconstruction refers to a sequence of accurate 3D reconstructions (including geometry, topology and surface properties) of a dynamic (evolving in time) real-world scene. We aim to build a robust lightweight capture system that can be easily installed and used (e.g. in the living room of a house, in outdoor environments, and broadly under various spatial and temporal constraints).

We are developing a novel interactive software system for motion estimation capitalizing on our experience from the predecessor project DENSITY and exploring new directions (new hardware and machine learning methods).

Specifically, the project DYNAMICS can be subdivided into several work packages according to the target scenarios and concerned areas of computer vision:

1) Software for an interactive monocular 4D reconstruction of non-rigid scenes. The main components are modules dealing with non-rigid structure from motion
(NRSfM) pipeline and non-rigid registration. Underlying technology will allow to reconstruct non-rigidly deforming scenes with a minimal number of assumptions from a single RGB camera.
Target scenarios include endoscopy, capture of facial expressions, small motion and post-factum reconstructions.

2) Software for robust 4D reconstruction from multiple views incorporating optical flow and scene flow with additional assumptions. We plan to assemble a capture studio with five Emergent HT-4000C high-speed cameras (a multi-view setting). Here, we aim at the highest precision and richness of detail in the reconstructions.

3) 3D shape templates with attributes derived from real data using deep learning techniques. The main objective of this work package is to provide statistical models as a prior knowledge in order to increase the robustness and accuracy of reconstructions. Furthermore, the shape templates will allow for more accurate reconstructions of articulated motion (e.g. skeleton poses) from uncalibrated
multi-view settings.

DYNAMICS is a BMBF project with an emphasis on development of core technologies applicable in other ongoing and forthcoming projects in the Augmented Vision Lab.

Contact person: Manthan Pancholi
Funding by: BMWi
Grant agreement no.: 01MD16003F
Begin: 01.01.2016
End: 30.06.2018

Project Partners:

            The aim of this research project is to create an open platform and associated digital infrastructure in the form of interfaces and architectures that allow market participants to achieve simple, secure and fair participation at different levels of value creation. This platform will be designed for the recording, analysis and aggregation of data acquired from everyday used sensors (Smart Objects), as well as the conversion of these data into digital services (Smart Services), both technically and with regard to the underlying business and business model. Furthermore, for an initial technical product area, sports shoes will be considered, the possibility is exerted to extend these to cyberphysical systems which go beyond the existing possibilities of data collection (pure sensors). This is to create a broadly communicable demonstrator object that can bring the possibilities of smart services and intelligent digitization to the general public. In addition, the technical prerequisites as well as suitable business models and processes are created in order to collect the data under strict observance of the legal framework as well as the consumer and data protection for the optimization of product development, production and logistics chains. Here, too, an open structure will allow market participants to work with these data at different levels of added value.

During the project period, national standardization and work for the necessary internationalization can be started in parallel, and all preparatory work should be done by the end of the project. Standardization is the foundation for trust to motivate companies to participate in the platform. On the basis of the defined standards a certification can take place. This is necessary to ensure data exchange (encryption) and the security of the networks (against cybercrime). This also includes clarifying questions about the right to data in the cloud.

Contact person: Stephan Krauß
Funding by: Stiftung RLP Innovation

The goal of the SwarmTrack project is the research and development of a novel method for accurate automatic tracking of objects moving in groups.

A common problem in object tracking are occlusions, where moving objects or static parts of the scene obstruct the view on the target objects. This may lead to tracking loss or wrong assignment of identity labels. Another important issue is confusion where nearby objects have a similar or identical appearance. This may lead to errors in the identity assignment, where labels are switched between objects.

SwarmTrack investigates possibilities for exploiting the group structure in the spatial and temporal domain to derive clues and constraints for correct identity assignment after occlusions and in the presence of similar objects. Such clues include the spatial arrangement of the tracked objects and its changes over time as well as motion continuity and coherency constraints. Furthermore, SwarmTrack investigates methods for the automatic creation of coherent groups as well as updating them, when objects leave the group or new objects join.

The resulting multi-target tracking approach has applications in traffic monitoring and analysis as well as other fields where objects move coherently in groups.

Contact person: Oliver Wasenmüller
Funding by: BMBF
Grant agreement no.: 01IS12050
Begin: 01.03.2015
End:  31.05.2016

Reconstruction and analyze of 3D human body models

Individual anthropometric measurements build the basis for a wide range of applications such as custom clothing or biometric identity verification. The possibility to extract these data automatically from 3D body models is therefore from high importance. Within the ‘Body Analyzer’ project a body scanner has been developed that is not only precise but also robust and manageable.

Body analyzer: precise, robust, manageable 

In this project, a body scanner has been developed, which is distinguished from its competition by its precision, robustness and manageability as well as the addition of an analysis component as a body analyzer. The required hardware for capturing the data is thereby reduced to a single camera.

Joint use of color and depth information

A visual body scanner reconstructs a closed 3D model of a person from different camera views. These views can be obtained on the basis of a camera by rotating the person in front of this camera around its own axis, or by moving the camera around the person. If a depth camera is used, recent filter and global non-rigid registration methods are used to transform the recorded data into a consistent body model.

Such systems have already been developed and published in the past. However, these systems have some limitations in precision, detail, and robustness that have made practical use difficult to date.

In this project, the limitations were largely eliminated by the joint use of color and depth information, and the system was expanded to include a cost-effective, non-technical body analyzer by supplementing an analytical component for extracting anthropometric measurements. Therefore, the project proposes a new method to define anthropometric measures once on a generic template using landmarks. After the initial definition the template can be registered against an individual body scan and the landmarks can be transferred to the scan using a new proposed algorithm. Exemplary extracted measurements are displayed in the figure above.

Further detail regarding RGB-D reconstruction and non-rigid movements (Link) and Precise and Automatic Anthropometric Measurement Extraction using Template Registration (Link).

Contact person: Oliver Wasenmüller
Funding by: BMBF
Grant agreement no.: 01IM13001J
Begin: 01.09.2013
End: 31.10.2016

3D Discrepancy Check for virtual product verification

The department Augmented Vision at DFKI deals in this project with a 3D Discrepancy Check for virtual product verification. Thereby a real object is captured in real-time with a RGB-D camera and compared to a reference model (e.g. CAD) of the object. Though algorithms for high precision reconstruction of objects with small or medium size using depth cameras are investigated and developed.

Capture of objects in real-time with RGB-D cameras

Discrepancy check is a well-known task in industrial application. Within this project, the department Augmented Vision at DFKI presents a new approach for Augmented Reality 3D Discrepancy Check consisting of three main contributions. First, a new two-step depth mapping algorithm for RGB-D cameras is proposed, which fuses depth images into consistent 3D models with an accuracy of around 0.01m outperforming state-of-the-art algorithms. Second, a semi-automatic alignment algorithm is proposed, which rapidly aligns a reference model to the reconstruction and third, an algorithm for 3D discrepancy check based on pre-computed distances is proposed.

The results of the project were summarized and published in the following paper: 3D discrepancy check

ARVIDA – Service oriented reference architecture for virtual technologies (VT)

The research on Discrepancy Check was performed within the ARVIDA project. This project is funded by the German Federal Ministry for Education and Research (BMBF) with actually 24 partners from research and industry. The main goal of the project ARVIDA is the creation of a service oriented reference architecture for virtual technologies (VT). The service orientation and the usage or rather adaption of available internet and VT-standards ensure interoperability between different modules and VT applications. A broad cross-company evaluation of the reference architecture in selected industrial scenarios guarantees that the results can be used as a future standard. As one exemplary application the 3D Discrepancy Check was developed.

Contact person: Dr.-Ing. Alain Pagani
Funding by: EU
Grant agreement no.: 643924
Funding programm: H2020
Begin: 01.01.2015
End: 30.06.2018

The aim of the European Eyes of Things it to build a generic Vision device for the Internet of Things.

Contact person: Dr.-Ing. Alain Pagani
Funding by: EU
Grant agreement no.: 641460
Funding programm: H2020
Begin: 01.02.2015
End: 30.06.2017

LARA is a European Project aiming at developing a new mobile device for helping employees of utilities companies in their work on the field. The device to be developed – called the LARA System – consists of a tactile tablet and a set of sensors that can geolocalise the device using the European GALILEO system and EGNOS capabilities.

The LARA system is produced under a collaborative work where different players, SMEs, large companies, universities and research institutes are contributing with different expertise.