DFKI AV – Stellantis Collaboration on Radar-Camera Fusion – Papers at GCPR and EUSIPCO

DFKI Augmented Vision is collaborating with Stellantis on the topic of Radar-Camera Fusion for Automotive Object Detection using Deep Learning. Recently, two new publications were accepted to the GCPR 2023 and EUSIPCO 2023 conferences.

The 2 new publications are:

1.  Cross-Dataset Experimental Study of Radar-Camera Fusion in Bird’s-Eye ViewProceedings of the 31st. European Signal Processing Conference (EUSIPCO-2023), September 4-8, Helsinki, Finland, IEEE, 2023.

Lukas Stefan Stäcker, Philipp Heidenreich, Jason Rambach, Didier Stricker

This paper investigates the influence of the training dataset and transfer learning on camera-radar fusion approaches, showing that while the camera branch needs large and diverse training data, the radar branch benefits more from a high-performance radar.

Cross-Dataset Experimental Study of Radar-Camera Fusion in Bird’s-Eye View

2. RC-BEVFusion: A Plug-In Module for Radar-Camera Bird’s Eye View Feature FusionProceedings of. Annual Symposium of the German Association for Pattern Recognition (DAGM-2023), September 19-22, Heidelberg, BW, Germany, DAGM, 9/2023.

Lukas Stefan Stäcker, Shashank Mishra, Philipp Heidenreich, Jason Rambach, Didier Stricker

This paper introduces a new Bird’s Eye view fusion network architecture for camera-radar fusion for 3D object detection that performs favorably on the NuScenes dataset benchmark.

RC-BEVFusion: A Plug-In Module for Radar-Camera Bird’s Eye View Feature Fusion

Contacts: Dr. Jason Rambach

ICCV 2023: 4 papers accepted

We are happy to announce that the Augmented Vision group will present 4 papers in the upcoming ICCV 2023 Conference, 2-6 October, Paris, France. The IEEE/CVF International Conference in Computer Vision (ICCV) is the premier international computer vision event. Homepage: https://iccv2023.thecvf.com/  

The 4 accepted papers are:

  1. U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds
    Yan Di, Chenyangguang Zhang, Ruida Zhang, Fabian Manhardt, Yongzhi Su, Jason Raphael Rambach, Didier Stricker, Xiangyang Ji, Federico Tombari
  2. FeatEnHancer: Enhancing Hierarchical Features for Object Detection and Beyond Under Low-Light Vision. Khurram Azeem Hashmi, Goutham Kallempudi, Didier Stricker, Muhammad Zeshan Afzal
  3. Introducing Language Guidance in Prompt-based Continual Learning Muhammad Gulzain Ali Khan, Muhammad Ferjad Naeem; Luc Van Gool; Federico  Tombari; Didier Stricker, Muhammad Zeshan Afzal
  4. DELO: Deep Evidential LiDAR Odometry using Partial Optimal Transport Sk Aziz Ali, Djamila Aouada, Gerd Reis, Didier Stricker
AI-Observer Summer School

The first AI-Observer Summer School was held at the Eratosthenes Center of Excellence in Limassol, Cyprus, from July 10-14. Training sessions were given by Prof. Fabio Del Frate, Giorgia Guerrisi and Lorenzo Giuliano Papale (Tor Vergata University of Rome), and Dr. Gerd Reis (German Research Center for Artificial Intelligence). During the five-day hybrid event, more than 50 participants learned about the application of artificial intelligence in Earth observation, with special focus on disaster risk management. Topics included deforestation, flood detection, and natural hazard management using Sentinel-1 Synthetic Aperture RADAR (SAR), and Sentinel-2 Multi-Spectral Imaging (MSI) data.

AI-OBSERVER consortium:
Eratosthenes Center of Excellence
Deutsches Forschungszentrum für Künstliche Intelligenz
Università degli Studi di Roma Tor Vergata
Cellock Ltd.

Michael Lorenz won a best Paper award

We are glad to announce that our colleague Michael Lorenz won a best Paper award for his work On Motions artifacts arising when integrating inertial sensors into loose clothing such as a working jacket.


  • Inertial human motion capture (IHMC) has become a robust tool to estimate human kinematics in the wild such as industrial facilities.
  • In contrast to optical motion capture, where occlusions might take place, the kinematics of a worker can be continuously provided.
  • This is for instance a prerequisite for an ergonomic assessments of the workers.
  • State-of-the-art IHMC solutions require inertial sensors to be tightly attached to body segments.
  • This requires an additional setup time and lowers the practicability and ease of use when it comes to an industrial application.
  • In contrast, sensors integrated into loose clothing such as a working jacket, may yield corrupted kinematics estimates due to the additional motion of loose clothing.
  • In this work we present a study of orientations deviations obtained from kinematics estimates using tightly attached inertial sensors and into a working jacket integrated ones.
  • We performed a quantitative analysis using data from the two hardware setups worn by 19 subjects performing different industry related tasks and measures of their body shapes.
  • Using this data we approximated probability distributions of the deviation angles for each person and body segment.
  • Applying different statistical measures we could gain insights to questions like, how severe orientation deviations are, if there is an influence of body shapes on the distribution and how probability distributions of the deviation angles can indicate physical motion limitations of a sensor attached to a segment.
3rd place in Scan-to-BIM challenge (CV4_AEC Workshop, CVPR 2023) for HumanTech project team

The team of the EU Horizon Project HumanTech , consisting of Mahdi Chamseddine and Dr. Jason Rambach from DFKI Augmented Vision as well as Fabian Kaufmann from RPTU Kaiserslautern – department of Civil Engineering, received the 3rd place prize in the Scan-to-BIM challenge of the (Computer Vision in the Built Environment) CV4_AEC Workshop of the CVPR 2023 conference.

On the 18.6, the team presented their solution and results as part of the workshop program. Scan-to-BIM solutions are of great importance for the construction community as they automate the generation of as-built models of buildings from 3D scans, and can be used for quality monitoring, robotic task planning and XR visualization, among other applications.

HumanTech project: https://humantech-horizon.eu/

CV4AEC Workshop page: https://cv4aec.github.io/

Contact: Dr. Jason Rambach , Mahdi Chamseddine

Special Issue on the IEEE ARSO 2023 Conference: Human Factors in Construction Robotics

Dr. Jason Rambach, coordinator of the EU Horizon Project HumanTech co-organized a special session on “Human Factors in Construction Robotics” at the IEEE International Conference on Advanced Robotics and its Social Impacts (ARSO 2023) in Berlin, Germany (5.6-7.6). The organization of the special session was done by Jason Rambach, Gabor Sziebig, Research Manager at SINTEF, and Mihoko Niitsuma, Professor at Chuo University.

The program of the special session included the following talks:

  • Serena Ivaldi (INRIA) – Teleoperating a robot for removing asbestos tiles on roofs: Insights from a pilot study
  • Jason Rambach (DFKI) – Machine perception for human-robot handover scenarios in construction
  • Patricia Helen Rosen (BAUA) – Design recommendations for construction robots – a human-centred perspective
  • Dimitrios Giakoumis (CERTH ITI) – Designing human-robot interaction interfaces for shotcrete construction robots; the RobetArme project case

HumanTech project: https://humantech-horizon.eu/

Contact: Dr. Jason Rambach

Paper accepted at the ICRA conference

We are happy to announce that our paper titled

Structure PLP-SLAM: Efficient Sparse Mapping and Localization using Point, Line and Plane for Monocular, RGB-D and Stereo CamerasFangwen Shu, Jiaxuan Wang, Alain Pagani, Didier Stricker

has been accepted at the IEEE International Conference on Robotics and Automation (ICRA) 2023.

In this paper, we present a visual SLAM system that uses both points and lines for robust camera localization, and
simultaneously performs a piece-wise planar reconstruction (PPR) of the environment to provide a structural map in
real-time. Our proposed SLAM tightly incorporates the semantic and geometric features to boost both frontend pose tracking and backend map optimization.

The ICRA conference takes place this year in London, from May 29th to June 2nd.

Contact: Dr. Alain Pagani

Workshop on AI and Robotics in Construction at ERF 2023

Dr. Jason Rambach, coordinator of the EU Horizon Project HumanTech co-organized a workshop on “AI and Robotics in Construction” at the European Robotics Forum 2023 in Odense, Denmark (March 14th to 16th, 2023) in cooperation with the construction Robotics projects Beeyonders and RobetArme.

From the project HumanTech, Jason Rambach presented an overview of the project objectives as well as insights into the results achieved by Month 9 of the project. Patrick Roth from the partner Implenia, presented the perspective and challenges of the construction industry on the use of Robotics and AI in construction sites, while the project partners Dr. Bharath Sankaran (Naska.AI) and Dr. Gabor Sziebig (SINTEF) participated in a panel session discussing the future of Robotics in construction.

Workshop schedule: https://erf2023.sdu.dk/timetable/event/ai-and-robotics-in-construction/

HumanTech project: https://humantech-horizon.eu/                                                              

Contact: Dr. Jason Rambach

Dr. Jason Rambach giving his presentation.
Start of the EU project ExtremeXP

The kick-off meeting for the EU project ExtremeXP was held on January 26th and 27th, 2023, in the city of Athens, Greece.

The vision of ExtremeXP “EXPerimentation driven and user eXPerience-oriented analytics for eXtremely Precise outcomes and decisions” is to provide accurate, precise, fit-for-purpose, and trustworthy data-driven insights via evaluating different complex analytics variants, considering end users’ preferences and feedback in an automated way.

Dr. Alain Pagani enlightened the audience on the capabilities of the Augmented Vision department, highlighting the expertise and key strengths it brings to the table in the realm of Augmented Reality and Explainability.

ExtremeXP will provide:

  • Specification and semantics for modelling complex user-experience-driven analytics.
  • Automated and scalable data management for complex analytics workflow.
  • Scenario-driven and opportunistic machine learning to design and develop AutoML mechanisms for performing scenario-based algorithm and model selection considering on-demand user-provided constraints (performance, resources, time, model options).
  • User-experience- and experiment-driven optimization of complex analytics to design the architecture of the framework for experiment-driven optimisation of complex analytics.
  • Transparent decision making with interactive visualisation methods to explore how augmented reality, visual analytics, and other visualisation techniques and their combinations can enhance user experience for different use cases, actors, domains, applications, and problem areas
  • Extreme data access control and knowledge management
  • Test and validation framework and application on different impactful real-life use cases to incorporate the ExtremeXP tools, methods, models, and software into a scalable, usable, and interoperable integrated framework for complex experiment-driven analytics

The project consortium consists of 20 partners, which are:

  1. Athena Research Center (coordinator) [Greece]
  2. Activeeon [France]
  3. Airbus Defense and Spaces SLC [France]
  4. BitSparkles [France]
  5. Bournemouth University [United Kingdom]
  6. CS-Group [France]
  7. Charles University of Prague [Czech Republic]
  8. Deutsches Forschungszentrum für Künstliche Intelligenz [Germany]
  9. Fundacio Privada I2cat, Internet I Innovacio Digital A Catalunya [Spain]
  10. Institute of Communications and Computer Systems [Greece]
  11. IDEKO [Spain]
  12. INTERACTIVE4D [France]
  14. IThinkUPC [Spain]
  15. MOBY X [Cyprus]
  16. SINTEF [Norway]
  17. Technical University of Delft [Netherlands]
  18. University of Ljubljana [Slovenia]
  19. Universitat Politècnica De Catalunya [Spain]
  20. Vrije Universiteit Amsterdam [Netherlands]

Contact persons:

Mohamed Selim

Dr. Alain Pagani

Nareg Minaskan Karabid

Article in IEEE Robotics and Automation Letter (RA-L) journal

We are happy to announce that our article “OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection” was published in the prestigious IEEE Robotics and Automation Letters (RA-L) Journal. The work is a collaboration of DFKI with the TU Munich and Google. The article is openly accessible at: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=10021668                                                                      

Abstract: Monocular 3D object detection has recently made a significant leap forward thanks to the use of pre-trained depth estimators for pseudo-LiDAR recovery. Yet, such two-stage methods typically suffer from overfitting and are incapable of explicitly encapsulating the geometric relation between depth and object bounding box. To overcome this limitation, we instead propose to jointly estimate dense scene depth with depth-bounding box residuals and object bounding boxes, allowing a two-stream detection of 3D objects that harnesses both geometry and context information. Thereby, the geometry stream combines visible depth and depth-bounding box residuals to recover the object bounding box via explicit occlusion-aware optimization. In addition, a bounding box based geometry projection scheme is employed in an effort to enhance distance perception. The second stream, named as the Context Stream, directly regresses 3D object location and size. This novel two-stream representation enables us to enforce cross-stream consistency terms, which aligns the outputs of both streams, and further improves the overall performance. Extensive experiments on the public benchmark demonstrate that OPA-3D outperforms state-of-the-art methods on the main Car category, whilst keeping a real-time inference speed.

Yongzhi Su, Yan Di, Guangyao Zhai, Fabian Manhardt, Jason Rambach, Benjamin Busam, Didier Stricker and Federico Tombari “OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection.IEEE Robotics and Automation Letters (2023).

Contacts: Yongzhi Su, Dr. Jason Rambach