SSMP Team: Spatial Sensing and Machine Perception
Team SSMP (Spatial Sensing and Machine Perception) focuses on the use of diverse 3D/2D sensing modalities (RGB/Stereo/ToF cameras, Radar, Lidar) to address challenging scene perception problems using Machine Learning/Deep Learning and traditional geometric Computer Vision. Such problems include as Semantic Scene Reconstruction, 6DoF Object Pose Estimation, Unsupervised Anomaly Detection, SLAM, Deep Sensor Fusion. Indicative application areas are Smart building sensing, AI in construction, Autonomous Driving, Industrial Robotics and Augmented Reality.
Projects of SSMP Team
GreifbAR

The goal of the GreifbAR project is to make mixed reality (MR) worlds, including virtual (VR) and augmented reality (“AR”), tangible and graspable by allowing users to interact with real and virtual objects with their bare hands. Hand accuracy and dexterity is paramount for performing precise tasks in many fields, but the capture of hand-object interaction in current MR systems is woefully inadequate. Current systems rely on hand-held controllers or capture devices that are limited to hand gestures without contact with real objects. GreifbAR solves this limitation by introducing a sensing system that detects both the full hand grip including hand surface and object pose when users interact with real objects or tools. This sensing system will be integrated into a mixed reality training simulator that will be demonstrated in two relevant use cases: industrial assembly and surgical skills training. The usability and applicability as well as the added value for training situations will be thoroughly analysed through user studies.
Partners
NMY – Mixed Reality Communication GmbH Hanauer Landstraße 188, 60314 Frankfurt am Main
Charité – Universitätsmedizin Berlin Charitéplatz 1, 10117 Berlin
Universität Passau vertreten durch den Präsidenten, Prof. Dr. Ulrich Bartosch Innstraße 40, 94032 Passau Wissenschaftliche Leitung: Prof. Dr. Susanne Mayr Lehrstuhl für Psychologie mit Schwerpunkt Mensch – Maschine – Interaktion
Contact
Dr.-Ing. Jason Raphael Rambach
- Jason_Raphael.Rambach@dfki.de
- Phone +49 631 20575 3740
- Yongzhi.Su@dfki.de
- Phone +49 631 20575 3416
VIZTA
Vision, Identification with Z-sensing Technologies and key Applications
VIZTA project, coordinated by ST Micrelectronics, aims at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value in several key applications including automotive, security, smart buildings, mobile robotics for smart cities, and industry4.0. The key differentiating 12-inch Silicon sensing technologies developed during VIZTA are:
• Innovative SPAD and lock-in pixel for Time of Flight architecture sensors
• Unprecedent and cost-effective NIR and RGB-Z filters on-chip solutions
• complex RGB+Z pixel architectures for multimodal 2D/3D imaging For short-range sensors : advanced VCSEL sources including wafer-level GaAs optics and associated high speed driver.
These developed differentiating technologies allows the development and validation of innovative 3D imaging sensors products with the following highly integrated prototypes demonstrators:
• High resolution (>77 000 points) time-of-flight ranging sensor module with integrated VCSEL, drivers, filters and optics.
• Very High resolution (VGA min) depth camera sensor with integrated filters and optics.
For Medium and Long range sensing, VIZTA also adresses new LiDAR systems with dedicated sources, optics and sensors Technology developments of sensors and emitters are carried out by leading semiconductor product suppliers (ST Microelectronics, Philips, III-V Lab) with the support of equipment suppliers (Amat, Semilab) and CEA Leti RTO.
VIZTA project also include the developement of 6 demonstrators for key applications including automotive, security, smart buildings, mobile robotics for smart cities, and industry4.0 with a good mix of industrial and academic partners (Ibeo, Veoneer, Ficosa, Beamagine, IEE, DFKI, UPC, Idemia, CEA-List, ISD, BCB, IDE, Eurecat). VIZTA consortium brings together 23 partners from 9 countries in Europe: France, Germany, Spain, Greece, Luxembourg, Latvia, Sweden, Hungary, and United Kingdom.
Partners :
Universidad Politecnica Catalunya
Commisariat a l Energie Atomique et aux Energies Alternatives (CEA Paris)
Fundacio Eurecat
STMICROELECTRONICS SA
BCB Informática y Control
Alter Technology TÜV Nord SA
FICOMIRRORS SA
Philips Photonics GmbH
Applied Materials France SARL
SEMILAB FELVEZETO FIZIKAI LABORATORIUM RESZVENYTARSASAG
ELEKTRONIKAS UN DATORZINATNU INSTITUTS
LUMIBIRD
IEE S.A.
IBEO Automotive Systems GmbH
STMICROELECTRONICS RESEARCH & DEVELOPMENT LTD
STMICROELECTRONICS SA
IDEMIA IDENITY & SECURITY FRANCE
Beamagine S.L.
Integrated Systems Development S.A.
VEONEER SWEDEN AB
III-V Lab
STMICROELECTRONICS (ALPS) SAS
STMICROELECTRONICS GRENOBLE 2 SAS
Kontakt: | Jason Rambach ; Bruno Mirbach |
E-Mail: | Jason.Rambach@dfki.de |
Telefon: | +49 631 20575 3740 |
SSMP News
2023
Workshop on AI and Robotics in Construction at ERF 2023
Dr. Jason Rambach, coordinator of the EU Horizon Project HumanTech co-organized a workshop on “AI and Robotics in Construction” at the European ... Read more
Article in IEEE Robotics and Automation Letter (RA-L) journal
We are happy to announce that our article “OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection” was published in the prestigious IEEE Robotics and Automation Letters (RA-L) Journal. The work is a collaboration of DFKI with the... Read more
Radar Driving Activity Dataset (RaDA) Released
DFKI Augmented Vision recently released the first publicly available UWB Radar Driving Activity Dataset (RaDA), consisting of over 10k data samples from 10 different participants annotated with 6 driving activities. The dataset was recorded in the DFKI driving simulator e... Read more
2022
VIZTA Project successfully concluded after 42 months
The Augmented Vision department of DFKI participated in the VIZTA project, coordinated by ST Microelectronics, aiming at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value ... Read more
DFKI Augmented Vision Researchers win two awards in Object Pose Estimation challenge (BOP Challenge, ECCV 2022)
DFKI Augmented Vision researchers Yongzhi Su, Praveen Nathan and Jason Rambach received their 1st place award in the prestigious BOP Challenge 2022 in the categories Overall Best Segmentation Method and The Best BlenderProc-Trained Segmentation Method.... Read more
Kick-Off for EU Project “HumanTech”
Our Augmented Vision department is the coordinator of the new large European project "HumanTech". The Kick-Off meeting was held on July 20th, 2022, at DFKI in Kaiserslautern. Please read the whole article here: Artificia... Read more
ARTIFICIAL INTELLIGENCE FOR A SAFE AND SUSTAINABLE CONSTRUCTION INDUSTRY
Please check out the article "Artificial intelligence for a safe and sustainable construction industry (dfki.de)" concerning the new EU project HumanTech which is coordinated by Read more
Augmented Vision @CVPR 2022
DFKI Augmented Vision had a strong presence in the recent CVPR 2022 Conference held on June 19th-23rd, 2022, in New Orleans, USA. The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) is the premier annual computer vision event internationally. Homepage: ... Read more
Keynote Presentation by Dr. Jason Rambach in Computer Vision session of the Franco-German Research and Innovation Network event
On June 14th, 2022, Dr. Jason Rambach gave a keynote talk in the Computer Vision session of the Franco-German Research and Innovation Network event held at the Inria headquarters in Versailles, Paris, France. In the talk, an overview of the current activities of the... Read more
CVPR 2022: Two papers accepted
We are happy to announce that the Augmented Vision group will present two papers in the upcoming CVPR 2022 Conference from June 19th-23rd in New Orleans, USA. The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) is the premier annual computer vision even... Read more
2021
2 Papers accepted at BMVC 2021 Conference
We are happy to announce that the Augmented Vision group will present 2 papers in the upcoming BMVC 2021 Conference, 22-25 November, 2021:
The British Machine Vision Conference (BMVC) is the British Machine Vision Assoc... Read more
DFKI AV – Stellantis Collaboration on Radar-Camera Fusion – 2 publications
DFKI Augmented Vision is working with Stellantis on the topic of Radar-Camera Fusion for Automotive Object Detection using Deep Learning since 2020. The collaboration has already led to two publications, in ICCV 2021 (International Conference on Computer Vision – Read more
VIZTA Project Time-of-Flight Camera Datasets Released
As part of the research activities of DFKI Augmented Vision in the VIZTA project (https://www.vizta-ecsel.eu/), two publicly available datasets have been released and are available for download. TIMo dataset is a ... Read more
XR for nature and environment survey
On July 29th, 2021, Dr. Jason Rambach presented the survey paper “A Survey on Applications of Augmented, Mixed and Virtual Reality for Nature and Environment” at the 23rd Human Computer Interaction Conference Read more
VIZTA Project 24M Review and public summary
DFKI participates in the VIZTA project, coordinated by ST Micrelectronics, aiming at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value in several key applications in... Read more
Paper accepted at ICIP 2021
We are happy to announce that our paper “SEMANTIC SEGMENTATION IN DEPTH DATA : A COMPARATIVE EVALUATION OF IMAGE AND POINT CLOUD B... Read more
TiCAM Dataset for in-Cabin Monitoring released
As part of the research activities of DFKI Augmented Vision in the VIZTA project (https://www.vizta-ecsel.eu/), we have published the open-source dataset for automotive in-cabin monitoring with a wide-angle time-of-flight depth se... Read more
Presentation on Machine Learning and Computer Vision by Dr. Jason Rambach
On March 4th, 2021, Dr. Jason Rambach gave a talk on Machine Learning and Computer Vision at the GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit) workshop o... Read more
VIZTA project: 18-month public project summary released
DFKI participates in the VIZTA project, coordinated by ST Micrelectronics, aiming at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value in several key applications in... Read more
3 Papers accepted at VISAPP 2021
We are excited to announce that the Augmented Vision group will present 3 papers in the upcoming VISAPP 2021 Conference, February 8th-10th, 2021:
The International Conference on Computer Vision Theory and Applications ... Read more
Article at MDPI Sensors journal
We are happy to announce that our paper “SynPo-Net–Accurate and Fast CNN-Based 6DoF Object Pose Estimation Using Synthetic Training” has been accepted for publication at the MDPI Sensors journal, Special Issue Object Tracking and Motion Analysis. Sensors (ISSN 14... Read more
2020
Four papers accepted at WACV 2021
The Winter Conference on Applications of Computer Vision (WACV 2021) is IEEE’s and the PAMI-TC’s premier meeting on applications of computer vision. With its high quality and low cost, it provides an exceptional value for students... Read more
Head of SSMP Team
Member | Position | Phone | ||
---|---|---|---|---|
Dr. Jason Raphael Rambach | Senior Researcher | jason_raphael.rambach@dfki.de | +49 631 20575-3740 |
Members of SSMP Team
Member | Position | Phone | ||
---|---|---|---|---|
Dr. Bruno Mirbach | Senior Researcher | Bruno_Walter.Mirbach@dfki.uni-kl.de | +49 631 20575 3511 | |
Henri Hoyez | Researcher | Henri.Hoyez@dfki.de | ||
Jigyasa Singh Katrolia | Researcher | Jigyasa_Singh.Katrolia@dfki.de | +49 631 20575 3528 | |
Lukas Stefan Staecker | Researcher | Lukas_Stefan.Staecker@dfki.uni-kl.de | ||
Mahdi Chamseddine | Researcher | Mahdi.Chamseddine@dfki.de | +49 631 20575-3527 | |
Pascal Schneider | Researcher | Pascal.Schneider@dfki.uni-kl.de | ||
Praveen Nathan | Researcher | praveen.annamalai_nathan@dfki.de | ||
Suresh Guttikonda | Researcher | Suresh.Guttikonda@dfki.de | +49 631 20575 3580 | |
Yongzhi Su | Researcher | yongzhi.su@dfki.de | +49 631 20575-3416 | |
Yuriy Anisimov | Researcher | yuriy.anisimov@dfki.de | +49 631 20575-3750 |
Publications of SSMP Team
2023
ONTOLOGY-BASED SEMANTIC LABELING FOR RGB-D AND POINT CLOUD DATASETS
Computing in Construction. European Conference on Computing in Construction (EC3-2023), July 10-12, Crete, Greece, EC3, 2023.
Details
| Link 1
OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection
IEEE Robotics and Automation Letters (RA-L), Vol. 8, Pages 1327-1334, IEEE, 3/2023.
Details
| Link 1
Principles of Object Tracking and Mapping
Andrew Yeh Ching Nee; Soh Khim Ong. Springer Handbook of Augmented Reality. Pages 53-84, Springer Handbooks, ISBN 978-3-030-67821-0, Springer, Switzerland, 1/2023.
Details
| Link 1
Driving Activity Recognition Using UWB Radar and Deep Neural Networks
Sensors - Open Access Journal (Sensors) 23 2 Seiten 1-15 MDPI 1/2023 .
Details
| Link 1
| Link 2
2022
Unsupervised Image-to-Image Translation: A Review
Sensors - Open Access Journal (Sensors) 2022, 22, 8540 MDPI 2022 .
Details
| Link 1
Editorial – Advanced Scene Perception for Augmented Reality
Journal of Imaging (MDPI J) Special Issue Advanced Scene Perception for Augmented Reality MDPI 10/2022 .
Details
| Link 1
Unsupervised Anomaly Detection from Time-of-Flight Depth Images
CVPR Workshop on Perception Beyond the Visible Spectrum. CVPR Workshop on Perception Beyond the Visible Spectrum (PBVS-2022) befindet sich International Conference on Computer Vision and Pattern Recognition June 19-20 United States IEEE Computer Society 2022 .
Details
| Link 1
TIMo—A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera
Sensors - Open Access Journal (Sensors) 22 11 Seiten 3992-4005 MPDI Basel 5/2022 .
Details
| Link 1
ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation
IEEE/CVF. International Conference on Computer Vision and Pattern Recognition (CVPR-2022) June 19-24 New Orleans Louisiana United States IEEE/CVF 2022 .
Details
| Link 1
Nonlinear Optimization of Light Field Point Cloud
Academic Editor Denis Laurendeau (Hrsg.). Sensors - Open Access Journal (Sensors) 22(3) Seiten 814-829 MDPI 1/2022 .
Details
| Link 1
| Link 2
Fusion Point Pruning for Optimized 2D Object Detection with Radar-Camera Fusion
2022 Proceedings of the. IEEE Winter Conference on Applications of Computer Vision (WACV-2022) January 4-8 Hawaii HI United States IEEE 2022 .
Details
| Link 1
TICaM: A Time-of-flight In-car Cabin Monitoring Dataset
British Machine Vision Conference (Hrsg.). Proceedings of the. British Machine Vision Conference (BMVC-2021) Online BMVA 2021 .
Details
| Link 1
PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image
British Machine Vision Conference. British Machine Vision Conference (BMVC-2021) November 22-25 United Kingdom British Machine Vision Conference 11/2021 .
Details
| Link 1
2021
Deployment of Deep Neural Networks for Object Detection on Edge AI Devices with Runtime Optimization
IEEE (Hrsg.). Proceedings of the IEEE International Conference on Computer Vision Workshops - ERCVAD Workshop on Embedded and Real-World Computer Vision in Autonomous Driving. International Conference on Computer Vision (ICCV-2021) October 11-17 Online/Virtual IEEE 2021 .
Details
| Link 1
Visual SLAM with Graph-Cut Optimized Multi-Plane Reconstruction
IEEE. IEEE International Symposium on Mixed and Augmented Reality (ISMAR-2021) October 4-8 Bari Italy IEEE 2021 .
Details
| Link 1
SEMANTIC SEGMENTATION IN DEPTH DATA : A COMPARATIVE EVALUATION OFIMAGE AND POINT CLOUD BASED METHODS
Proceedings of ICIP. IEEE International Conference on Image Processing (ICIP-2021) 28th IEEE International Conference on Image Processing (IEEE - ICIP) September 19-22 Anchorage, Alaska Alaska United States IEEE 2021 .
Details
| Link 1
Mixed reality applications in urology: Requirements and future potential
Elsevier (Hrsg.). Annals of Medicine and Surgery 66 Seiten 1-6 Elsevier 2021 .
Details
| Link 1
PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN
IEEE. IEEE International Conference on Robotics and Automation (ICRA-2021) May 30-June 5 Xi'an China IEEE 2021 .
Details
| Link 1
An Adversarial Training based Framework for Depth Domain Adaptation
Proceedings of the 16th VISAPP. International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP-2021) 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications February 8-10 online Springer 2021 .
Details
| Link 1
A Survey on Applications of Augmented, Mixed and Virtual Reality for Nature and Environment
Proceedings of the 23rd HCI. International Conference on Human-Computer Interaction (HCII-2021) July 24-29 Online United States Springer 2021 .
Details
SynPo-Net–Accurate and Fast CNN-Based 6DoF Object Pose Estimation Using Synthetic Training
Special Issue Object Tracking and Motion Analysis, Sensors - Open Access Journal (Sensors) 21 Seite 300 MDPI 2021 .
Details
| Link 1
OFFSED: Off-Road Semantic Segmentation Dataset
VISAPP 2021 Proceedings. International Conference on Computer Vision Theory and Applications (VISAPP-2021) 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications February 8-10 Online ISBN TBA SCITEPRESS Digital Library 2021 .
Details
| Link 1
2020
Ghost Target Detection in 3D Radar Data using Point Cloud based Deep Neural Network
International Conference on Pattern Recognition. International Conference on Pattern Recognition (ICPR-2020) January 12-15 Milano Italy IEEE 2021 .
Details
| Link 1
Learning Priors for Augmented Reality Tracking and Scene Understanding
-Thesis Technische Universität Kaiserslautern ISBN 978-3-8439-4555-4 Dr.Hut München 9/2020 .
Details
TGA: Two-level Group Attention for Assembly State Detection
Proceedings of the 19th IEEE ISMAR. IEEE International Symposium on Mixed and Augmented Reality (ISMAR-2020) November 9-13 Recife/Porto de Galinhas Brazil IEEE 2020 .
Details
| Link 1
2019
SlamCraft: Dense Planar RGB Monocular SLAM
Proceedings of. IAPR Conference on Machine Vision Applications (MVA-2019) May 27-31 Tokyo Japan IAPR 2019 .
Details
| Link 1
Deep Multi-State Object Pose Estimation for Augmented Reality Assembly
Proceedings of the 18th IEEE ISMAR. IEEE International Symposium on Mixed and Augmented Reality (ISMAR-2019) October 14-18 Beijing China IEEE 2019 .
Details
| Link 1
Augmented Reality in Physics education: Motion understanding using an Augmented Airtable
EuroVR (Hrsg.). European Association on Virtual and Augmented Reality. EuroVR (EuroVR-2019) October 23-25 Tallin Estonia Springer 2019 .
Details
| Link 1