1. Home
  2. AV Teams
  3. SSMP Team

SSMP Team: Spatial Sensing and Machine Perception

Team SSMP (Spatial Sensing and Machine Perception) focuses on the use of diverse 3D/2D sensing modalities (RGB/Stereo/ToF cameras, Radar, Lidar) to address challenging scene perception problems using Machine Learning/Deep Learning and traditional geometric Computer Vision. Such problems include as Semantic Scene Reconstruction, 6DoF Object Pose Estimation, Unsupervised Anomaly Detection, SLAM, Deep Sensor Fusion. Indicative application areas are Smart building sensing, AI in construction, Autonomous Driving, Industrial Robotics and Augmented Reality.

Projects of SSMP Team

GreifbAR

GreifbAR Project Image
Tangible reality – skilful interaction of user hands and fingers with real tools in mixed reality worlds

The goal of the GreifbAR project is to make mixed reality (MR) worlds, including virtual (VR) and augmented reality (“AR”), tangible and graspable by allowing users to interact with real and virtual objects with their bare hands. Hand accuracy and dexterity is paramount for performing precise tasks in many fields, but the capture of hand-object interaction in current MR systems is woefully inadequate. Current systems rely on hand-held controllers or capture devices that are limited to hand gestures without contact with real objects. GreifbAR solves this limitation by introducing a sensing system that detects both the full hand grip including hand surface and object pose when users interact with real objects or tools. This sensing system will be integrated into a mixed reality training simulator that will be demonstrated in two relevant use cases: industrial assembly and surgical skills training. The usability and applicability as well as the added value for training situations will be thoroughly analysed through user studies.

Partners

NMY – Mixed Reality Communication GmbH Hanauer Landstraße 188, 60314 Frankfurt am Main

Charité – Universitätsmedizin Berlin Charitéplatz 1, 10117 Berlin

Universität Passau vertreten durch den Präsidenten, Prof. Dr. Ulrich Bartosch Innstraße 40, 94032 Passau Wissenschaftliche Leitung: Prof. Dr. Susanne Mayr Lehrstuhl für Psychologie mit Schwerpunkt Mensch – Maschine – Interaktion

Contact

Dr.-Ing. Jason Raphael Rambach

Yongzhi Su, M.Sc.

VIZTA

Vision, Identification with Z-sensing Technologies and key Applications

VIZTA project, coordinated by ST Micrelectronics, aims at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value in several key applications including automotive, security, smart buildings, mobile robotics for smart cities, and industry4.0. The key differentiating 12-inch Silicon sensing technologies developed during VIZTA are:

• Innovative SPAD and lock-in pixel for Time of Flight architecture sensors
• Unprecedent and cost-effective NIR and RGB-Z filters on-chip solutions
• complex RGB+Z pixel architectures for multimodal 2D/3D imaging For short-range sensors : advanced VCSEL sources including wafer-level GaAs optics and associated high speed driver.

These developed differentiating technologies allows the development and validation of innovative 3D imaging sensors products with the following highly integrated prototypes demonstrators:

• High resolution (>77 000 points) time-of-flight ranging sensor module with integrated VCSEL, drivers, filters and optics.
• Very High resolution (VGA min) depth camera sensor with integrated filters and optics.

For Medium and Long range sensing, VIZTA also adresses new LiDAR systems with dedicated sources, optics and sensors Technology developments of sensors and emitters are carried out by leading semiconductor product suppliers (ST Microelectronics, Philips, III-V Lab) with the support of equipment suppliers (Amat, Semilab) and CEA Leti RTO.

VIZTA project also include the developement of 6 demonstrators for key applications including automotive, security, smart buildings, mobile robotics for smart cities, and industry4.0 with a good mix of industrial and academic partners (Ibeo, Veoneer, Ficosa, Beamagine, IEE, DFKI, UPC, Idemia, CEA-List, ISD, BCB, IDE, Eurecat). VIZTA consortium brings together 23 partners from 9 countries in Europe: France, Germany, Spain, Greece, Luxembourg, Latvia, Sweden, Hungary, and United Kingdom.

 

Partners :

Universidad Politecnica Catalunya
Commisariat a l Energie Atomique et aux Energies Alternatives (CEA Paris)
Fundacio Eurecat
STMICROELECTRONICS SA
BCB Informática y Control
Alter Technology TÜV Nord SA
FICOMIRRORS SA
Philips Photonics GmbH
Applied Materials France SARL
SEMILAB FELVEZETO FIZIKAI LABORATORIUM RESZVENYTARSASAG
ELEKTRONIKAS UN DATORZINATNU INSTITUTS
LUMIBIRD
IEE S.A.
IBEO Automotive Systems GmbH
STMICROELECTRONICS RESEARCH & DEVELOPMENT LTD
STMICROELECTRONICS SA
IDEMIA IDENITY & SECURITY FRANCE
Beamagine S.L.
Integrated Systems Development S.A.
VEONEER SWEDEN AB
III-V Lab
STMICROELECTRONICS (ALPS) SAS
STMICROELECTRONICS GRENOBLE 2 SAS

 

Kontakt: Jason Rambach ; Bruno Mirbach
E-Mail: Jason.Rambach@dfki.de
Telefon: +49 631 20575 3740

SSMP News


2023

Workshop on AI and Robotics in Construction at ERF 2023

Dr. Jason Rambach, coordinator of the EU Horizon Project HumanTech co-organized a workshop on “AI and Robotics in Construction” at the European ... Read more

Article in IEEE Robotics and Automation Letter (RA-L) journal

We are happy to announce that our article “OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection” was published in the prestigious IEEE Robotics and Automation Letters (RA-L) Journal. The work is a collaboration of DFKI with the... Read more

Radar Driving Activity Dataset (RaDA) Released

DFKI Augmented Vision recently released the first publicly available UWB Radar Driving Activity Dataset (RaDA), consisting of over 10k data samples from 10 different participants annotated with 6 driving activities. The dataset was recorded in the DFKI driving simulator e... Read more


2022

VIZTA Project successfully concluded after 42 months

The Augmented Vision department of DFKI participated in the VIZTA project, coordinated by ST Microelectronics, aiming at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value ... Read more

DFKI Augmented Vision Researchers win two awards in Object Pose Estimation challenge (BOP Challenge, ECCV 2022)

DFKI Augmented Vision researchers Yongzhi Su, Praveen Nathan and Jason Rambach received their 1st place award in the prestigious BOP Challenge 2022 in the categories Overall Best Segmentation Method and The Best BlenderProc-Trained Segmentation Method.... Read more

Kick-Off for EU Project “HumanTech”

Our Augmented Vision department is the coordinator of the new large European project "HumanTech". The Kick-Off meeting was held on July 20th, 2022, at DFKI in Kaiserslautern. Please read the whole article here: Artificia... Read more

ARTIFICIAL INTELLIGENCE FOR A SAFE AND SUSTAINABLE CONSTRUCTION INDUSTRY

Please check out the article "Artificial intelligence for a safe and sustainable construction industry (dfki.de)" concerning the new EU project HumanTech which is coordinated by Read more

Augmented Vision @CVPR 2022

DFKI Augmented Vision had a strong presence in the recent CVPR 2022 Conference held on June 19th-23rd, 2022, in New Orleans, USA. The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) is the premier annual computer vision event internationally. Homepage: ... Read more

Keynote Presentation by Dr. Jason Rambach in Computer Vision session of the Franco-German Research and Innovation Network event

On June 14th, 2022, Dr. Jason Rambach gave a keynote talk in the Computer Vision session of the  Franco-German Research and Innovation Network event held at the Inria headquarters in Versailles, Paris, France. In the talk, an overview of the current activities of the... Read more

CVPR 2022: Two papers accepted

We are happy to announce that the Augmented Vision group will present two papers in the upcoming CVPR 2022 Conference from June 19th-23rd in New Orleans, USA. The IEEE/CVF Computer Vision and Pattern Recognition Conference (CVPR) is the premier annual computer vision even... Read more


2021

2 Papers accepted at BMVC 2021 Conference

We are happy to announce that the Augmented Vision group will present 2 papers in the upcoming BMVC 2021 Conference, 22-25 November, 2021:

The British Machine Vision Conference (BMVC) is the British Machine Vision Assoc... Read more

DFKI AV – Stellantis Collaboration on Radar-Camera Fusion – 2 publications

DFKI Augmented Vision is working with Stellantis on the topic of Radar-Camera Fusion for Automotive Object Detection using Deep Learning since 2020. The collaboration has already led to two publications, in ICCV 2021 (International Conference on Computer Vision – Read more

VIZTA Project Time-of-Flight Camera Datasets Released

As part of the research activities of DFKI Augmented Vision in the VIZTA project (https://www.vizta-ecsel.eu/), two publicly available datasets have been released and are available for download. TIMo dataset is a ... Read more

XR for nature and environment survey

On July 29th, 2021, Dr. Jason Rambach presented the survey paper “A Survey on Applications of Augmented, Mixed and Virtual Reality for Nature and Environment” at the 23rd Human Computer Interaction Conference Read more

VIZTA Project 24M Review and public summary

DFKI participates in the VIZTA project, coordinated by ST Micrelectronics, aiming  at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value in several key applications in... Read more

Paper accepted at ICIP 2021

We are happy to announce that our paper “SEMANTIC SEGMENTATION IN DEPTH DATA : A COMPARATIVE EVALUATION OF IMAGE AND POINT CLOUD B... Read more

TiCAM Dataset for in-Cabin Monitoring released

As part of the research activities of DFKI Augmented Vision in the VIZTA project (https://www.vizta-ecsel.eu/), we have published the open-source dataset for automotive in-cabin monitoring with a wide-angle time-of-flight depth se... Read more

Paper accepted at ICRA 2021

We are delighted to announce that our paper “Read more

Presentation on Machine Learning and Computer Vision by Dr. Jason Rambach

On March 4th, 2021, Dr. Jason Rambach gave a talk on Machine Learning and Computer Vision at the GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit) workshop o... Read more

VIZTA project: 18-month public project summary released

DFKI participates in the VIZTA project, coordinated by ST Micrelectronics, aiming  at developing innovative technologies in the field of optical sensors and laser sources for short to long-range 3D-imaging and to demonstrate their value in several key applications in... Read more

3 Papers accepted at VISAPP 2021

We are excited to announce that the Augmented Vision group will present 3 papers in the upcoming VISAPP 2021 Conference, February 8th-10th, 2021:

The International Conference on Computer Vision Theory and Applications ... Read more

Article at MDPI Sensors journal

We are happy to announce that our paper “SynPo-Net–Accurate and Fast CNN-Based 6DoF Object Pose Estimation Using Synthetic Training” has been accepted for publication at the MDPI Sensors journal, Special Issue Object Tracking and Motion Analysis. Sensors (ISSN 14... Read more


2020

Four papers accepted at WACV 2021

The Winter Conference on Applications of Computer Vision (WACV 2021) is IEEE’s and the PAMI-TC’s premier meeting on applications of computer vision. With its high quality and low cost, it provides an exceptional value for students... Read more

Head of SSMP Team

Member Position E-Mail Phone
Profile picture of Jason Raphael Rambach
Dr. Jason Raphael Rambach Senior Researcher jason_raphael.rambach@dfki.de +49 631 20575-3740

Members of SSMP Team

Member Position E-Mail Phone
Profile picture of Bruno Mirbach
Dr. Bruno Mirbach Senior Researcher Bruno_Walter.Mirbach@dfki.uni-kl.de +49 631 20575 3511
Profile picture of Henri Hoyez
Henri Hoyez Researcher Henri.Hoyez@dfki.de
Profile picture of Jigyasa Singh Katrolia
Jigyasa Singh Katrolia Researcher Jigyasa_Singh.Katrolia@dfki.de +49 631 20575 3528
Profile picture of Lukas Stefan Staecker
Lukas Stefan Staecker Researcher Lukas_Stefan.Staecker@dfki.uni-kl.de
Profile picture of Mahdi Chamseddine
Mahdi Chamseddine Researcher Mahdi.Chamseddine@dfki.de +49 631 20575-3527
Profile picture of Pascal Schneider
Pascal Schneider Researcher Pascal.Schneider@dfki.uni-kl.de
Profile picture of Praveen Nathan
Praveen Nathan Researcher praveen.annamalai_nathan@dfki.de
Profile picture of Suresh Guttikonda
Suresh Guttikonda Researcher Suresh.Guttikonda@dfki.de +49 631 20575 3580
Profile picture of Yongzhi Su
Yongzhi Su Researcher yongzhi.su@dfki.de +49 631 20575-3416
Profile picture of Yuriy Anisimov
Yuriy Anisimov Researcher yuriy.anisimov@dfki.de +49 631 20575-3750

Publications of SSMP Team

2023

ONTOLOGY-BASED SEMANTIC LABELING FOR RGB-D AND POINT CLOUD DATASETS
authors Fabian Kaufmann, Mahdi Chamseddine, Suresh Guttikonda, Christian Glock, Didier Stricker, Jason Raphael Rambach,
Computing in Construction. European Conference on Computing in Construction (EC3-2023), July 10-12, Crete, Greece, EC3, 2023.
Details | Link 1

OPA-3D: Occlusion-Aware Pixel-Wise Aggregation for Monocular 3D Object Detection
authors Yongzhi Su, Yan Di, Fabian Manhardt, Guangyao Zhai, Jason Raphael Rambach, Benjamin Busam, Didier Stricker, Federico Tombari,
IEEE Robotics and Automation Letters (RA-L), Vol. 8, Pages 1327-1334, IEEE, 3/2023.
Details | Link 1

Principles of Object Tracking and Mapping
authors Didier Stricker, Jason Raphael Rambach, Alain Pagani,
Andrew Yeh Ching Nee; Soh Khim Ong. Springer Handbook of Augmented Reality. Pages 53-84, Springer Handbooks, ISBN 978-3-030-67821-0, Springer, Switzerland, 1/2023.
Details | Link 1

Driving Activity Recognition Using UWB Radar and Deep Neural Networks
authors Iuliia Brishtel, Stephan Krauß, Mahdi Chamseddine, Jason Raphael Rambach, Didier Stricker,
Sensors - Open Access Journal (Sensors) 23 2 Seiten 1-15 MDPI 1/2023 .
Details | Link 1 | Link 2

2022

Unsupervised Image-to-Image Translation: A Review
authors Henri Hoyez, Cedric Schockaert, Jason Raphael Rambach, Bruno Mirbach, Didier Stricker,
Sensors - Open Access Journal (Sensors) 2022, 22, 8540 MDPI 2022 .
Details | Link 1

Editorial – Advanced Scene Perception for Augmented Reality
authors Jason Raphael Rambach, Didier Stricker,
Journal of Imaging (MDPI J) Special Issue Advanced Scene Perception for Augmented Reality MDPI 10/2022 .
Details | Link 1

Unsupervised Anomaly Detection from Time-of-Flight Depth Images
authors Pascal Schneider, Jason Raphael Rambach, Bruno Mirbach, Didier Stricker,
CVPR Workshop on Perception Beyond the Visible Spectrum. CVPR Workshop on Perception Beyond the Visible Spectrum (PBVS-2022) befindet sich International Conference on Computer Vision and Pattern Recognition June 19-20 United States IEEE Computer Society 2022 .
Details | Link 1

TIMo—A Dataset for Indoor Building Monitoring with a Time-of-Flight Camera
authors Pascal Schneider, Yuriy Anisimov, Raisul Islam, Bruno Mirbach, Jason Raphael Rambach, Didier Stricker, Frédéric Grandidier,
Sensors - Open Access Journal (Sensors) 22 11 Seiten 3992-4005 MPDI Basel 5/2022 .
Details | Link 1

ZebraPose: Coarse to Fine Surface Encoding for 6DoF Object Pose Estimation
authors Yongzhi Su, Mahdi Saleh, Torben Fetzer, Jason Raphael Rambach, Nassir Navab, Benjamin Busam, Didier Stricker, Federico Tombari,
IEEE/CVF. International Conference on Computer Vision and Pattern Recognition (CVPR-2022) June 19-24 New Orleans Louisiana United States IEEE/CVF 2022 .
Details | Link 1

Nonlinear Optimization of Light Field Point Cloud
authors Yuriy Anisimov, Jason Raphael Rambach, Didier Stricker,
Academic Editor Denis Laurendeau (Hrsg.). Sensors - Open Access Journal (Sensors) 22(3) Seiten 814-829 MDPI 1/2022 .
Details | Link 1 | Link 2

Fusion Point Pruning for Optimized 2D Object Detection with Radar-Camera Fusion
authors Lukas Stefan Stäcker, Philipp Heidenreich, Jason Raphael Rambach, Didier Stricker,
2022 Proceedings of the. IEEE Winter Conference on Applications of Computer Vision (WACV-2022) January 4-8 Hawaii HI United States IEEE 2022 .
Details | Link 1

TICaM: A Time-of-flight In-car Cabin Monitoring Dataset
authors Jigyasa Singh Katrolia, Ahmed Elsherif, Hartmut Feld, Bruno Mirbach, Jason Raphael Rambach, Didier Stricker,
British Machine Vision Conference (Hrsg.). Proceedings of the. British Machine Vision Conference (BMVC-2021) Online BMVA 2021 .
Details | Link 1

PlaneRecNet: Multi-Task Learning with Cross-Task Consistency for Piece-Wise Plane Detection and Reconstruction from a Single RGB Image
authors Yaxu Xie, Fangwen Shu, Alain Pagani, Jason Raphael Rambach, Didier Stricker,
British Machine Vision Conference. British Machine Vision Conference (BMVC-2021) November 22-25 United Kingdom British Machine Vision Conference 11/2021 .
Details | Link 1

2021

Deployment of Deep Neural Networks for Object Detection on Edge AI Devices with Runtime Optimization
authors Lukas Stefan Stäcker, Juncong Fei, Philipp Heidenreich, Frank Bonarens, Jason Raphael Rambach, Didier Stricker, Christoph Stiller,
IEEE (Hrsg.). Proceedings of the IEEE International Conference on Computer Vision Workshops - ERCVAD Workshop on Embedded and Real-World Computer Vision in Autonomous Driving. International Conference on Computer Vision (ICCV-2021) October 11-17 Online/Virtual IEEE 2021 .
Details | Link 1

Visual SLAM with Graph-Cut Optimized Multi-Plane Reconstruction
authors Fangwen Shu, Yaxu Xie, Jason Raphael Rambach, Alain Pagani, Didier Stricker,
IEEE. IEEE International Symposium on Mixed and Augmented Reality (ISMAR-2021) October 4-8 Bari Italy IEEE 2021 .
Details | Link 1

SEMANTIC SEGMENTATION IN DEPTH DATA : A COMPARATIVE EVALUATION OFIMAGE AND POINT CLOUD BASED METHODS
authors Jigyasa Singh Katrolia, Lars Kraemer, Jason Raphael Rambach, Bruno Mirbach, Didier Stricker,
Proceedings of ICIP. IEEE International Conference on Image Processing (ICIP-2021) 28th IEEE International Conference on Image Processing (IEEE - ICIP) September 19-22 Anchorage, Alaska Alaska United States IEEE 2021 .
Details | Link 1

Mixed reality applications in urology: Requirements and future potential
authors Gerd Reis, Mehmet Yilmaz, Jason Raphael Rambach, Alain Pagani, Rodrigo Suarez-Ibarrola, Arkadiusz Miernik, Paul Lesur, Nareg Minaskan Karabid,
Elsevier (Hrsg.). Annals of Medicine and Surgery 66 Seiten 1-6 Elsevier 2021 .
Details | Link 1

PlaneSegNet: Fast and Robust Plane Estimation Using a Single-stage Instance Segmentation CNN
authors Yaxu Xie, Jason Raphael Rambach, Fangwen Shu, Didier Stricker,
IEEE. IEEE International Conference on Robotics and Automation (ICRA-2021) May 30-June 5 Xi'an China IEEE 2021 .
Details | Link 1

An Adversarial Training based Framework for Depth Domain Adaptation
authors Jigyasa Singh Katrolia, Lars Krämer, Jason Raphael Rambach, Bruno Mirbach, Didier Stricker,, , , and
Proceedings of the 16th VISAPP. International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP-2021) 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications February 8-10 online Springer 2021 .
Details | Link 1

A Survey on Applications of Augmented, Mixed and Virtual Reality for Nature and Environment
authors Jason Raphael Rambach, Gergana Lilligreen, Alexander Schäfer, Ramya Bankanal, Alexander Wiebel, Didier Stricker,, and
Proceedings of the 23rd HCI. International Conference on Human-Computer Interaction (HCII-2021) July 24-29 Online United States Springer 2021 .
Details

SynPo-Net–Accurate and Fast CNN-Based 6DoF Object Pose Estimation Using Synthetic Training
authors Yongzhi Su, Jason Raphael Rambach, Alain Pagani, Didier Stricker,, , and
Special Issue Object Tracking and Motion Analysis, Sensors - Open Access Journal (Sensors) 21 Seite 300 MDPI 2021 .
Details | Link 1

OFFSED: Off-Road Semantic Segmentation Dataset
authors Peter Neigel, Jason Raphael Rambach, Didier Stricker,, and
VISAPP 2021 Proceedings. International Conference on Computer Vision Theory and Applications (VISAPP-2021) 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications February 8-10 Online ISBN TBA SCITEPRESS Digital Library 2021 .
Details | Link 1

2020

Ghost Target Detection in 3D Radar Data using Point Cloud based Deep Neural Network
authors Mahdi Chamseddine, Jason Raphael Rambach, Oliver Wasenmüller, Didier Stricker,, and
International Conference on Pattern Recognition. International Conference on Pattern Recognition (ICPR-2020) January 12-15 Milano Italy IEEE 2021 .
Details | Link 1

Learning Priors for Augmented Reality Tracking and Scene Understanding
authors Jason Raphael Rambach,
-Thesis Technische Universität Kaiserslautern ISBN 978-3-8439-4555-4 Dr.Hut München 9/2020 .
Details

TGA: Two-level Group Attention for Assembly State Detection
authors Hangfan Liu, Yongzhi Su, Jason Raphael Rambach, Alain Pagani, Didier Stricker,, , and
Proceedings of the 19th IEEE ISMAR. IEEE International Symposium on Mixed and Augmented Reality (ISMAR-2020) November 9-13 Recife/Porto de Galinhas Brazil IEEE 2020 .
Details | Link 1

2019

SlamCraft: Dense Planar RGB Monocular SLAM
authors Jason Raphael Rambach, Paul Lesur, Alain Pagani, Didier Stricker,, , and
Proceedings of. IAPR Conference on Machine Vision Applications (MVA-2019) May 27-31 Tokyo Japan IAPR 2019 .
Details | Link 1

Deep Multi-State Object Pose Estimation for Augmented Reality Assembly
authors Yongzhi Su, Jason Raphael Rambach, Nareg Minaskan Karabid, Paul Lesur, Alain Pagani, Didier Stricker,, , , , and
Proceedings of the 18th IEEE ISMAR. IEEE International Symposium on Mixed and Augmented Reality (ISMAR-2019) October 14-18 Beijing China IEEE 2019 .
Details | Link 1

Augmented Reality in Physics education: Motion understanding using an Augmented Airtable
authors Nareg Minaskan Karabid, Jason Raphael Rambach, Alain Pagani, Didier Stricker,, , and
EuroVR (Hrsg.). European Association on Virtual and Augmented Reality. EuroVR (EuroVR-2019) October 23-25 Tallin Estonia Springer 2019 .
Details | Link 1