Model Based Deep Learning for Multi-sensor 3D Imaging

This project aims to design new computational algorithms at the interface of deep learning, statistical modelling, and optimisation to improve 3D imaging using multi-sensor data.

For cutting-edge imaging applications, the granularity extends to particle levels, encompassing photons and electrons. This precision finds its application in a gamut of systems, from advanced 3D single-photon Lidar imaging for autonomous navigation to electron energy loss spectroscopy and microscopy. Despite recent advances, current systems can still be optimized to enhance the spatial/temporal resolution and imaging quality in low particle conditions (e.g. low light imaging), hence enabling object recognition and 3D imaging fast varying environments at kHz rates.

In this project, we will propose new algorithms for multimodal imaging to enable 3D imaging at unprecedented scales: km range, kHz frame rates and Mega-pixels. Our image processing group will work closely with system design groups in Heriot-Watt to combine multi-sensor data acquired at high frame rates using a 3D single-photon Lidar system and other sensing modalities such as a passive optical imaging system, fast event-based imaging systems or a radar. This combination aims to reduce the noise affecting the 3D images (due to imaging through fog, rain), and to improve the spatial resolution of 3D Lidar videos thus enabling targets reconstruction or higher level information extraction such as objects recognition and human pose estimation (see [1]). Current solutions developed by our group show promising results at high frame rates (acquisition and processing at 500 frames per second in [2,3]) and the student will generalize them to enhance the spatial resolution (a necessity for accurate objects recognition) and/or temporal resolution (a requirement for rapidly changing environments). A focus will be on the combination of statistical Bayesian models [4], optimization algorithms and state-of-the-art deep learning methods to solve these challenging inverse problems. In particular, the candidate will investigate the design of efficient networks (eg using unrolling [5], or plug-and-play approaches), the development of generative models for continuous 3D/4D scene representation and the use of knowledge distillation and burst imaging for multimodal data. Multi-sensor data is already available to test proposed computational solutions, but we can acquire new data if needed.

The candidate will closely collaborate with world-leading computational imaging groups in Heriot-Watt University (single-photon group, Quantum Optics and Computational Imaging), Edinburgh University and the wider community via the EPSRC Quantum Technology Hub in Quantum Imaging (Quantic). The research will be conducted in collaboration with industrials partners: STMicroelectronics and Leonardo. Outstanding students have the opportunity to pursue further research through a funded Ph.D. program.

Please contact Dr. Halimi for more information regarding the project. More information regarding the group can be accessed in: https://sites.google.com/site/abderrahimhalimi/home

Software Needs and Skills:
Deep learning, Statistical signal and Image processing, Bayesian methods, optimization.

Matlab, Python, C/C++.

[1] A. Ruget, M. Tyler, G. Mora-Martín, S. Scholes, F. Zhu, I. Gyongy, B. Hearn, S. McLaughlin, A. Halimi, J. Leach, "Pixels2Pose: Super-Resolution Time-of-Flight Imaging for 3D Pose Estimation", Science Advances, 2022
[2] S. Plosz, A. Maccarone, S. McLaughlin, G. S. Buller, A. Halimi, "Real-Time Reconstruction of 3D Videos from Single-Photon LiDaR Data in the Presence of Obscurants", IEEE-TCI, vol. 9, p 106-119, Feb. 2023.
[3] I. Gyongy, S. W. Hutchings, A. Halimi, M. Tyler, S. Chan, F. Zhu, S. McLaughlin, R. K. Henderson, and J. Leach, "High-speed 3D sensing via hybrid-mode imaging and guided upsampling," Optica, Vol. 7, Issue 10, Sept. 2020.
[4] A. Halimi, A. Maccarone, R. Lamb, G. Buller, S. McLaughlin, "Robust and Guided Bayesian Reconstruction of Single-Photon 3D Lidar Data: Application to Multispectral and Underwater Imaging," IEEE-TCI, 2021.
[5] J. Koo, A. Halimi, S. McLaughlin, "A Bayesian Based Deep Unrolling Algorithm for Single-Photon Lidar Systems", IEEE-JSTSP, 2022.