Multi-Modal Data Processing

Abstract: Many modern datasets are heterogeneous, containing data from different sources and with different representations. Making sense of these combined data is a challenging task, but it is increasingly important for both academia and industry. In this work, we look at sparsity-based approaches to extract information from multi-modal data. We start with the problem of separating the x-rays of the paintings in the door panels of the Ghent Altarpiece, a 15th century art work by Van Eyck currently under restoration. Our method uses the visual images to aid the x-ray separation process and outperforms prior state-of-the-art methods, such as morphological component analysis. We will then see that the model used in our approach is quite general and can be applied to other signal processing tasks that use multi-modal data, or even same modality data in the form of prior information. Some instances of that model allow establishing rigorous performance guarantees. One example is the problem of integrating prior knowledge into sparse reconstruction schemes. We overview some of these results and point out their importance towards the goal of, for example, reducing the time of MRI scans.