- Test-Time Generalization of Vision-Language Models
Pre-trained vision-language models (VLMs), such as CLIP [1], have showcased exceptional performance in zero-shot classification tasks, eliminating the need for additional training on specific downstream tasks. However, when faced with domain shifts, these models often experience a significant drop in performance.
In this project, we aim to explore test-time generalization techniques for VLMs to address the performance degradation caused by domain shifts. These techniques leverage unlabeled test samples to adapt models dynamically to diverse data distributions, making them highly valuable for real-world applications. Among these approaches, methods such as test-time prompt tuning [2, 3] and prototype-based learning [4] have been introduced to enhance the adaptability of VLMs during test-time adaptation. The proposed approach will be evaluated on a variety of vision benchmarks, with a particular emphasis on histopathology image classification.
Mandatory Requirements:
– Proficient Python programming skills.
– Experience with Deep Learning libraries such as PyTorch.
– Familiarity with Transformer architectures and Vision-Language Models (e.g., CLIP).
References:
[1] Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al. “Learning transferable visual models from natural language supervision.” In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
[2] Shu, Manli, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. “Test-time prompt tuning for zero-shot generalization in vision-language models.” Advances in Neural Information Processing Systems 35 (2022): 14274-14289.
[3] Hakim, Gustavo Adolfo Vargas, David Osowiechi, Mehrdad Noori, Milad Cheraghalikhani, Ali Bahri, Moslem Yazdanpanah, Ismail Ben Ayed, and Christian Desrosiers. “CLIPArTT: Light-weight Adaptation of CLIP to New Domains at Test Time.” arXiv preprint arXiv:2405.00754 (2024).
[4] Zhang, Ce, Simon Stepputtis, Katia Sycara, and Yaqi Xie. “Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models.” arXiv preprint arXiv:2410.12790 (2024).
Assistant: Dr Behzad Bozorgtabar ([email protected])
Supervisor: Prof. Jean-Philippe Thiran
2. Diffusion Models for Magnetic Resonance Imaging Reconstruction (Master thesis in industry – PDMe)
Magnetic Resonance Imaging (MRI) is a well-established medical modality for imaging the human brain and body, capable of producing high resolution images with many different soft tissue contrasts without ionizing radiation. However, as MRI scans are slow due to the physics of acquisition, clinical scans are accelerated by using only incomplete measurements incomplete measurements of the Fourier spectrum (or k-space) of the desired image. Consequently, algorithms for reconstructing MRI images with incomplete measurement data have been and are currently of great importance for clinical imaging.
As with many computational imaging tasks, machine learning models are now the state of the art for MRI reconstruction, where deep neural networks are trained to act as priors within optimization algorithms using datasets where the acquisitions have been fully sampled (see Figure below). Recently, diffusion models have emerged as an extremely effective way to encode image priors, being used successfully for image generation, image restoration, and image reconstruction.
OBJECTIVES: In this project, we would like to:
- • Train and compare several baseline, existing 2D diffusion model reconstructions to our current in-house DL reconstruction method.
- • Investigate new diffusion model architectures and reconstruction frameworks and compare these to the baselines.
- • Assess the feasibility of a 3D diffusion reconstruction for MRI (never assessed so far to our knowledge).
Available data:
Several hundred 2D and 3D MRI scans with full measurement data.
Computing resources:
3 compute servers with 6 GPUs (Quadro RTX 8000 and A6000, each with 40-50GB of RAM)
REQUIRED SKILLS
Background in computer science, mathematics, electrical engineering, physics, biomedical engineering, or similar. Previous programming experience in Python, in particular using Pytorch. Good knowledge of machine learning, ideally with some experience having used diffusion models for any application in imaging.
COMPANY INFORMATION
Siemens Healthineers International AG
ACIT – EPFL QI-E, 1015 Lausanne, Switzerland.
CONTACT INFORMATION
Dr Tom Hilbert, email: [email protected].
Dr Thomas Yu, email: [email protected].
Dr Gian Franco Piredda, email: [email protected].
3. Synthetic data for fetal and neonatal segmentation – Collaboration with UNIL-CHUV
Automated methods for medical image segmentations suffer heavily from domain shifts: the datasets used are typically small and heterogeneous, which poses a challenge for generalization of learning-based methods. This problem is even stronger in fetal and neonatal MRI, where the brain anatomy undergoes rapid and large changes. Recent works, based on synthetic data and domain randomization [1,2] are promising avenues for building robust models to tackle this challenge.
In this project, we will aim at building a synthetic data generator for fetal and neonatal images, integrating specific artifacts [3]. Based on this generator, we will train a segmentation model that will then be tested on various clinical datasets. If time allows, we will also explore how these models could be fine-tuned on different tasks to maximize their performance and re-usability [4]. This project will provide valuable input to an ongoing research effort to characterize fetal and neonatal neurodevelopment using low-field 0.55T MRI scanners, which are expected to be much more accessible to low-income countries than conventional MRI scanners.
The student will learn to : 1) handle and process 3D clinical fetal and neonatal MR images, 2) learn to use state-of-the-art domain randomization techniques, 3) explore how these models could be fine-tuned to maximize their performance in related tasks. The ideal candidate for this project should have solid programming skills with proficiency in PyTorch and a strong foundation in image processing and deep learning.
References:
[1] Billot, Benjamin, et al. “SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining.” Medical image analysis 86 (2023): 102789.
[2] Zalevskyi, Vladyslav, et al. “Maximizing domain generalization in fetal brain tissue segmentation: the role of synthetic data generation, intensity clustering and real image fine-tuning.” arXiv preprint arXiv:2411.06842 (2024).
[3] Pérez-García, Fernando, Rachel Sparks, and Sébastien Ourselin. “TorchIO: a Python library for efficient loading, preprocessing, augmentation and patch-based sampling of medical images in deep learning.” Computer methods and programs in biomedicine 208 (2021): 106236.
[4] Wortsman, Mitchell, et al. “Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.” International conference on machine learning. PMLR, 2022.
Supervisor:
Prof. Jean-Philippe Thiran (EPFL-LTS5)
Co-supervisors:
- Dr. Thomas Sanchez CHUV-UNIL – Medical Image Analysis Lab ([email protected])
- Dr. Meritxell Bach Cuadra CHUV-UNIL – Medical Image Analysis Lab ([email protected])
Spherical Deconvolution Algorithms for Intra-Voxel Fiber Estimation and Brain Connectivity Mapping
The LTS5 Diffusion group focuses on brain tissue microstructure and structural connectivity –estimated by diffusion Magnetic Resonance Imaging (dMRI) data, with a particular focus on the reconstruction of the nerve fiber orientation distribution function (ODF) per voxel (see the figure below). This information is important for the reconstruction of the brain’s white matter by using fiber tracking algorithms (see ref [1]).
We have implemented various novel reconstruction algorithms (e.g., see refs. [2-5]) and we plan to develop a new generation of methods using Machine Learning techniques. The goals of this project are: (1) create a large database of fiber ODFs and corresponding dMRI signals, (2) Design, train, and optimize a neural network using this dataset, (3) predict the fiber ODFs from new dMRI data, and (4) compare the implemented algorithm with state-of-the-art techniques using both synthetic and real dMRI data acquired from human brains. The results will be published in international conferences and relevant journals.
Requirements: The project will be implemented in Python, so good knowledge is required. This project is ideal for a computer scientist, mathematician, physicist, or engineer interested in medical imaging, machine learning, signal processing, and optimization.
References:
[1] https://www.sciencedirect.com/science/article/abs/pii/S1053811914003541
[2] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4607500
[3] https://www.sciencedirect.com/science/article/abs/pii/S1053811918307699
[4] https://onlinelibrary.wiley.com/doi/10.1002/mrm.21917
Supervisors: Dr. Erick J. Canales-Rodríguez ([email protected]), Dr. Gabriel Girard ([email protected]), and Prof. Jean-Philippe Thiran
Myelin Water Imaging Using T2 Relaxometry
Myelin is a lipid-rich substance that surrounds the axons in the brain, which is essential for the proper functioning of the nervous system. Myelin water imaging is a magnetic resonance imaging (MRI) method that can be used to quantify and visualize myelination in the brain and spinal cord in vivo. The signal coming from the MRI machine (using a multi-echo T2 relaxometry sequence) can be decomposed into components, including that originated by water molecules trapped between the lipid bilayers of myelin. The correct estimation of this component provides a myelin-specific MRI biomarker to monitor brain changes in cerebral white matter. Myelin quantification has important implications for understanding various neurodegenerative diseases, including multiple sclerosis.
We are looking for a motivated student to (1) learn about the MRI and signal processing theory behind this modality, (2) improve the current estimation methods, and (3) test and compare the new results with the current methods and histological measurements.
The project builds on top of previous cutting-edge research carried out in our lab (for more details see our multi-component T2 reconstruction toolbox and related references: https://github.com/ejcanalesr/multicomponent-T2-toolbox). The results will be published in international conferences and relevant journals.
Requirements: The project will be implemented in Python, so good knowledge is required. This project is ideal for a computer scientist, mathematician, physicist, or engineer interested in medical imaging, optimization, and signal processing.
Supervisors: Dr. Erick J. Canales-Rodríguez ([email protected]) and Prof. Jean-Philippe Thiran
Deep learning based shape analysis of cardiac biventricular meshes – collaboration with CHUV
The goal of the project is to make the student familiar with the current trends in medical cardiac imaging analysis. Cardiac motion and shape analysis have proved to be useful to characterize differences in clinical diseases [1,2]. However, the pipeline to obtain suitable meshes from cardiac magnetic resonance images is relatively complex and the displacement estimation relies on single image modalities with a single point of view. Recent work focused on the integration of multiple cardiac image modalities covering different points of view to jointly predict mesh displacements throughout the cardiac cycle [3]. Nevertheless, the exploration of how useful mesh descriptors can be in large-scale datasets remains relatively unexplored [2]. In this project, the student will use state-of-the-art deep learning approaches based on differential geometry and graph neural networks to explore the potential of shape descriptors to stratify subjects by clinical diagnosis in a large-scale cohort, the UK Biobank, containing thousands of cardiac images.
Therefore, the goals are 1) apply existing meshing methods to reliably obtain biventricular meshes [4,5] from multi-structure segmentations obtained from previous works based on convolutional neural networks (CNN) [6,7], 2) generate a statistical shape atlas to study the main modes of variation [8] and 3) explore the use of state-of-the art shape analysis tools to characterize the cardiac shape of each subject [9]
The project will provide valuable input to an ongoing research effort between Lausanne and Geneva in integrative characterisation of heart failure, and therefore has the potential to contribute to advancing medical science and ultimately benefit patients with cardiac pathologies.
References:
[1]: Mansi, T., et al..: A statistical model for quantification and prediction of cardiac remodelling: Application to Tetralogy of Fallot. IEEE Trans Med Imaging
[2]: Bello, G. A.,et al. (2019). Deep-learning cardiac motion analysis for human survival prediction. Nature Machine Intelligence
[3]: Meng, Q., Bai, W., Liu, T., O’Regan, D. P., & Rueckert, D. (2022). Mesh-Based 3D Motion Tracking in Cardiac MRI Using Deep Learning, MICCAI 2022
[4]: Wickramasinghe, U., Remelli, E., Knott, G., & Fua, P. (2020). Voxel2Mesh: 3D Mesh Model Generation from Volumetric Data. MICCAI 2020
[5]: William E. Lorensen and Harvey E. Cline. 1987. Marching cubes: A high resolution 3D surface construction algorithm. SIGGRAPH Comput. Graph. 21, 4 (July 1987)
[6]: Bai, W., et al. (2018). Automated cardiovascular magnetic resonance image analysis with fully convolutional networks. Information and Computing Sciences. Artificial Intelligence and Image Processing. Journal of Cardiovascular Magnetic Resonance, 20(1).
[7]: Byrne, N., Clough, J. R., Valverde, I., Montana, G., & King, A. P. (2022). A persistent homology-based topological loss for CNN-based multi-class segmentation of CMR. IEEE Transactions on Medical Imaging.
[8]: Bai, W., et al. (2015). A bi-ventricular cardiac atlas built from 1000+ high resolution MR images of healthy subjects and an analysis of shape and motion. Medical Image Analysis
[9]:Sharp, N., Attaiki, S., Crane, K., & Ovsjanikov, M. (2022). DiffusionNet: Discretization Agnostic Learning on Surfaces. ACM Transactions on Graphics, 41(3), 1–16.
Requirements: The project will be implemented in Python. Good knowledge of Pytorch as well as familiarity with deep learning are desirable.
Supervisor:
Prof. Jean-Philippe Thiran (EPFL-LTS5)
Co-supervisors:
Dr. Jaume Banus Cobo CHUV-Translational Machine Learning Lab ([email protected])
Dr. Jonas Richiardi CHUV-Translational Machine Learning Lab ([email protected])
High-resolution quantitative susceptibility mapping with 7T MRI
Using dedicated image acquisition and processing techniques, MRI allows mapping the distribution of magnetic susceptibility in living tissues, non-invasively and without injected contrasts. Magnetic susceptibility is currently under research for numerous clinical applications such as the detection of microbleeds, calcifications and abnormal iron metabolism, in the brain as well as other organs.
We work on improving quantitative susceptibility mapping (QSM) approaches at 7 Tesla, to generate exquisitely detailed, sub-millimeter resolution images. To achieve this goal, the topics of interest include the mitigation of interference from motion, the reduction of effects from blood vessels and blood flow, and the improvement of QSM reconstruction techniques based on acquired field maps (ill-posed inverse problem). The solutions include cutting-edge approaches in model-based image processing and/or deep learning.
Within this range of topics, diverse MSc projects can be planned and flexibly adapted to the student’s background and interests. Motivated students are encouraged to contact us to discuss available projects. These projects are held in collaboration with CSEM.
Ideal requirements:
- Interest in brain imaging
- Good knowledge in Python
- Knowledge of image processing fundamentals
- Fluent in written and spoken English
Supervisors:
Dr. João Jorge – [email protected] – CSEM
Dr. Meritxell Bach Cuadra – [email protected] – CIBM SP CHUV-UNIL
Prof. Jean-Philippe Thiran – [email protected]
Multimodal imaging of brain function at ultra-high spatial & temporal resolution with combined EEG-fMRI at 7T
Electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) are valuable brain imaging tools that can detect, respectively, electrical and vascular changes that occur during brain function, non-invasively. We work on the combination of these techniques at a magnetic field of 7T, where fMRI has strong boosts in sensitivity, enabling unprecedented levels of spatial and temporal specificity. Within this domain, several lines of research are open for study :
– Methodology-oriented: Both EEG and fMRI can be affected by important degradation effects when acquired together, especially at 7T. Dedicated improvements at the level of the signal acquisition and/or signal processing and analysis are vital to obtain high-quality data.
– Application-oriented: Our unique, optimized datasets allow studying subtle features of brain function such as thalamocortical interactions and cortical layer-specific activity, which remain largely unexplored. This can be investigated using cutting-edge signal processing and analysis techniques, to be adapted to these unique datasets.
Within this range of topics, MSc projects can be planned and flexibly adapted to the student’s background and interests. Motivated students are encouraged to contact us to discuss available projects. These projects are held in collaboration with CSEM.
Ideal requirements:
- Interest in brain imaging
- Good knowledge in Python
- Knowledge of signal and/or image processing fundamentals
- Fluent in written and spoken English
Supervisors:
Dr. João Jorge – [email protected] – CSEM
Dr. Meritxell Bach Cuadra – [email protected] – CIBM SP CHUV-UNIL
Prof. Jean-Philippe Thiran – [email protected]
Explainable AI for the Detection of Rim Lesions in 3D MRI Scans
This project aims to contribute to the development of artificial intelligence-aided (AI-aided) tools for the prognosis of multiple sclerosis (MS) disease in clinical practice. The focus of this project is on the detection of rim lesions in MS, which are specific patterns of brain lesions visible on magnetic resonance imaging (MRI) scans [1]. Rim lesions are believed to indicate ongoing inflammation and active demyelination, particularly in the outer edges of pre-existing lesions.
The current manual process of detecting rim lesions in MRI scans is time-consuming and requires significant expertise. Therefore, automating this task through deep learning (DL) techniques, specifically 3D semantic segmentation, appears to be a promising solution [2]. However, due to the inherent noise in ground truth segmentation masks for both lesion detection and delineation, employing weakly supervised segmentation (WSS) methods becomes an attractive alternative. In this project, we will explore the use of saliency detection methods for weakly supervised segmentation purposes [3]. Thus, the student project will encompass two main stages: 1. Creating a DL model for a preliminary regression task of the number of lesions in a scan. 2. Applying Explainable AI (XAI) techniques, particularly saliency maps [4], for the detection of rim lesions.
Requirements for this project include experience in deep learning with Python (PyTorch) and experience in image/signal processing.
The project will be supervised by Prof. Jean-Philippe Thiran (EPFL), with co-supervision from Nataliia Molchanova ([email protected]) (CHUV, HESSO) and Meritxell Bach Cuadra ([email protected]) (CIBM).
References:
[1] https://onlinelibrary.wiley.com/doi/pdf/10.1002/ana.25877? casa_token=j0AshM4gBNgAAAAA:3pYQW2ovUNtxSBoWZLaQklEHmsU1muopaB4WirKr5zYaTsf NioNkySneboE-VJF91mUkcjvnWtFMDgUnjg
[2] https://www.sciencedirect.com/science/article/pii/S2213158220301728
[3] https://openaccess.thecvf.com/content_ICCV_2019/papers/ Zeng_Joint_Learning_of_Saliency_Detection_and_Weakly_Supervised_Semantic_Segmentation_I CCV_2019_paper.pdf
[4] https://pubs.rsna.org/doi/pdf/10.1148/ryai.2021200267