Foundation models & multimodal remote sensing

Remote sensing data come in a variety of formats, acquired by sensors operating at different spatial scales and through distinct physical principles. This diversity enables us to observe the Earth from multiple perspectives — from color and spectral composition to 3D geometry. Moreover, different perspectives, matched by geographical coordinates, also provide us with different points of view of the same objects. For example, the same building can be observed by a satellite in orbit (showing its roof) or by a ground-based sensor mounted on a car (showing its facade). When co-registered in geographic space, these heterogeneous observations offer complementary views of the same environment.

Leveraging these multiple data sources concurrently allows for a far richer understanding of environmental processes than any single modality could provide. By developing methodologies to align sensor data acquired from different signals and perspectives, we can create richer semantic spaces that enable advanced search and analysis across large spatial datasets.

Today, foundation models are transforming multimodal remote sensing by providing a common representational backbone that bridges across sensor types and data modalities. Pretrained on massive and diverse geospatial data, these models can generalize to new sensors and regions with minimal supervision. In our lab, we explore how these large-scale multimodal models can enhance transferability between heterogeneous data sources and open new pathways for scalable, knowledge-guided Earth observation.

Papers