Open Student Projects

Evaluation Framework for Cross-Domain Visual Retrieval in Cultural Heritage Image Collections

Semester project / Master thesis (DH/IC)

Supervisor: Tsz Kin Chau and Sarah Kenderdine

Project description:

We have developed a Human-in-the-Loop (HITL) toolset for exploring and comparing large visual heritage collections. The primary case study is the Murten Panorama, a 10×100m oil painting (1893) digitized at 1.6 terapixels, studied alongside 15th-century illuminated manuscripts. The toolset supports correspondence detection and prototype discovery at scale, combining vision-language embedding models (e.g., SigLIP 2, CLIP) with VLM embedding models (e.g., Qwen3-VL). It includes interactive UMAP-based visualization of embedding spaces, enabling scholars to explore visual similarity across collections.

However, we currently lack a rigorous, reproducible evaluation framework to assess the performance of these tools under their various configurations.

The core objective is to develop an evaluation framework for assessing the performance of cross-domain visual retrieval and visualization tools in the context of cultural heritage image collections. The student will work on three interconnected components:

• Dataset survey and repurposing. Identify existing datasets that can be adapted as benchmarks for cross-domain visual similarity tasks. The goal is to assemble a portfolio of evaluation sets, each testing a different facet of the retrieval pipeline.
• Targeted evaluation set construction. For aspects not covered by existing datasets, the student will construct targeted evaluation sets with generative AI.
• Evaluation protocol and benchmarking. Define and implement evaluation metrics for two complementary tasks: (1) retrieval accuracy and (2) visualization fidelity.

The scope and emphasis across these three components is flexible and should be discussed with the student based on their interests and background. The project does not require model training. It is fundamentally about evaluation methodology.

Prerequisite: 

• Excellent Python skills
• Experience in AI image generator
• Interest in evaluation methodology and experimental design
• Experience with computer vision or digital humanities is a plus but not required

What the project offers:

The student will work at the intersection of computer science and digital humanities. The evaluation framework produced will have direct impact on how cross-domain visual retrieval tools are developed and assessed in the cultural heritage domain. The project offers hands-on experience with state-of-the-art vision-language models, embedding pipelines, and dimensionality reduction techniques, applied to a unique and visually rich dataset.

Reference:

Tschirschwitz, D., Klemstein, F., Schmidgen, H., & Rodehorst, V. (2023). Drawing the Line: A Dual Evaluation Approach for Shaping Ground Truth in Image Retrieval Using Rich Visual Embeddings of Historical Images. Proceedings of the 7th International Workshop on Historical Document Imaging and Processing, 13–18. https://doi.org/10.1145/3604951.3605524

Jiang, Z., Meng, R., Yang, X., Yavuz, S., Zhou, Y., & Chen, W. (2024, October 4). VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. The Thirteenth International Conference on Learning Representations.