Available Projects – Fall 2026

Startup company Innoview has developed a software framework to create hidden watermarks printed on paper and to acquire and decode them by a smartphone. The acquisition by smartphone comprises many separate parametrizable parts. The project consists in improving some of the parts of the acquisition pipeline in order to optimize the recognition rate of the hidden watermarks (under Android).

Deliverables:

• Report and running prototype.

Prerequisites:

• basic knowledge of image processing and computer vision,
• Coding skills in Java Android, C#, and/or Matlab

Level: BS or MS semester project

Supervisors:

Dr. Romain Rossier, Innoview Sàrl, [email protected], tel 078 664 36 44

Prof. Roger D. Hersch, BC320, [email protected], cell: 077 406 27 09

Startup company Innoview has developed arrangements of transparent lenslets and of opaque structures that yield interesting moiré effects.

The goal is to create plastic objects composed of a revealing layer made of transparent lenses and of a base layer made of partly opaque structures. The superposition of the two layers shows interesting moiré evolutions. Once created as 3D volumes, their aspect can be simulated in Blender. After simulation and verification, these objects are to be printed by a 3D printer.

Deliverables:

    Report and running prototype (Matlab). Blender lenslet simulations. Fabricated 3D objects showing the moiré evolutions.

Prerequisites:

1. Good knowledge of computer graphics, especially the construction
of 3D mesh objects,
2. Basic knowledge of Blender,
3. Good coding skills in Matlab

Level: BS or MS semester project, master’s project

Supervisors:

Prof. Roger D. Hersch, BC320, [email protected], cell: 077 406 27 09

Dr. Romain Rossier, Innoview Sàrl, [email protected], tel 078 664 36 44

Startup company Innoview has developed arrangements of lenslets that can be used to create document security features. By simulating the interaction of light with these 3D lenslet structures one can try to improve these security features. The interaction of light with the lenslets is simulated by ray tracing, applying Snell’s law and the Fresnel equations. Possibly, these ray-tracing based simulations can be compared with the simulations obtained with the Blender software.

Deliverables:
Report and running simulation prototype (Matlab), ray-tracing based and Blender simulations for various configurations.

Prerequisites:
Knowledge of computer graphics, interaction of light with surfaces,
basic knowledge of Blender, coding skills in Matlab

Level: BS or MS semester project

Supervisors:

Prof. Roger D. Hersch, BC320, [email protected], cell: 077 406 27 09
Dr. Romain Rossier, Innoview Sàrl, [email protected], tel 078 664 36 44

Introduction:

Image relighting is the problem of modifying the illumination of a scene captured in a photograph. Recent work by Careaga and Aksoy [1] introduced a physically controllable relighting pipeline for RGB images that allows users to insert explicit light sources, such as point lights, spot lights, and environmental illumination, into the scene. Their method combines monocular geometry estimation, intrinsic image decomposition [7, 8], physically-based rendering (PBR), and neural rendering in a self-supervised framework trained on real-world photograph collections.

However, this pipeline operates exclusively in the RGB color space, which inherently limits its physical accuracy. RGB captures only three broad spectral bands, leading to ambiguities such as metamerism, where materials with different spectral reflectance curves produce identical RGB values but respond differently to changes in illumination. Multi-spectral images, which capture many narrowband channels across the visible (and potentially near-infrared) spectrum, provide a much richer description of surface reflectance and illumination. This additional spectral information can reduce material ambiguities, enable more accurate intrinsic decomposition, and support physically faithful relighting under light sources with arbitrary spectral power distributions (SPDs).

This project proposes to extend the physically controllable relighting framework of Careaga and Aksoy [1] to multi-spectral images. By operating in the spectral domain, the method can leverage richer material information for more accurate intrinsic decomposition, perform physically correct light transport simulation across all spectral bands, and produce relit multi-spectral images that faithfully capture the interaction between spectral illumination and spectral reflectance.

Methodology:

Data Preparation

• Use existing multi-spectral image datasets (e.g., KAUST reflectance dataset [5]) as the basis for training and evaluation.
• We will also collect our own multi-spectral datasets with a mobile phone equipped with a multi-spectral sensor (the lab will provide this device).

Spectral Intrinsic Decomposition

• This stage separates a multi-spectral image into spectral reflectance and shading components. Two approaches could be explored:
• Existing methods: Adopt and evaluate established multi-spectral intrinsic decomposition algorithms, such as MIID [2] (subspace-constrained Retinex) or low-rank factorization approaches [3]. These optimization-based methods offer interpretable priors (spectral smoothness, low-rank structure) and do not require training data.
• Learning-based approach: Train a neural network that takes multi-spectral input and outputs spectral reflectance and shading maps. The network can be pre-trained on synthetic spectral data (rendered with known ground truth) and optionally fine-tuned on real captures.

Geometry Estimation and Scene Reconstruction

• Estimate monocular depth from the multi-spectral input (using a pseudo-RGB conversion or a visible-band subset) with an existing monocular depth estimation model (e.g., Depth Anything [6]).
• Reconstruct a textured mesh of the scene using the estimated depth and spectral reflectance, creating a 3D representation suitable for spectral path tracing.

Spectral Physically-Based Rendering

• Allow users to define light sources with explicit spectral power distributions in the 3D scene (point lights, spot lights, area lights, environmental illumination with spectral HDR maps).
• Render the scene using Mitsuba 3’s spectral path tracing engine [4], producing an approximate multi-spectral rendering under the target illumination.

Spectral Neural Renderer

• Train a neural renderer that takes the approximate spectral PBR rendering as input and produces a photorealistic multi-spectral relighting result.
• Adapt the self-supervised training strategy of [1] to the spectral domain: use differentiable spectral rendering to reconstruct the original illuminant SPD from a multi-spectral image, generating training pairs (spectral PBR rendering, real multi-spectral image) without explicit relighting ground truth.

Training and Evaluation

• Train and evaluate the full pipeline on multi-spectral relighting tasks. Compute the peak signal-to-noise ratio (PSNR) and root mean square error (RMSE) between the original input multi-spectral image and the output from the neural renderer.

Deliverables:

• Well-documented code and trained models for the whole relighting pipeline.
• A final report detailing methodology, experiments, results, and analysis.

Type of work:

Master semester project

50% research, 50% engineering

Prerequisites:

Proficiency in coding with deep learning frameworks (e.g., PyTorch)

Familiarity with image processing and computer vision fundamentals

Basic understanding of physically-based rendering and spectral imaging

Supervisor:

Liying Lu ([email protected])

References:

[1]. Careaga, Chris, and Yağız Aksoy. “Physically Controllable Relighting of Photographs.” Proceedings of ACM SIGGRAPH. 2025.

[2]. Huang, Qian, et al. “Multispectral Image Intrinsic Decomposition via Subspace Constraint.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2018.

[3]. Zheng, Yinqiang, et al. “Illumination and Reflectance Spectra Separation of a Hyperspectral Image Meets Low-Rank Matrix Factorization.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.

[4]. Jakob, Wenzel, et al. “Mitsuba 3 renderer.” 2022. https://mitsuba-renderer.org.

[5]. Li, Yuqi, et al. “Multispectral illumination estimation using deep unrolling network.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[6]. Lin, Haotong, et al. “Depth anything 3: Recovering the visual space from any views.” ArXiv Preprint arXiv:2511.10647 (2025).

[7]. Careaga, Chris, and Yağız Aksoy. “Colorful diffuse intrinsic image decomposition in the wild.” ACM Transactions on Graphics 43.6 (2024): 1-12.

[8]. Careaga, Chris, and Yağız Aksoy. “Intrinsic image decomposition via ordinal shading.” ACM Transactions on Graphics 43.1 (2023): 1-24.

 

Introduction:

Color constancy is the problem of estimating the true colors of objects in a scene under varying illumination conditions. Traditional methods rely on low-level statistics of the image, such as Gray-World or White-Patch assumptions, but these approaches often fail in complex real-world scenes. Recent deep-learning methods [1,2,3,4,5] improve performance by learning from large datasets, yet they primarily focus on pixel-level or patch-level cues, ignoring higher-level semantic information.

Semantic information can provide strong priors for color constancy [6]. For example, knowing that a region corresponds to the sky, foliage, or human skin allows the algorithm to better infer its true color regardless of illumination. Modern segmentation and detection methods, such as SAM (Segment Anything Model [7]), make it feasible to extract semantic cues from images efficiently.

This project proposes to leverage semantic information as an additional cue for color constancy. By integrating semantic segmentation maps with existing color constancy networks, the model can use object-level knowledge to guide illuminant estimation and improve color correction.

Methodology:

1. Data Preparation
   • Use existing color constancy datasets (e.g., LSMI [3])
   • Extract semantic information using pretrained models such as SAM.
2. Baseline Implementation
   • Implement or reproduce a standard deep-learning-based color constancy network using only image-level features.
3. Semantic Integration
   • Incorporate semantic information by concatenating segmentation maps or embedding semantic features alongside image features.
   • Explore attention-based mechanisms to allow the network to weigh semantic cues appropriately.
4. Training and Evaluation
   • Train the semantic-guided network and compare performance with baseline methods.
   • Evaluate using standard metrics such as angular error and mean-squared error on illuminant estimation.
5. Analysis
   • Perform ablation studies to understand the contribution of semantic cues.
   • Analyze cases where semantic information most improves or fails to improve performance.

Deliverables:

• Well-documented code for semantic extraction and semantic-guided color constancy.
• Trained models capable of leveraging semantic information for illuminant estimation.
• Experimental results comparing baselines.
• A final report detailing methodology, experiments, results, and analysis.

 

Type of work:

 

Master / bachelor semester project

80% research, 20% engineering

 

Prerequisites:

 

Proficiency in coding with deep learning frameworks (e.g., PyTorch)

Familiarity with image processing and computer vision fundamentals

 

 

Supervisor:

 

Liying Lu ([email protected])

 

 

Reference:

 

[1]. Afifi, Mahmoud, Marcus A. Brubaker, and Michael S. Brown. “Auto white-balance correction for mixed-illuminant scenes.” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.

[2]. Kim, Dongyoung, et al. “Attentive illumination decomposition model for multi-illuminant white balancing.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3]. Kim, Dongyoung, et al. “Large scale multi-illuminant (lsmi) dataset for developing white balance algorithm under mixed illumination.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[4]. Barron, Jonathan T. “Convolutional color constancy.” Proceedings of the IEEE International Conference on Computer Vision. 2015.

[5]. Afifi, Mahmoud, et al. “Cross-camera convolutional color constancy.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[6]. Lindner, Albrecht, and Sabine Süsstrunk. “Semantic-improved color imaging applications: It is all about context.” IEEE Transactions on Multimedia 17.5 (2015): 700-710.

[7]. Kirillov, Alexander, et al. “Segment anything.” Proceedings of the IEEE/CVF international conference on computer vision. 2023.