Multi-Spectrum Optics Module

A tri-band sensor fusion system for firefighting droids enabling visibility in smoke and darkness.

The Challenge

Firefighting is an inherently dangerous profession, where rapid search and rescue operations must be conducted in burning buildings. While robotic droids offer a way to reduce human risk, their effectiveness is often limited by the operator’s interface.

Standard firefighting droids rely on multiple independent camera units (RGB, Night Vision, Thermal) to handle varying lighting conditions. This forces the operator to split their attention across separate screens, each with its own interpretability quirks and image setting controls. This requires the operator to perform “on-the-fly knowledge fusion” in high-stakes, split-second scenarios. Consequently, effective droid control becomes a high-cost skill requiring rigorous training, which limits accessibility—particularly in Southeast Asian countries where such technology is less established than in the West.

The Goal

The primary objective was to synthesize a consensus of information from multiple overlapping streams (RGB, Thermal, NoIR). The system aimed to highlight shared reality while preserving unique, critical details from each feed—such as heat signatures or structural edges—without losing information due to occlusion or lighting shifts.

The Solution

The final proposed solution implements a Weighted Sum of Contours algorithm that fuses information from $N$ overlapping camera feeds. This process filters visual data to extract consistent structural features across the spectrum while applying thermal masking for specific temperature ranges. The pipeline is optimized for real-time performance using MATLAB’s GPU acceleration.

Validation was conducted in two phases:

  • Synthetic Testing: Utilized a GrabCut dataset to verify robustness against strong hue shifts, distortions, and masked-out objects.
  • Physical Testing: Deployed on a rig featuring RGB, NoIR-CMOS, and FLIR A320 (LWIR) sensors. The system demonstrated efficacy in extreme low-light conditions, simulated using a controlled Short-Wave Infrared (SWIR) leak source (sunlight directed through a narrow slit blocked by heavy curtains).
System Architecture: Fusing three camera streams into a single operator display.

Project Images

Reimagined Prototype Setup with RGB, NoIR, and FLIR sensors. (The original was a cardboard box).
NoIR Camera Feed.
Raw Thermal Feed.
Final Fused Output.

Possible Future Work

This project serves as a Proof of Concept (PoC) for commercially viable modules. Future iterations could explore:

  • Advanced Sensor Integration: Leveraging high-grade RGB feeds combined with Short/Medium/Long-Wave Infrared (S/M/L-WIR) cameras to operate effectively in dense smoke, pitch darkness, or strong fires.
  • Active Illumination: Incorporating strong short-wave illuminators to enhance visibility in zero-light environments.
  • Patch-Based Compositing: Moving beyond individual feed processing to (include some kind of) patch-based local structure compositing across all sensors, ensuring better feature preservation during fusion.

Demos: Project Videos

Video.

News

  • Also super cool to see how far the field has progressed in producing cameras can capture and enhance different bands with mechanical filters www.spectralcam.com/maia-m2-modular/ www.thorlabs.com/multispectral-imaging-system
  • Jun 2020 Prior ideas for a licence/patent disbanded due to covid pandemic
  • May 2020 Final presentation and virtual graduations.
  • Feb/Mar 2020 project barely concluded prior to covid-shutdowns.