Research: Multispectral Foundation Models

Parameter-Efficient Adaptation of Geospatial Foundation Models through Embedding Deflection

What if you could teach an ML model to see beyond RGB using just a fraction of the computational power?

Romain Thoreau et al. solve this key challenge in Earth observation AI. They present DEFLECT, an elegant approach to adapt foundation models trained on RGB to work with multispectral satellite imagery that contains valuable information in bands invisible to the human eye.

The proposed method:
- Creates a pathway to leverage spectral bands beyond RGB (like near-infrared, shortwave infrared) with minimal parameter updates
- Uses an "untangled patch embedding layer" that processes the additional radiometric information from multispectral imagery
- Preserves the pre-trained model's spatial pattern recognition while incorporating new spectral dimensions

This has important implications for applications like land cover mapping, crop monitoring, and environmental assessment where spectral information beyond RGB is crucial for accurate analysis.

Podcast: Impact AI

Early Wildfire Detection with Shahab Bahrami from SenseNet

The recent destruction of the Pacific Palisades in Los Angeles was a brutal reminder of why we need robust early wildfire detection systems. Joining me today is Shahab Bahrami, the co-founder and CTO at SenseNet Inc. – a company that provides advanced AI-powered cameras and sensors to protect communities and valuable assets against wildfires.

Shahab is passionate about using interdisciplinary research to bridge the gap between machine learning and optimization, and he begins today’s conversation by detailing his professional background and how it led him to co-found SenseNet. Then, we unpack SenseNet and how its technology works, how it gathers data for its AI models, the challenges of relying on images and other sensor data to train machine learning models, and how SenseNet uses multiple sources to detect or define any one problem. To end, we learn why and how SenseNet uses various AI models in a single sensor, how it measures the overall impact of its tech, where the company plans to be in the next five years, and Shahab’s valuable advice for other leaders of AI-powered startups.

Research: Bias

OccamNets: Mitigating Dataset Bias by Favoring Simpler Hypotheses

What if we could make AI models think more like scientists?

Machine learning models often struggle with dataset biases, frequently learning misleading shortcuts instead of fundamental patterns. Traditional solutions try to patch these issues by tweaking loss functions or resampling data.

Robik Shrestha et al. took a fundamentally different approach: designing smarter neural networks that naturally resist spurious correlations.

Their "OccamNet" leverages two key principles inspired by scientific reasoning:
- Prefer the simplest explanation possible
- Focus only on the most relevant visual information

The network is designed to:
1. Use fewer convolutional layers when possible
2. Concentrate on the most critical image regions for classification
3. Develop complex hypotheses only when absolutely necessary

By mimicking the scientific principle of Occam's Razor – that the simplest explanation is often the best – these networks outperformed existing bias reduction techniques.

In essence, OccamNets teach AI to be more disciplined and discerning learners.

Code

Insights: Distribution Shift

Addressing Covariate Shift in Pathology AI

During my recent webinar on distribution shift, an interesting question arose about a challenge that's particularly relevant to computational pathology:

Q: What are your preferred methods for covariate shift in pathology? Pre-processing (e.g., stain normalization) versus image augmentation, or both?

This question tackles a fundamental challenge in pathology AI - how do we build models that can generalize across different labs, scanners, and staining protocols?

𝐓𝐡𝐞 𝐋𝐚𝐧𝐝𝐬𝐜𝐚𝐩𝐞 𝐨𝐟 𝐒𝐨𝐥𝐮𝐭𝐢𝐨𝐧𝐬
The field has explored numerous approaches:
- Stain normalization techniques
- Image augmentation strategies
- Domain adversarial learning
- Foundation models pre-trained on pathology data

𝐖𝐡𝐚𝐭 𝐀𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐖𝐨𝐫𝐤𝐬 𝐁𝐞𝐬𝐭?
Based on my research and practical experience:

𝐅𝐢𝐫𝐬𝐭 𝐏𝐥𝐚𝐜𝐞: 𝐓𝐚𝐫𝐠𝐞𝐭𝐞𝐝 𝐈𝐦𝐚𝐠𝐞 𝐀𝐮𝐠𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧
Augmentation strategies that specifically simulate staining variations have consistently shown the strongest results. This approach helps models learn features that are invariant to the precise staining characteristics.

𝐒𝐭𝐫𝐨𝐧𝐠 𝐑𝐮𝐧𝐧𝐞𝐫-𝐔𝐩: 𝐏𝐚𝐭𝐡𝐨𝐥𝐨𝐠𝐲 𝐅𝐨𝐮𝐧𝐝𝐚𝐭𝐢𝐨𝐧 𝐌𝐨𝐝𝐞𝐥𝐬
Using foundation models pre-trained on diverse pathology datasets provides a robust starting point. These models have already learned to extract meaningful features across different staining conditions.

𝐓𝐡𝐞 𝐏𝐨𝐰𝐞𝐫 𝐨𝐟 𝐂𝐨𝐦𝐛𝐢𝐧𝐚𝐭𝐢𝐨𝐧
Importantly, these approaches aren't mutually exclusive. The most robust solutions often combine targeted augmentation with foundation models for maximum effectiveness.

𝐓𝐡𝐞 "𝐓𝐫𝐲 𝐁𝐮𝐭 𝐕𝐚𝐥𝐢𝐝𝐚𝐭𝐞" 𝐏𝐡𝐢𝐥𝐨𝐬𝐨𝐩𝐡𝐲
One key takeaway from our discussion: there's no universal solution. While research points to certain approaches being generally effective:
- Always test multiple strategies on your specific data
- Rigorously validate performance across different data sources
- Be prepared to iterate if shifts aren't fully addressed

Covariate shift remains a persistent challenge in computational pathology, but with these approaches, we're making significant progress toward truly generalizable models.

_{Enjoy this newsletter? Here are more things you might find helpful:}

_{1 Hour Strategy Session -- What if you could talk to an expert quickly? Are you facing a specific machine learning challenge? Do you have a pressing question? Schedule a 1 Hour Strategy Session now. Ask me anything about whatever challenges you’re facing. I’ll give you no-nonsense advice that you can put into action immediately.
Schedule now}

Did someone forward this email to you, and you want to sign up for more? Subscribe to future emails
This email was sent to _t.e.s.t_@example.com. Want to change to a different address? Update subscription
Want to get off this list? Unsubscribe
My postal address: Pixel Scientia Labs, LLC, PO Box 98412, Raleigh, NC 27624, United States