Podcast: Impact AI

Foundation Model Series: Transforming Biology with Zelda Mariet from Bioptimus

Zelda Mariet, Co-Founder and Principal Research Scientist at Bioptimus, joins me to continue our series of conversations on the vast possibilities and diverse applications of foundation models. Today’s discussion focuses on how foundation models are transforming biology. Zelda shares insights into Bioptimus’ work and why it’s so critical in this field. She breaks down the three core components involved in building these models and explains what sets their histopathology model apart from the many others being published today. They also explore the methodology for properly benchmarking the quality and performance of foundation models, Bioptimus’ strategy for commercializing its technology, and much more. To learn more about Bioptimus, their plans beyond pathology, and the impact they hope to make in the next three to five years, tune in now.

Research: Multimodal Foundation Models

FLAVARS: A Multimodal Foundational Language and Vision Alignment Model for Remote Sensing

There are a growing number of vision-language models available for general purpose tasks. Most of these models are trained on image-caption pairs and don't include many specialized images like remote sensing.

Isaac Corley et al. proposed a vision-language model for satellite images called FLAVARS.

Starting with a dataset of satellite images and their OpenStreetMap annotations, they adapted it by generating full captions using the OpenStreetMap bounding boxes. They also included the geospatial coordinates of each image.

To train their model, they used the contrastive objective from CLIP plus masked image and masked language modeling objectives. They also added an additional contrastive objective to align the location embeddings.

In validating their model on 12 remote sensing classification tasks, FLAVARS was typically the winner. FLAVARS also out-performed alternative image encoders for semantic segmentation.

Research: Slide-Level SSL

Multistain Pretraining for Slide Representation Learning in Pathology

Numerous foundation models for pathology have now been developed, but most operate on image tiles, not the whole slide.

Guillaume Jaume et al. introduced MADELEINE as a pretraining strategy for whole slides using multiple stains. This model was trained on H&E and 4 or 5 adjacent IHC slides.

They used the CONCH foundation model to encode image patches. Following that, separate networks are used to encode H&E images and various IHC stains. The goal of MADELEINE is to align the representations.

This approach was validated on breast and kidney datasets, each with independent validation sets for few-shot classification and survival prediction.

MADELEINE out-performed other slide representations like multiple instance learning, and an ablation study provided more detailed insights.

They found that the cross-model alignment objective was the most important innovation.

Code

Model

Insights: Foundation Models

Optimizing AI in Digital Pathology: The Power of Model Diversity

A question from my recent webinar on foundation models for pathology: If I’m not getting the expected results with one foundation model, should I try out different foundation models?

When working with foundation models in digital pathology, it's crucial to remember that no single model excels at everything.

Here's why experimenting with different models is key:

𝟏. 𝐓𝐚𝐬𝐤-𝐒𝐩𝐞𝐜𝐢𝐟𝐢𝐜 𝐏𝐞𝐫𝐟𝐨𝐫𝐦𝐚𝐧𝐜𝐞
Different models may excel at various tasks. I have yet to read an independent benchmark study that finds a single winner.

𝟐. 𝐄𝐚𝐬𝐲 𝐄𝐱𝐩𝐞𝐫𝐢𝐦𝐞𝐧𝐭𝐚𝐭𝐢𝐨𝐧
Swapping foundation models is a relatively straightforward way to explore performance improvements.

𝟑. 𝐃𝐚𝐭𝐚𝐬𝐞𝐭 𝐕𝐚𝐫𝐢𝐚𝐭𝐢𝐨𝐧𝐬
Models trained on different datasets may perform differently on your specific data.

𝟒. 𝐂𝐨𝐦𝐩𝐮𝐭𝐚𝐭𝐢𝐨𝐧𝐚𝐥 𝐂𝐨𝐧𝐬𝐢𝐝𝐞𝐫𝐚𝐭𝐢𝐨𝐧𝐬
Some models may require high-end hardware. Consider your available resources when selecting models.

𝟓. 𝐂𝐨𝐦𝐛𝐢𝐧𝐢𝐧𝐠 𝐌𝐨𝐝𝐞𝐥 𝐄𝐦𝐛𝐞𝐝𝐝𝐢𝐧𝐠𝐬
If you want to take it a step further, you could even use the embeddings from multiple foundation models to take advantage of their diverse feature sets.

💡 𝐏𝐫𝐨 𝐓𝐢𝐩: Create a benchmark of relevant tasks for your specific use case and systematically evaluate multiple foundation models to find the best fit.

Insights: Foundation Models

Expanding AI in Pathology; Beyond the Oncology Focus

A question from my recent webinar on foundation models for pathology: Are there pathology foundation models available for applications beyond oncology?

Here's the current landscape:

𝟏. 𝐃𝐢𝐯𝐞𝐫𝐬𝐞 𝐓𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐃𝐚𝐭𝐚
Most H&E-based models are trained on a mix of cancerous and non-cancerous tissues. However, there's typically a bias towards oncology applications.

𝟐. 𝐌𝐨𝐝𝐞𝐥 𝐒𝐞𝐥𝐞𝐜𝐭𝐢𝐨𝐧 𝐢𝐬 𝐂𝐫𝐮𝐜𝐢𝐚𝐥
Always verify the pre-training dataset for each model you consider. Models heavily focused on cancer may not be optimal for non-oncology tasks.

𝟑. 𝐁𝐞𝐧𝐜𝐡𝐦𝐚𝐫𝐤𝐢𝐧𝐠 𝐁𝐞𝐲𝐨𝐧𝐝 𝐂𝐚𝐧𝐜𝐞𝐫
One study compared foundation model performance on IHC and immunology applications: “Going Beyond H&E and Oncology: How Do Histopathology Foundation Models Perform for Multi-stain IHC and Immunology?” UNI and CTransPath were the top-performing models.

𝟒. 𝐀𝐝𝐚𝐩𝐭𝐚𝐭𝐢𝐨𝐧 𝐒𝐭𝐫𝐚𝐭𝐞𝐠𝐢𝐞𝐬
For non-cancer applications, consider a second pre-training phase on relevant data. This allows the model to learn features specific to your target disease or condition.

𝟓. 𝐅𝐮𝐭𝐮𝐫𝐞 𝐎𝐩𝐩𝐨𝐫𝐭𝐮𝐧𝐢𝐭𝐢𝐞𝐬
The field is ripe for expansion into broader pathology applications. Developing models for autoimmune diseases, rare conditions, and other non-cancer pathologies represents an exciting frontier.

_{Enjoy this newsletter? Here are more things you might find helpful:}

Team Workshop: Harnessing the Power of Foundation Models for Pathology - Ready to unlock new possibilities for your pathology AI product development? Join me for an exclusive 90 minute workshop designed to catapult your team’s model development.

Schedule now

Did someone forward this email to you, and you want to sign up for more? Subscribe to future emails
This email was sent to _t.e.s.t_@example.com. Want to change to a different address? Update subscription
Want to get off this list? Unsubscribe
My postal address: Pixel Scientia Labs, LLC, PO Box 98412, Raleigh, NC 27624, United States