Hi! I am Andrii Buvailo, and this is my weekly newsletter, ‘Where Tech Meets Bio,’ where I talk about technologies, breakthroughs, and great companies moving the industry forward.
If you've received it, then you either subscribed or someone forwarded it to you. If the latter is the case, subscribe by pressing this button:
Now, let’s get to this week’s topics!
Delivering a keynote speech, ‘The Age of AI: Shaping the DeepTech Future of Life Sciences,” during the annual industry event in Belgium. Image credit: Bio.be/Essenscia
The rise of foundation models in life sciences was one of the central topics of my keynote speech this week during an annual industry event in Brussels, Belgium, organized by Bio.be (part of essenscia). Essenscia is the Belgian federation representing biotech and the life sciences industry. In Belgium, they seem to take artificial intelligence adoption in life sciences very seriously, since even Alexander De Croo, Prime Minister of Belgium, was speaking at the event.
But what are AI foundation models anyway, and why is all the buzz lately? And most importantly, why should we care about this in the life sciences?
As you know, deep learning models have been trending ever since 2012, after a famous ImageNet competition where a deep convolutional neural network, AlexNet, achieved outstanding performance for image classification tasks, marking the beginning of the deep learning revolution in computer vision, with convolutional neural nets (CNNs) playing a key role.
A new era of AI: deep learning and large-scale models. Image credit: BiopharmaTrend
While CNNs are great for grid-like data structures, such as images and video frames, they are not good for sequential data, like time series. So, recurrent neural nets (RNNs) and LSTMs have joined the leaderboards of deep learning popularity quite soon, in particular in biology research.
CNNs, RNNs, and other deep learning architectures were quite popular in drug discovery, with lots of startups being founded during 2013–2017 exploiting the idea of AI-driven drug design. Some of these companies are now unicorns with AI-inspired molecules already in clinical development—Exscientia, Insilico Medicine, BenevolentAI, and Relay Tx, to name a few.
Notably, machine learning strategies of those years were primarily focused on supervised learning based on a properly laballed dataset, and datasets were relatively small (up to millions or hundreds of millions of parameters at best). Most importantly, the AI models of this period are specific to a certain narrow task. For instance, modeling target-ligand interaction or de-novo generation of small molecules biased towards desired properties, etc.
A really remarkable development occurred in 2017, a pivotal year for the AI field. Google engineers introduced a so-called transformer architecture for neural nets and a mathematical algorithm of attention in a seminal paper, ‘Attention is all you need.‘
This changed completely the playing field of deep learning and marked a paradigm-shifting rise of larger and more generalizable models. The most famous series of models includes the GPT-X family, with GPT-3 and GPT-4 being the foundations of ChatGPT, a recent phenomenon that I am sure you know well enough without my explanations.
A foundation model represents an AI neural network extensively trained on vast amounts of unprocessed data through unsupervised (self-supervised) learning, enabling its adaptability to a wide array of tasks.
Two key concepts underpin this overarching category: the simplification of data collection and the expansive range of opportunities it presents. Foundation models primarily learn from unlabeled datasets, eliminating the need for manual data labeling.
In contrast to earlier neural networks, which were narrowly specialized for specific tasks, foundation models can easily transition to various roles with minor adjustments, from text translation to medical image analysis.