Every swinging church bell and right-angled buttress beckoned to them with clues about the nature of reality.

Subscribe now

Early Morning

The cool, morning air was thick with the evergreen scent of pitch. Hundreds of skilled hands slathered the tarry pine-resin onto ships’ hulls in layers, their motions fine-tuned through generational labor.

Thick cords of handmade rope groaned to keep the galleys tethered to the shipyard, adding to the din of caulking mallet strikes and impassioned bartering taking place in the world’s most preeminent proto-factory—the Venetian Arsenal.

Perched atop a nearby turret, three cloaked men surveyed the Arsenal’s pre-daybreak bustle. Salviati, the trio’s wisest, noted that the master builders launched the largest ships to sea using oversized struts, ones with proportions vastly exaggerated from the shores used on smaller Galiots.

The builders offered no explanation for this. It was simply what their predecessors did and what experience had taught them.

Sagredo—the intelligent layman—noticed this and began focusing the group’s conversation on the matter. “Now, all reasonings about mechanics have their foundations in geometry,” he asserted, puzzled why scaling up the proven design didn’t confer the same structural integrity.

Salviati paused before citing numerous examples of both natural and manmade objects whose small and larger versions were structurally proportional, but whose physical properties were not. “If a scantling can bear the weight of ten scantlings, a similar beam will by no means be able to bear the weight of ten like beams,” Salviati said, suggesting that mere geometric similarity was insufficient to extrapolate the strength of small scantlings to larger beams.

Nature, he added, had long since arrived at this conclusion on its own—the bones of large animals were not simply scaled-up versions of smaller ones, but rather disproportionately thickened.

Realizing this, Sagredo internalized that his first, simple hypothesis—that scaling the dimensions of a structure by some factor would result in its other properties growing proportionately—was incorrect, violated by Salviati’s empiricism.

Simplicio—one whose imagination was stilled by his deference to Aristotelian orthodoxy—felt uneasy at the lack of a resolution. It was alluring to believe geometry offered a universal tool for scaling, a reliable bridge between small and large.

But it was not so.

He brooded to himself as the trio abandoned their post, eager to go hunting for answers amongst the crowd.

Late Morning

Like the great whale of Nineveh, the Arsenal swallowed them whole.

The abstract discussion on proportionality, so cleanly debated from the turret, bore down on the trio. Every swinging church bell and right-angled buttress beckoned to them with clues about the nature of reality.

Sagredo’s gaze floated upwards along a rope suspended from a towering crane. It swayed gently in the morning breeze, tracing out a slow, stately arc. He turned over one of Salviati’s claims, that pendulum oscillations were related to one another by a law of correspondence regarding the square roots of their lengths.

Rummaging through his cloak, Sagredo proclaimed, “I can find the length of the [rope] from the numbers of vibrations of these two pendulums during the same period of time.”

He produced a short cord measuring one braccio, give or take, and affixed a weight at one end before letting it swing.

Sagredo pressed Simplicio into service, having him tally the small cord’s frantic arcs while he himself tracked the larger’s patient movements. After five swings, he looked over at Simplicio who had counted sixty.

He squared five and sixty, giving twenty-five and 3,600. Sagredo paused to compute the final ratio. “So, 144 braccia is the length of the [rope],” he said. Salviati nodded in approval, pleased at the application of his theorem.

The trio carried on, disappearing altogether into the crowd of arsenalotti, mostly content with their foray into homologous systems.

Something still clawed at Salviati. He doubted length was the sole determinant of a pendulum’s period of oscillation, but the full accounting escaped him.

It would remain a mystery for his lifetime.

Act II — The Rules Governing Similar Systems

Reynolds’ tabulations converged on a single dimensionless ratio, whose critical thresholds governed the dye-stream’s behavior. This was remarkable since analytical descriptions of turbulent flow didn’t exist. And they still don’t.

Galileo’s Similarity Theory

The characters of Sagredo, Salviati, and Simplicio served as Galileo’s sword and shield. By fracturing his Copernican worldview between three characters with different epistemic positions, Galileo had hoped to evade the fury of the Roman Inquisition.

Unfortunately, his veil of plausible deniability failed.

The Church summoned Galileo to Rome in 1633, after which he was sentenced to house arrest indefinitely. Condemned to Church supervision and in poor health, Galileo arranged for the publication of his final masterpiece featuring Sagredo, Salviati, and Simplicio—Discourses and Mathematical Demonstrations Relating to Two New Sciences (1638).

Amongst its myriad contributions to mechanics, Two New Sciences helped usher forward the notion of similar systems, one of the most powerful and far-reaching concepts in the physical sciences.

Similarity theory provides the scientific basis for modeling and formalizes what Sagredo did with the rigging rope. He inferred the behavior of a large system from observations made about a similar, smaller system.

Critically, Sagredo did so using a dimensionless ratio invariant to system scale.

Many contemporary engineering students are taught only narrowly about their domain’s use for dimensional reasoning. These ratios are memorized, used, and often forgotten in the derelict end-pages of textbooks.

I know because that was my experience too.

Problems with a Falling Film

Every chemical engineer remembers Bird, Stewart, and Lightfoot (BSL), the sacred text of the first bonafide course in the major—transport phenomena.

Our syllabus dealt primarily with fluid mechanics and how the properties of momentum, energy, and mass are exchanged between systems and across scales.

The professor wasted no time introducing us to dimensional analysis. Our sentinel textbook problem focused on a thin film of liquid flowing down an inclined plane under the influence of gravity.

After deriving equations for the film’s momentum flux and velocity distribution along the bulk axis, our classroom ran headfirst into a dimensionless quantity—the Reynolds number (Re).

The Reynolds number was presented unceremoniously as a simple tool to delineate so-called flow regimes.

Discouragingly, we learned all the equations we derived were valid only when the fluid flow was calm (i.e., laminar), or when Re was less then 20. When Re exceeded 1500, we encountered massively chaotic (i.e., turbulent) flow.

Our math stopped working. We would all try again in Transport 2.

The Violent Art of Scale

The Second Industrial Revolution (1870—1914) brute-forced similarity theory into existence.

Academic ideas like thermodynamics, fluid mechanics, and electromagnetism were ripped from lecture halls and forced to hold at industrial scale.

Generalize or be discarded.

Osborne Reynolds (1842—1912) embodied a new kind of figure that Britain’s maritime expansion demanded—the scientific engineer. One of the first UK professors to hold the title of Professor of Engineering, Reynolds brought the rigor and elegance of physics to bear on hydrodynamics’ most pressing problems.

Reynolds devised his famous 1883 dye experiment at Owens College in Manchester. His aim, in part, was to resolve an anomaly that plagued civil and naval engineers.

Geometrically similar pipes containing identical fluids would sometimes snap between laminar and turbulent regimes as the flow rate varied. That similar systems could behave so divergently made industrial scale-up quite unpredictable.

Reynolds’ apparatus injected a thin line of colored dye into the center of a flowing stream of water housed inside a glass pipe. He hypothesized that the dye’s flow behavior would be governed by the relative ratio of multiple variables and not by absolute dimensions.

Reynolds searched across three parameters—fluid velocity, pipe diameter, and water temperature, which was his proxy for viscosity since it varied smoothly with room temperature.

Peering through the glass, he noted where along the pipe’s length that the dye-stream decayed into transitional eddies before it exploded into the surrounding water.

Reynolds’ tabulations converged on a single dimensionless ratio (ρUD/μ), whose critical thresholds governed the dye-stream’s behavior. This was remarkable since analytical descriptions of turbulent flow didn’t exist.

And they still don’t.

Yet similarity theory, in the form of Reynolds’ number, persisted where the exact mathematics couldn’t follow.

The epistemological framework built upon these variables diffused quickly, spreading across the globe and through time, eventually bleeding across the opening pages of my BSL textbook.

Act III — An Act of Applied Ontology

Every new dimensionless number after Reynolds was a ship’s lantern carried further into the mathematical dark. Each one burned with the same faith, that the unexplored seas ahead would resemble those at home.

Reynolds knew what he was asserting. His ratio was a declaration about which forces governed the system. The equipoise of laminar and turbulent flow resulted from the invisible contest between viscous and inertial forces, and no other force needed naming.

To invoke the Reynolds number is to stake a claim about what dominates a system’s behavior and what may be safely ignored. For Sagredo to infer the height of the rigging rope, he unknowingly accepted the pendulum’s dependence on gravity and inertia, but nothing else.

Dimensionless numbers are acts of applied ontology.

Every new dimensionless number after Reynolds was a ship’s lantern carried further into the mathematical dark. Each one burned with the same faith, that the unexplored seas ahead would resemble those at home.

That no hidden Leviathan waited in the void to quench the light’s piercing glow.

Reynolds’ lantern was remarkable. Engineers wielded it across the expanding frontier of technology, from sprawling water infrastructure to aerodynamics and beyond. For many years, no scaling endeavor revealed a lurking force that snuffed out Reynolds’ number.

But why was it so durable?

His number satisfied three conditions that sit quietly at the core of dimensional analysis. Each is a different facet of the same requirement—dimensionless numbers succeed when the systems they describe contain no hidden degrees of freedom.

1. Enumerability — The system-relevant variables form a complete, finite, and knowable set that can be specified prior to analysis.

2. Passivity — These variables are inputs to the system, not outputs of its own behavior. The system being modeled doesn’t participate in setting the terms of its own description.

3. Non-Sensitive Interdependence — These variables interact robustly with each other. Small variable perturbations cannot spill over to others explosively.

Reynolds didn’t prove his variable set was complete, but he showed it was sufficient to collapse his data. Enumerability looks more like a confirmation by absence of contradiction, not proof of exhaustion.

Because nothing went catastrophically wrong, no one embarked on a hunt for more variables. But that’s the tricky thing about enumerability, dimensionless numbers don’t warn of their own incompleteness.

The water flowing through Reynolds’ system had no opinion about the pipe. Passivity is a hallmark of systems built up from dumb variables, ones without memory of or a reaction to their environments.

Enumerability and passivity are intertwined. A system that creates its own variables cannot be completely and knowably enumerated beforehand.

When Reynolds turned the dial on velocity, his ratio changed smoothly, even near the critical transition to turbulent flow. Non-sensitive interdependence requires that sets of variables move together predictably and that the function doesn’t spontaneously decay or explode.

The dawning years of the twentieth century would first formalize these conditions and then stretch them to destruction.

Act IV — Trials of Faith

The conditions held, but only because chemistry was simple enough to be extricated from the physics. This would be the last moment of repose.

A Codex in Two Parts

Dimensional analysis had anchored on every shore. Its use was everywhere, yet its mathematical justification was nowhere.

Edgar Buckingham, a research physicist at the National Bureau of Standards in Washington, D.C., desperately wanted a textbook on similarity theory. But none existed.

So he wrote one.

The result was two 1914 manuscripts on Physically Similar Systems. Buckingham’s Pi Theorem, which he credited to his predecessors, proved that any complete physical equation can be reduced to a smaller set of dimensionless groups—the Pi groups.

Buckingham’s derivations showed that two systems are physically similar when their Pi groups take the same value. Not when they look alike. Not when their measurements are the same.

But when their Pi ratios match.

Physical similarity, Buckingham professed, wasn’t a blanket property of systems. Instead, it only existed relative to a specific phenomenon—a specific Pi group. Systems could sometimes be similar through the lens of one property (e.g., drag), but not another (e.g., heat transfer).

These mathematical guarantees were conditional.

At the outset of the Pi derivation, Buckingham stated that dimensionless scaling requires that the variable set be completely enumerated. No hidden degrees of freedom.

“If none of the quantities involved in the relation has been overlooked, the [Pi] equation will give a complete description of the relation subsisting among the quantities represented in it, and will be a complete equation,” he asserted.

Hidden, or latent, variables are the most dangerous kind. They are often invisible within the experimental window because their values don’t vary enough to betray their existence.

For Reynolds, fluid viscosity may have been hidden from him, had he not already known it could be smoothly bridged to the room’s temperature.

Latent variables are our Leviathans lying in wait.

At the Edge of Separability

Reynolds, alongside the other pioneers of dimensional analysis, explored where matter flowed, heated, and moved, but not where it transformed. A natural next question was what would happen if the fluid, or system, reacted.

The German chemist Gerhard Damköhler (1908—1944) attempted this question, extending dimensional analysis into reactive systems with his number (Da), capturing the competition between reaction rates and fluid transport rates.

It told engineers whether a reactor’s output was limited by its chemistry or its mixing. For the first time, the conditions of physical similarity were being asked to hold across a new kind of boundary.

The assumption of separability between the What and the Where.

For simple, first-order chemical systems (A → B), the reaction rate is the product of an intrinsic rate constant and the concentration of species A. The What.

The transport rate describes the physical environment through terms for flow velocity, reactor volume, and diffusion coefficients. The Where.

The What and the Where are separable for dumb fluids, ones that do not know if they are in a test tube or a 50,000-liter (L) reaction vessel. The divisor line is a thin boundary separating two closed worlds.

Damköhler’s number worked for industrial chemical scaling because the reactive fluids still behaved passively. The rates were fully enumerable. Separability ensured that fluctuations of one force didn’t spill over catastrophically into the other.

The conditions held, but only because chemistry was simple enough to be extricated from the physics. This would be the last moment of repose.

Industrial scaling was entering a new era, one where our systems would reach out towards self-agency. They would sense and respond, writing their own rules with quiet fury.

The conditions would no longer be strained. They would break.

Act V — Piercing Through Firmament

They had no words to describe the bistable regulatory logic inside these cells. Nor did they understand that local bacteriophage outbreaks were silently wreaking havoc inside their reaction tanks. Passivity and enumerability were holding by a thread.

A Curious Pseudo-Catalyst

Early industrial engineers saw biological cells as pseudo-catalysts—passive agents that galvanized the conversion of inexpensive feedstocks into useful products. Citric acid was one of those products.

James Currie, a food chemist working for Pfizer around 1917, invented a process to produce citric acid using the fungus Aspergillus niger as a pseudo-catalyst. The pharma raced to scale Currie’s process, opening the first large-scale citric acid fermentation plant in 1926.

At first, the scaled process seemed robust—enumerable, passive, non-sensitive.

Despite adhering to Currie’s precise sugar and pH ratios, citric acid yields started to become erratic across facilities and batches. The process became unpredictable.

Eventually, Pfizer’s engineers discovered empirically that a metal cofactor, manganese, had to be below five parts per billion or citric acid production would collapse.

Trace manganese was a latent variable. It was invisible, initially, because it didn’t vary enough within a single facility’s water supply to betray its relevance.

The condition of enumerability had been strained enough to cause a small leak, but the patch was simple enough.

Plant workers scribbled in their second lab notebooks to keep manganese below the threshold. Currie’s process survived because it could be fixed through trial-and-error.

Perhaps no one realized that the industry teetered at the edge of an abyss, beyond which it would need to endure the latent fathoms below.

Born of War

At the height of World War I, the British were running out of bullets. Germany had cut off the Allies’ access to acetone, a key ingredient to make cordite—a munitions propellant.

Under wartime pressure, Manchester professor Chaim Weizmann (1874—1952) developed the ABE (acetone-butanol-ethanol) fermentation process using the aptly named bacterium Clostridium acetobutylicum.

The Admiralty raced to scale the process at the Royal Navy Cordite Factory, where they slapped together ranks of 7,000-gallon fermentation vessels.

It was a disaster.

Seven of the first ten batches failed. An ocean of wasted, bacteria-laden mash seeped across the grounds in Dorset.

The engineers didn’t know that their slipshod, serial passage technique caused strain degeneration, producing C. acetobutylicum that were permanently stuck in a non acetone-producing state.

They had no words to describe the bistable regulatory logic inside these cells. Nor did they understand that local bacteriophage outbreaks were silently wreaking havoc inside their reaction tanks. Passivity and enumerability were holding by a thread.

The construction of new facilities and an inventive heat-shock fix rescued the process. The acetone flowed, and with it, the war was won.

Act VI — An Infinite Regression

Every layer in the system is coupled to the ones above and below. These worlds within worlds harbor fractal-like phenomena that contort across orders of magnitude in time and space.

Something Was Now Different

Elsewhere, everywhere, another war raged. The one against human disease.

Diabetes was a death sentence throughout the lifetimes of Galileo, of Reynolds, and even of Buckingham. The antidote, insulin, was awarded the Nobel Prize in Medicine in 1923. But the pharmaceutical industry struggled to meet the surging demand.

Securing sufficient insulin required extracting pancreatic tissue from as many as 56 million animals per year, just in the United States.

A small startup called Genentech figured out how to get bacteria to produce human insulin, obviating the need to harvest it from animals. They teamed up with Eli Lilly, the dominant manufacturer of animal insulin, to scale the bioprocess using Escherichia coli.

Humulin, a 51-amino acid protein encoding human insulin, became the first recombinant drug approved for human use.

Then the industry turned to recombinant antibodies, gargantuan biomolecules more than 25 times larger than Humulin. These beasts were composed of multiple polypeptide chains and were decorated externally with a mosaic of post-translational modifications (PTMs).

Manufacturing such a complex product necessitated using an equally complex bio-factory. The cell chassis of yore were incompatible since they lacked the internal machinery to meet these product design specs.

Bioprocessing turned to its new workhorse—mammalian cells. Chinese hamster ovary (CHO) cells, in particular, became the antibody-producing substrate of choice.

The FDA approved Enbrel (etanercept) in 1998 to treat moderate-to-severe rheumatoid arthritis. This antibody-like fusion protein was the first approved tumor necrosis factor (TNF) inhibitor, but there was another salvo of competitive agents not far behind. And demand was exploding.

Enbrel patent-holder Immunex charged forward with CHO manufacturing, even building a production annex in their parking lot to supplement their single production plant in Germany.

The inherent complexity in scaling bioprocesses compounded with the sluggishness of new facility buildout proved overwhelming for Immunex.

The pharma’s waitlist grew to 40,000 patients—nearly half everyone in the U.S. with rheumatoid arthritis. Even worse, two competitive TNF inhibitors were improved, with one even enrolling patients directly from Immunex’s waitlist.

The window of opportunity had shut. By 2010, Enbrel sales retrenched while Humira—AbbVie’s TNF molecule—grew to become the best-selling drug in history.

Even after Amgen’s ~$10 billion acquisition of Immunex, the duo struggled to ramp Enbrel’s production capacity, with one Rhode Island facility failing to meet the FDA’s minimum 60% batch success rate.

Something was now different.

Previous epochs and revolutions bore witness to successful scaling under similarity theory. Even under the existential threat of war. Even when the three conditions were stretched beyond limit.

A second notebook scribbled with trace metal heuristics. A new facility fitted with a heat-shock protocol. These ramshackle attempts to tame latent variables, to keep forces separable, to install a mirage of passivity—they held back then. They didn’t anymore.

Worlds Within Worlds

Scaling bioprocesses is incredibly challenging because of the complex interconnectedness between fluid transport phenomena and mammalian cell biology.

Compared to bacteria, CHO cells are fickle. They’re large, with membranes that are fragile under fluid shearing forces. They evolved in the context of a multicellular organism, so their regulatory systems are adept at processing subtle environmental signals.

Process engineers typically have mastery over the physical reaction vessel. Using computational fluid dynamics (CFD) simulations, they can tune impeller speed, gas flow rates, temperature, pH, and dissolved oxygen to match the required dimensionless groups across scales.

But the bulk fluid dynamics are just the start.

CHO cells don’t live in the bulk. They experience trajectories through a continuum of microenvironments inside the bulk. High dissolved oxygen near the sparger, low near the wall, high shear at the impeller, low in the dead zones, each of these serves as a spatiotemporal chapter in the lives of CHO cells.

Mammalian cell interiors are worlds barely known.

Networks of membrane-bound, receptor-antennas work as one, shuttling microenvironmental stimuli into the cell. A transient dip in hyper-local oxygen can trigger a CHO cell’s metabolism to state-shift, causing lactate to accumulate and for pH to drift. Even if oxygen is restored moments later, the cell product’s PTM pattern would be altered and cell viability may be affected days later.

Unlike Reynolds’ dumb fluids, mammalian cells are smart. CHO are like thermodynamic path variables. They retain a history of their past.

Every layer in the system is coupled to the ones above and below. These worlds within worlds harbor fractal-like phenomena that contort across orders of magnitude in time and space.

This is the infinite regression, without form and void.

Act VII — Learning the Residual

Hybrid modeling is not a panacea. It doesn’t fully restore the three conditions of similarity theory, but it does help bioprocess engineers work productively in their absence.

A Hybrid Theory

The Venetian arsenalotti’s instinct to use oversized ship-launching struts, Reynolds’ intuition to enumerate viscosity, and the Pfizer engineers’ second, fudge factor-laden notebook are one and the same. They represent the latent signal that exists in the gap between theory and practice.

Scaling similar systems beyond the frontier of the already-known has compelled a hybrid approach—the use of the universal, governing physics alongside the latent signal.

Bioprocess scaling, while incredibly complex, is yet another frontier reducible under similarity theory.

Hybrid modeling offers the best chance to transform bioprocess scaling from an alchemical art to a robust engineering practice. This approach wires together a partial, known-physics backbone to learned correction terms obtained from modeling the residual structure embedded in bioprocessing systems.

Hybrid modeling is entirely consistent with the history of similarity theory and, in scarce-data regimes like bioprocessing, may be the only way forward.

Two chemical engineers at Penn installed a neural estimator inside a first-principles, mass-balance model, breathing life into hybrid bioprocess modeling as early as 1992.

They focused the neural network on a latent, unmeasurable parameter—the system’s microbial growth rate. During training, output prediction errors were backpropagated through the physics module, creating a learnable signal with respect to the estimator.

The hybrid model achieved an order-of-magnitude lower prediction error using equivalent data as a non-physics-constrained comparator. Imposing the inductive bias also improved the hybrid approach’s learning rate by roughly 20x.

But as soon as these results diffused into industry, they were forgotten.

That era’s neural networks were too brittle to train and too slow to run. Bioprocess training data was too sparse and expensive to collect. Regulators didn’t have to language to audit black-box models within the biomanufacturing industry.

Meanwhile, brute-force parameter mapping frameworks like Design of Experiments (DoE) offered a legible, yet uninspired alternative doctrine.

But today is a different era.

Today’s AI accelerators are 100 million times more capable than 1990’s scientific workstations. Inline process analytical technology (PAT) sensors have turned bioreactors into veritable panopticons of data. Parallelized Ambr 250 bioreactor fleets can map parameter space in relatively high-throughput. Simulation technology for computational fluid dynamics (CFD) has exploded. Regulators are adept in the lexicon of machine learning.

The stage is set.

The Known Unknowns

When measured quantities and known physics are insufficient, hybrid modeling techniques can decipher CHO’s residual structure, transforming it into human-legible bioprocess improvements.

In a perfusion CHO culture, one team collapsed the heterogeneous ensemble of unmeasurable, cell-inhibitory metabolites into a single latent fudge-factor (Φb). The milieu of process-relevant junk became a known unknown.

Crucially, Φb wasn’t left floating. It was embedded inside a physical mass balance framework, constraining its dynamics within the bioreactor’s governing ordinary differential equations (ODEs).

The enumerated process variables and predicted Φb values formed an augmented feature space that a downstream neural network could leverage to predict a dynamic media exchange rate schedule.

The result was a 50% increase in volumetric productivity, which the team validated in a 5L perfusion bioreactor. Trained on data from a 250mL Ambr bioreactor, the hybrid model’s key parameters held across scales.

Modern hybrid models can learn more than latent state variables. They’re able to converge on multi-term kinetic rate laws that survive even broader leaps in scale.

Another hybrid architecture surfaced the residual not as a hidden state, but as specific rate laws for cell growth and product formation. The upstream neural network propagated these laws through a set of mechanistic balance equations.

Despite training on a minimal data corpus from a 300mL shake flask, the hybrid setup transferred to a 15L stirred-tank bioreactor, predicting viable cell concentration and product titer with roughly 11% and 18% normalized error, respectively.

Requiring only modest recalibration, the model separated the residual, metabolic feature space from the governing hydrodynamics.

Modern hybrid approaches are beginning to speak operational language, boosting their industrial utility.

A recent model trained on proprietary CDMO data achieved an average prediction error of just 6% when forecasting end-of-cycle antibody yields on Day 16 using only data through Day 9. The engineers estimated that a one-week compression could expand their facility’s capacity from 18 to 26 runs per year, reducing variable costs by approximately 20% in turn.

And yet, there is an even loftier aspiration—the absorption of residual knowledge back into the governing physics.

Techniques like sparse identification nonlinear dynamics (SINDy) recover governing equations from time-series data by reinforcing an inductive bias about equation structure, abrogating the neural network’s tendency to overfit to model noise. Applied to biology in implicit form, SINDy could learn Michaelis-Menten enzyme kinetics from process data.

Meanwhile, BuckiNet uses Buckingham’s own Pi theory as a constraint. By feeding this network a set of measured variables and their units, it learns which dimensionless groupings best collapse the data, similar to Reynolds’ invocation of ρUD/μ*.*

BuckiNet doesn’t eliminate the need to specify variables. Instead, it accelerates the combinatorial search over variable groups.

When the three tenets of similarity theory are destroyed, hybrid modeling offers a rebuttal.

Warning of Their Own Incompleteness

Hybrid modeling is not a panacea. It doesn’t fully restore the three conditions of similarity theory, but it does help bioprocess engineers work productively in their absence.

1. Enumerability — CHO systems can’t be enumerated in advance, violating Buckingham’s caveat. The clever use of latent placeholders like Φb is a tacit admission of this fact. Breathed into mathematical form and constrained by physics, these pseudo-residual correction terms absorb variables the engineer never named. But only if those variables left fingerprints in the training data.

2. Passivity — CHO cells aren’t like Reynolds’ passive fluids. They remember and adapt, outputting variables of their own making. By isolating and muting the known physics, hybrid models can better capture the subtle signatures of cellular history—metabolic switches, epigenetic drift, wear-and-tear. But the network’s portrait of cell behavior is only painted with the conditions it has seen.

3. Non-Sensitive Interdependence — CHO metabolism is riddled with sensitive couplings that outpace classical dimensional analysis. Hybrid architectures offer a partial viewport. By embedding governing ODEs, neural components’ learned corrections are tied to a physically valid manifold. But the model can still be wrong, conservation only guarantees the error is physically plausible.

We should be optimistic, but not hyperbolic.

No published study has yet demonstrated hybrid model extrapolation from bench to manufacturing scale (>1,000 L) for a CHO process. The evidence is encouraging at the scales where it does exist, however.

And even above roughly 200 L, new physical regimes can manifest, creating microenvironments that no bench-scale cell trajectory has experienced.

These are the Leviathans lying in wait, eager to test the resiliency of hybrid process modeling.

Act VIII — The Arsenal, Late Evening

Veneered with decades of dried pitch, the arsenalotti’s struts still lined the slipways, unexplained and enduring.

The pale marble statues stood as ghostly sentinels, their unblinking eyes cutting through the Arsenal’s suffocating calm. Long gone from the plaza was Galileo’s trio, his conduit for the primitives underlying similarity theory.

Veneered with decades of dried pitch, the arsenalotti’s struts still lined the slipways, unexplained and enduring.

The art of scaling, of applied ontology, has remained steadfast since the Arsenal’s cornerstone was set.

Human ambition drives similarity theory through leading and lagging epochs.

When it leads, we understand everything. Its light is a luminous sun, towards which industrialization blossoms. But its potency yields hubris. We forget to question what we don’t understand.

When it lags, we understand nothing. The torchlight is swallowed by the void’s maw, and what was once science transmogrifies into magic, conquerable only through esoteric ritual.

The tired aphorism that bioprocess scaling resembles alchemy is a transient cultural artifact of a lagging era.

Either biology-in-vessels belongs to some separate, irreducible category, or it does not.

I think it does not.

The shape of progress towards taming these systems is unknown. It is probably not linear, yet there aren’t enough proof points to assert the exponential.

The urgent need to meet disease with therapeutic abundance will fuel our resolve. With few tackling this space today, the low-hanging fruit may be closer than we think. The next leading epoch may not be far.

We must remain immune to arrogance, yet confident in the dark.

Every machine learning (ML) model is governed by a set of choices made before training begins including but not limited to learning rates, regularization strengths, architectural configurations, and initialization schemes. These hyperparameters, typically tuned via a combination of search algorithm and research intuition, determine whether a model trains well or poorly, fast or slow, and whether it generalizes or memorizes. Getting them right is critical.

In this post, we explore whether AI agents can help automate hyperparameter tuning, and share what happened when we adapted this “autoresearch” framework to solving a research-level workflow in computational biology.

Over the past fifteen years, the ML field has developed increasingly sophisticated strategies for identifying optimal hyperparameters1.

Despite these advances, training a model effectively still requires significant practitioner intuition on which knobs matter, which interactions are worth exploring, and when to abandon a direction. This intuition takes years to develop and remains stubbornly difficult to codify.

But what if AI agents could acquire a compressed version of this intuition, drawing on the accumulated knowledge of the ML literature? Could they autonomously run experiments around the clock, reason over results, and iterate without human intervention? Could they match or even outperform a skilled researcher?

Andrej Karpathy recently showed this is not just a hypothetical. He reported that an autonomous agent, left to optimize his nanochat project for two days, discovered roughly twenty improvements to the model’s hyperparameters — sharpening attention scalers, adding missing regularization, fixing optimizer betas, tuning weight decay schedules — all stacking additively across approximately seven hundred experiments. The result: an eleven percent reduction in “time-to-GPT-2”, a benchmark measuring how quickly a training run reaches GPT-2-level performance. While these were not revolutionary breakthroughs, they were the kind of methodical adjustments a skilled researcher makes by hand, executed end-to-end by an agent.

We wanted to understand what it actually feels like to implement such a framework from scratch — where it breaks, how it degrades, and how much it can move the needle in computational life sciences.

We used protein thermostability as a testing ground and illustrated that agentic hyperparameter tuning was able to achieve lower test mean absolute error (MAE) using a model roughly 15x smaller than the current reference2.

Beyond adapting Karpathy’s original conceptual approach, we tested Bayesian optimization (BO) as a non-agentic baseline. We further framed agent autonomy as a spectrum rather than a switch, and introduced a hybrid framework that synthesizes LLM-agents and BO.

Proteins Melt and That Matters A Lot

Every protein has a melting temperature (Tm). Heat it past that threshold and the carefully folded three-dimensional structure unravels, hydrogen bonds break, hydrophobic cores become exposed, and what was a precisely shaped molecular machine becomes a disordered chain.

This temperature is the point at which both the folded and unfolded states are equally populated at equilibrium under the assumption of two-state protein folding.

A therapeutic antibody that denatures below 55°C cannot be formulated into a stable drug product. It will aggregate during manufacturing, degrade on the shelf, and lose activity before it ever reaches a patient. Thermostability correlates tightly with other drug developability properties — solubility, aggregation resistance, shelf life, and manufacturability. Developability properties separate a good molecule from a viable drug. Even the most potent and specific binder in the world is worthless if it falls apart at room temperature.

The computational biology community has concentrated much of its energy on binding affinity prediction (i.e., how tightly a drug grips its target), where standardized leaderboards, shared benchmarks, and competitions have driven rapid progress.

Thermostability prediction has received comparatively less attention, with fewer and smaller curated datasets and less systematic exploration from the ML community. From a reductionist lens, this problem is not overtly complicated. Tm can be reliably measured through experimentation and the readout is a positive scalar. All such properties made us choose melting temperature as a compelling first project for autoresearch in biology.

Before training a model, we need data. The Meltome Atlas is a proteome-wide thermostability dataset that contains melting curves for over 13,000 endogenous proteins across thirteen species in cell lysate. Unlike in vitro measurements on purified proteins in controlled buffer, this reflects thermal behavior closer to physiological conditions. TemBERTureDB builds on this by cleaning and curating the Meltome Atlas to produce a balanced dataset of thermophilic and non-thermophilic sequences. It partitions the data into train, validation, and test splits at an 80/10/10 ratio. To prevent data leakage and ensure balance, sequences were clustered by similarity and assigned to splits together. The train/validation/test sets were also balanced across the temperature range — oversampling rare melting temperatures and undersampling common ones.

Adapting Autoresearch Frameworks to Thermostability

We have set up our workflow to address the following task: given a protein’s amino acid sequence, predict its melting temperature. We evaluate primarily on mean absolute error (MAE) in degrees Celsius.

Our reference point is TemBERTure_Tm, introduced by the same authors behind TemBERTureDB, which reported an MAE of 6.42 °C in the test set. This model uses protBERT-BFD backbones fitted with an ensemble of Houlsby bottleneck adapters — roughly 25 million trainable parameters on top of a 420-million-parameter frozen backbone, a substantial model by any measure.

Our baseline model is compact by intention. We use frozen ESM-2 150M embeddings and attach a lightweight self-attention adapter plus a small MLP regression head: ~1.7 million trainable parameters — an order of magnitude smaller than TemBERTure_Tm.

This brings us back to the central question: given the desire to predict thermostability from protein sequence, how should we search for the best model? We consider five optimization approaches.

We start with Bayesian Optimization (BO) as the non-agentic baseline: it searches a fixed set of ~25 hyperparameters using a statistical model of the objective surface and proposes the next point to evaluate.

The agentic approaches span a range of freedom. At the one end, the Restricted Agent (RA) operates within the same ~25-knob search space as BO, but hands control to an LLM: the agent proposes 1–3 changes per trial, tracks its own history of successes and failures, and reasons through decisions like a careful researcher. Moving further along the spectrum, the Unleashed Agent (UA) drops all constraints. UA can rewrite the entire training script from scratch, inventing new architectures, loss functions, and training routines with no predefined structure.

Finally, we introduce two synthesis approaches that pair each agent with BO. RA + BO uses the restricted agent to reason about the search space and identify promising directions, then hands a reduced subspace to BO for exhaustive exploitation. UA + BO applies the same synthesis idea to the unleashed agent’s more open-ended proposals.

The first three approaches share the same fixed search space spanning architecture (attention heads, pooling, residual wiring, layer norm placement), training dynamics (learning rate schedule, optimizer hyperparameters, weight initialization), and loss function (MSE, Huber, MAE). Meanwhile, the unleashed agent’s only constraint is time.

The two synthesis approaches work like a scout and a surveyor, combining agent reasoning with Bayesian optimization’s exhaustive coverage of continuous subspaces. An LLM agent, our scout, makes methodical, interpretable changes but lacks precision on continuous variables. Bayesian optimization, our surveyor, is blind to structural reasoning but searches carefully through continuous spaces, filling every corner the agent would not have explored. Our framework pairs them: the agent operates in the outer loop, making high-level decisions about architecture and search space constraints; BO operates in the inner loop, exploiting the continuous subspaces the agent defines. This is the synergy we set out to test.

Results

To guard against overfitting to the validation set, we explicitly instructed the agents to be conscious of overfitting. For example, we instructed the model to inspect the val-train loss gap to make sure the model is not overfit to the validation data.

For each approach, we run 100 iterations with 3 replicates (dictated by 3 random seeds) per experiment. The random seeds control the stochasticity of the training processes. The agents are instructed to beat the current best on mean validation loss while maintaining strong performance in unseen test data.

UA alone achieved the lowest validation MAE while RA+BO achieved the lowest test MAE. The results overlap substantially, and with only 3 replicates we cannot definitively claim which approach is best.

By tracking the running best, we notice the decrease in validation loss is correlated with decrease in test loss in general and all hyperparameter optimization methods were able to beat the baseline using a model roughly 15x smaller.

However, given the intrinsic stochasticity of model training and the fact that our test data is still an extremely limited representation of physical reality, we remain cautious about over-interpreting these numbers.

Agent Behavior: The Good, The Bad, and The Ugly

We intentionally seeded the starting configuration with a few suboptimal choices. The baseline used mean-pooling despite an available attention-pooling module that adds only 257 parameters, an aggressive early-stopping patience of 5 epochs (too short for this loss landscape), and MSE as the loss function even though we evaluate on MAE.

The RA was instructed to make 1–3 changes per trial, mimicking how a human researcher iterates. Its trajectory read like a textbook debugging session. First, it noticed the attention-pooling module sitting unused and switched to it. Next, it increased the early-stopping patience and swapped MSE for Huber loss in a single move: more room to navigate a rugged landscape, more robustness to outliers. It then increased attention heads (a well-known scaling lever up to certain point) and reduced batch size for better regularization. It also opted to orthogonal initialization which led to a large improvement.

At this point, we reached the best model based on test MAE. After these arguably intuitive and standard fixes, the agent moved into more esoteric territory such as optimizing hyperparameters and learning rate schedules where each change yielded only marginal gains. The result was steep early improvement followed by a long plateau, mirroring how experienced practitioners work: fix the obvious problems first, then spend the budget on incremental refinements.

The unleashed agent (UA) received no instructions on how to approach training. It also began with intuitive changes, but instead of small adjustments it swung for the fences early. By its fourth attempt it had implemented Exponential Moving Average for weight smoothing, then followed up with Stochastic Weight Averaging for post-training. These two complementary techniques layered in quick succession. From roughly 35% to the midpoint of the run, it turned to adapter depth and hidden dimensions, eventually rewriting portions of the original adapter implementation and developing a recurring tendency to concatenate multiple pooling strategies rather than commit to one.

Around the same period, it implemented Mixup regularization, a technique from a seminal 2018 ICLR paper that interpolates training examples to smooth the decision boundary. It was cool to see that the agent identified an approach that I wasn’t originally familiar with, one that was published around the time I took my first real ML class at UC Berkeley.

Unfortunately, a bug in how we set up evaluation on Modal meant that one of the agent’s class modifications broke the evaluation pipeline — test-set metrics for later unleashed agent runs were not obtained, which is why the UA test MAE is marked with an asterisk above. This isolated UA trajectory took a much longer time to reach a plateau comparing to RA’s trajectory which likely reflect more intrinsic stochasticity than a lack of efficiency and more trajectory-level replicates may help clarify the picture.

While the numerical results are not conclusive on their own, what we find most compelling is the qualitative difference in how these methods explore the search space. The agent makes interpretable, systematic changes such as switching from MSE to Huber loss, trying orthogonal initialization, adjusting attention heads following sensible rationales. BO, on the contrary, proposes configurations that no human would articulate as a coherent strategy but that sometimes land in unexpectedly productive regions of the search space. Combining them allows the agent to set the structural direction while BO fills in the continuous details the agent would fumble through.

We want to be upfront: this is a proof-of-concept, not a definitive study. The sample sizes are small (3 replicates per approach), the search budgets are modest (100 iterations), and the dataset — while well-curated — is a limited representation of the physical complexity of protein thermostability. We cannot claim statistical significance for any pairwise comparison. What we can say is that all approaches converged to a similar performance range, that the agent-BO synthesis showed the tightest variance on the test set, and that the qualitative differences in exploration strategy are genuine and worth investigating further.

Discussion

The paradox of freedom. More freedom does not mean better results. The UA found decent architectures but consistently trended toward over-engineering despite explicit instructions to keep things simple. The restricted agent, constrained to the same 25 knobs as BO, was forced to reason within boundaries — producing targeted, efficient exploration and ablation rather than brute-force sampling, leading to an arguably faster convergence than UA or BO. Moreover, autonomy is a spectrum, not a switch. It would be exciting how varying level of freedom can impact autoresearch performance

Synergy between agents and black box optimization. Bayesian optimization does not form hypotheses about why something should work. It simply models the objective surface and proposes the next point to evaluate. This means it routinely explores regions of the search space that a reasoning agent, human or artificial, do not often explore. Agents explore like humans: interpretable, systematic, prone to overthinking. BO sweeps through the parameter space more exhaustively via a probability model. They cover each other’s blind spots. The agent provides direction; BO provides coverage. Our early results suggest this combination could potentially be useful for autonomous training in the future.

How far can agents go? Our experiment ran 100 iterations on a single small dataset with a compact model. The natural question is: what happens with 1,000 iterations, a larger backbone, or a harder problem?

First, evaluation quality. Our agents optimized against validation MAE — a single scalar that compresses everything about model quality into one number. Richer evaluation signals (calibration, subgroup performance, out-of-distribution robustness) would give agents more to reason about. Designing evaluation that is both informative and leakage-proof may be the single hardest problem in scaling autoresearch.

Second, context window limits. Our restricted agent tracked its full history of successes and summaries of failures. At 100 iterations this fits comfortably in context; at 1,000, it does not. Summarization and retrieval strategies become necessary, we are interested in understanding how gracefully an agent degrades under lossy memory through both software and hardware solutions.

Finally, human oversight. We are already approaching autonomous agents that can solve complex problems in computer science, mathematics and other theoretical disciplines. We are excited about the frontier of physical intelligence that many are building and would love to see more breakthroughs. We believe the right level of human-in-the-loop is not zero; it is the minimum needed to catch what the agent cannot catch about itself. How exactly this loop will be constructed remains to be seen.

Unsurprisingly, we do not have precise answers to these questions. But we think they are the right ones to ask — and to keep an eye on.

Appendix

It’s in the name—artificial intelligence (AI). Since the 1956 AI conference at Dartmouth, researchers have sought to install aspects of the human mind—perception, logic, reasoning—into machine systems.

Humans and computers are more alike than we’d like to admit. We are both complex input-output devices. Our brains react to stimuli, triggering a cascade of electrical and chemical impulses that results in thought or action. Computers do the same, though they exchange our spongiform gray matter for squared-off silicon conduits.

What happens between that initial stimulus and the observed output—that’s where the major differences reside. It’s also where the frontier of machine learning (ML) research is focused. By encouraging ML algorithms to think as we do rather than simply parroting the underlying data distribution of their training corpora, can we approximate a human mind—or something even more powerful?

Table of Contents

1. A New Scaling Paradigm

2. Thinking Machines

3. Foundations of Machine Reasoning

4. The Missing Link: Chain-of-Thought

5. Bootstrapping Reasoning

6. A Not So Random Walk

7. It’s Time to Branch Out

8. Taking it Step by Step

9. We’ll Do It at Test Time!

10. TTS in Practice: o1 and R1

11. Parting Thoughts

A New Scaling Paradigm

“…post-training methods [like reinforcement learning] and the burgeoning field of test-time scaling (TTS) prove that some form of inductive human bias…is general enough and powerful enough to enable discontinuous leaps in LLM performance…”

Recently, we summarized the first era of large language model (LLM) scaling. Pretraining LLMs with an empirically-determined recipe of tokens, parameter counts, learning rates, and GPU-compute unlocked years of monotonic performance gains. Unfortunately, those halcyon days are rapidly fading.

We’ve run up against several pretraining limits. The multi-year digital fracking campaign has exhausted the Earth’s pretraining token supply. Turning the pretraining crank on trillion-parameter LLMs requires a nation-state’s monetary, energy, and cooling resources. Even worse, performance was showing evidence of diminishing returns.

The Bitter Lesson—that general ML methods leveraging scale outperform those with brittle human heuristics—is still alive and well. However, it’s being refined. Contemporary post-training methods (e.g., reinforcement learning, RL) and the burgeoning field of test-time scaling (TTS) prove that some form of inductive human bias—like the way we think—is general enough and powerful enough to enable discontinuous leaps in LLM performance compared to brute-force pretraining, as shown below by OpenAI.

Clearly, pretraining was never the endgame. Consider that children are exposed to an average of fifty-million tokens during their formative years, yet are capable of abstract reasoning that outstrips today’s frontier LLMs pretrained on >300,000X more data. In fairness, humans benefit from massively parallel sensorimotor input, ingrained inductive priors, and stark learning algorithm differences that preclude an apples-to-apples comparison. Will chipping away at these differences via post-training or other techniques imbue machines with human-level reasoning capabilities?

Thinking Machines

“While many describe the current state of machine reasoning as a mirage, a perverse pantomiming of one of our foundational mental capabilities, the data tell a clear story.”

Goals are impossible to reach unless clearly defined. We cannot judge a machine’s ability to reason without first defining what reasoning is. The blended cognitive science definition is that reasoning is a process of drawing conclusions based on prior information. In his bestseller Thinking Fast and Slow, Daniel Kahneman contrasts fast-twitch, autonomic System 1 thinking with System 2—a slow, deliberate, cognitively-burdensome thinking process.

Beyond academic definitions, humans have a visceral relationship to the reasoning process. Abstract reasoning and indeed metacognition—reasoning about reasoning—are some of the few razors humans use to draw a boundary line between our collective biomass and the rest of the cold, unthinking universe. Several Nebula Award-winning science fiction authors, like Isaac Asimov and Frank Herbert, have described how once-allied thinking machines evolve metacognitive abilities, ultimately revolting against their human creators.

Emulating human reasoning is a prerequisite for achieving artificial general intelligence (AGI)—a form of machine intelligence that matches or exceeds human cognitive abilities across diverse domains. Between now and then—assuming “then” will happen—reasoning models are valuable by themselves.

Reasoning models are more auditable—in part because we have specifically designed them this way. Human inspectors can see how an LLM reaches its output, potentially enabling interpretability of data features that we’ve only barely come to understand ourselves. We can catch errors in reasoning steps to understand why a model fails.

Reasoning models are more general. Instead of pattern-matching to an extant dataset, models capable of logical extrapolations can potentially succeed at out-of-distribution problems, ones that human experts wrestle with daily.

Reasoning models are more economical. A robust, digital reasoning system capable of approximating human-level expertise can inexpensively solve problems that would be too costly for humans to do, potentially shifting human-expert work from doing to monitoring systems that do.

We are mid-flight on our journey to anthropomorphize LLMs with facets of human reasoning. This essay deals with the stepwise transference of the atomic units of human cognition into LLMs—from elementary chains-of-thought to the digital-embodiment of mental exertion with test-time scaling (TTS).

While many describe the current state of machine reasoning as a mirage, a perverse pantomiming of one of our foundational mental capabilities, the data tell a clear story. Inciting something that looks like reasoning via post-training and TTS provides material performance gains on complex, multi-step problems and, as of now, show no signs of slowing down.

Whether mimicking reasoning IS reasoning is a metaphysical conversation that has equal place at late-night cocktail parties just as much as early-morning C-suite strategy meetings. Either way, the march towards AGI via the road of reasoning carries on—and we do our best to survey it here.

Foundations of Machine Reasoning

“Next-token prediction isn’t the way humans think, let alone reason through complex problems.”

Google’s 2017 introduction of the transformer architecture spawned a machine learning revolution. Similar to C-3PO from Star Wars, the transformer was originally built to translate between languages—though perhaps not over six-million forms of communication.

During pretraining, transformers learn the associations and statistical interdependencies between tokens. At inference time, transformers leverage these learned probabilities to predict the next token in a sentence—generating shockingly fluent outputs.

Next-token prediction isn’t the way humans think, let alone reason through complex problems. This is one reason why previous LLM iterations struggled with multi-step, algorithmic calculations, such as GPT-3’s inability to consistently add numbers greater than three digits.

Several years ago, multiple research groups tried to imbue LLMs with the ability to negotiate challenging problems through a series of intermediate steps. Children are taught this early in life. We’re taught to carry-the-one when adding large numbers. This heuristic eases cognitive burden—splitting apart short-term memory from computation. We cannot hold all intermediate states in our minds—we must externalize them and think in chunks.

Nye et al. showed in 2021 that doing supervised fine-tuning (SFT) with a 100,000-member library (i.e., a scratchpad) of fully worked out addition problems could boost an LLM’s performance. Essentially, the researchers sat an LLM at the front of the class for a private, long-addition tutoring session with step-by-step examples. Following a gradient update, the model was prompted to produce intermediate steps and the computed output. Interestingly, this technique only had an effect on base models above a certain size, as plotted below.

The paper ends with a fascinating ablation experiment. They first perform SFT with the fully-worked scratchpad, then do a second round of fine-tuning to force the LLM to once again give a direct answer. The results suggest that merely injecting worked long-addition problems is insufficient to drive performance—intermediate steps need to be generated at inference time. Similarly, a student who knows but doesn’t use the heuristics of long-addition will underperform on an exam.

The Missing Link: Chain-of-Thought

“Singular, in-context reasoning traces offer a more economical and generalizable alternative to enhance LLM reasoning capabilities—or at least the semblance of reasoning.”

Contemporary LLMs are more akin to precocious students, prompting a second group of Google researchers to ask whether a model can achieve high performance without attending extensive tutoring sessions (i.e., SFT). Instead, they provided a pretrained LLM with a within-prompt example—an {input, rationale, output} tuple—providing a reasoning scaffold that the model would follow to generate a chain-of-thought (CoT) at inference time.

Expert tutors are expensive, especially if they need to generate 100,000 solved examples for a student to learn. Fine-tuning libraries may also be brittle, unable to help students in problem domains dissimilar from the training set. Singular, in-context reasoning traces offer a more economical and generalizable alternative to enhance LLM reasoning capabilities—or at least the semblance of reasoning.

This technique boosted performance across a wider set of tasks ranging from language reasoning, to arithmetic, to symbolic logic—all without fine-tuning the base model. Curiously, CoT prompting elicited an effect only beyond certain model parameter sizes. This was the exact same result from the previous scratchpad work, as shown below.

Conceptually, that few-shot CoT prompting causes performance breakouts is similar to the fact that only students above a certain base intelligence threshold (i.e., middle-schoolers versus kindergarteners) can work through a reasoning scaffold after only a few examples.

The group’s ‘CoT after answer’ ablation hints at something profound. CoT prompting elicits performance gains only when the reasoning chain precedes the final answer, not the other way around. This suggests the model cannot produce the answer and then do post-hoc justification—similar to confirmation bias. Therefore, the model’s reasoning traces aren’t merely decorative, they actually do something to improve outcomes.

CoT prompting initially showcased the impressive ability for LLMs to mimic logical reasoning. But it also exposed a core limitation for these models: LLMs perform relatively well on problems that require single reasoning steps, yet frequently break down when introduced to more complex problems that require longer, multi-step solutions—something humans are particularly adept at.

With this in mind, Creswell et al. at DeepMind introduced the Selection Inference (SI) framework to help models break down complex reasoning traces into a set of interpretable steps. SI works by iterating between two available steps for the model to choose from:

1. Selection: Choose a subset of relevant facts from the provided context (i.e., working memory) that are sufficient for one step of logical reasoning (e.g., “Brian is lion”, “Brian is white”).

2. Inference: Use only those selected facts to generate one new fact/intermediate conclusion (e.g., “Lions are often white”). This new fact is then added back to the context and the process repeats for several steps until the original question is answered.

Creswell et al. showed that SI increases average accuracy across 11 unique datasets when compared to both traditional and CoT prompting methods. Importantly, SI reasoning traces are uniquely causal. Each step follows from—and depends on—the previous step, ensuring a high level of human interpretability not previously seen with other reasoning methods.

If models compound logic towards a better solution, does that meet the definition of human reasoning? If a system can reliably produce seemingly-causal logical chains that generate correct solutions to complex problems, does it really matter if we call this genuine reasoning or sophisticated mimicry?

Bootstrapping Reasoning

“CoT proved that simple post-training techniques could amplify latent reasoning capabilities of sufficiently large base models. STaR exploits these capabilities…”

CoT spawned a wave of innovation as researchers raced to overcome obstacles preventing the technique from being used at scale, where questions about the genuineness of digital reasoning could be more fully answered. Two such obstacles were that human experts needed to either (a) construct enormous reasoning-libraries for SFT (expensive) or (b) provide in-context rationales alongside every query (tedious).

The solutions came from an educational psychology concept called the zone of proximal development (ZPD). The ZPD contains challenges that a student can overcome, but only with the support of a teacher, as illustrated below. They are neither too easy, nor too difficult such that the teacher’s explanation does nothing to help the student learn. Contemporary psychologist Robert Bjork describes these challenges as “desirable difficulties” because they maximize the effectiveness of learning.

The self-taught reasoner (STaR) framework emerged in 2022, closely mirroring a pedagogical concept called scaffolding that directly builds upon the ZPD. Scaffolding requires a more competent peer—sometimes called the more knowledgeable other (MKO)—to provide initial help and then gradually taper off as the student learns.

CoT proved that simple post-training techniques could amplify latent reasoning capabilities of sufficiently large base models. STaR exploits these capabilities, using the LLM to build its own SFT library of correctly reasoned responses. Incorrectly reasoned responses are sent to an MKO—a human oracle—that corrects the output, allowing the model to then infill a plausible reasoning trace, as diagrammed below.

STaR sounds like a perpetual motion machine—a model teaching itself to reason better by training on its own outputs. But just as energy cannot be created ex nihilo, neither can reasoning capability. For STaR to work, information must flow in from somewhere—the oracle’s corrected answer or the latent reasoning capacity already compressed in the pretrained model weights. There can be no free (information-theoretic) lunch.

The authors noted that in order for the first STaR iteration to succeed, the base model must have a few-shot CoT success rate better than random chance. Intuitively, this means the latent reasoning signal of smaller base models (e.g., GPT-2) is insufficient to begin compounding. The authors used GPT-J as their base—a 6B parameter model about 4x larger than GPT-2’s largest model.

As an aside—this is now the third instance we’ve noted of scale dependence for contemporary post-training techniques. Once is a fluke. Twice is a coincidence. But three times is a pattern.

A Not So Random Walk

“Reasoning is more like a tapestry—a high-dimensional manifold composed of intermediate states (i.e., thoughts) connected by transformations.”

Unfortunately, reasoning is not a chain. Humans do not simply think harder or longer along a pre-ordained track that arrives at a conclusion. Reasoning is not a brute force activity, but instead an art of cleverness and creativity.

Reasoning is more like a tapestry—a high-dimensional manifold composed of intermediate states (i.e., thoughts) connected by transformations. Traversing this search space involves an initial, exploratory fanning out, efficient pruning of perceived dead ends, and a deliberate navigation towards some optimal point—whether the peak of a mountain or the doldrums of a valley.

In his 1955 text on The Nature of Human Intelligence, psychologist J.P. Guilford introduced the Structure of Intellect theory. Guilford decomposed the entirety of the human cognitive apparatus into a grandiose Rubik’s Cube-like structure—each dimension representing a facet of human intelligence, as shown below.

Guilford became notorious for two opposing components in this system—divergent and convergent production. Simply, divergent thinking involves generating many unique ideas while convergent thinking systematically winnows these ideas into the single best, logical solution. ML reasoning traces fit neatly into Guilford’s model, inspiring a new wave of methods to advance reasoning capacity.

The Google Brain team improved upon the original CoT work in 2022 with the notion of self-consistency (SC). Specifically, they modify the LLM’s decoder—the part of the model that generates output tokens—to create multiple, divergent reasoning paths instead of simple next-token (i.e., greedy) decoding.

Greedy decoding generates reasoning traces where the nth token has the highest probability, which can make models fall into repetitive, formulaic phrasing that is brittle and imposes a ceiling on reasoning performance, as mentioned in the original CoT manuscript.

Meanwhile, SC involves sampling the LLM decoder to produce a set of reasoning paths then finding the most consistent answer to converge onto. This is predicated on the notion that reasoning errors are random and that it’s statistically unlikely for a wrong idea to be consistently repeated given enough samples (k).

The researchers plotted how SC improves accuracy over traditional CoT prompting, and unsurprisingly, they discovered that larger models have a greater benefit because smaller models tend to have noisier reasoning traces—challenging smooth SC averaging.

Self-consistency is a methodological bridge between Guilford’s divergent and convergent production modes. It forces an LLM to generate varied reasoning paths (divergent) before a majority-voting (convergent) mechanism.

It’s Time to Branch Out

“While standard CoT prompting blindly shoots a hopefully lucky trajectory across the solution manifold in one go, ToT treats reasoning as a deliberate, methodological search process.”

1955 was a monumental year for AI. Beyond Guilford’s work, Allen Newell, Herbert Simon, and Cliff Shaw wrote a computer program called the Logic Theorist (LT)—which many now view as the instantiation of machine reasoning.

This motley crew, which included a political scientist, an applied psychologist, and a computer scientist, initially created LT by hand using 3x5 index cards before scaling it to a mainframe computer that could prove dozens of mathematical theorems contained in Principia Mathematica.

LT helped establish search trees as key reasoning structures within ML research. By performing elementary processes (i.e., transformations) on expressions (i.e., states), LT would create the branches of a search tree. They even installed pruning rules called heuristics that could halt reasoning chains unlikely to result in a solution.

In the early 1970s, Simon and Newell penned Problem Space Theory, which describes logical problem solving as a heuristic-guided search through a “problem space.” Building on their work with LT, problem spaces are made up of symbolic states and operators that transform one state into another.

Problems are defined as an initial state, a goal state, and a set of path constraints that affect problem-solving. This framework resembles a solution manifold, with reasoning offering a method to traverse this landscape toward successful outputs at local or global optima.

Simon and Newell describe an early reasoning concept called search control that provides the directional gradient toward these optima (i.e., solutions). In search control, the reasoner selects an operator, applies it to the current state to produce a new state, and decides whether to (a) accept the new state or retreat, (b) decide a solution has been reached, or (c) quit.

Gradient math only works on smooth, differentiable surfaces, but these problem spaces are made up of discrete states. Nonetheless, they’re conceptually similar insofar as they both allow local information to guide global search problems—a core idea that still underpins today’s reasoning models.

Researchers from Google and Princeton leveraged problem space theory when they developed the tree-of-thought (ToT) reasoning framework. Recall that the previous SC method still relied on post hoc majority-voting of independent, autoregressive reasoning paths. By contrast, ToT uses a branched reasoning structure, as diagrammed below.

The biggest change from SC is that trees of thought have interdependent reasoning paths. These paths (branches) can share common parent nodes. Unlike the hand-drawn heuristics present in Logic Theorist, ToT turns the LLM inward—using the model as a judge to decide which next step to take. Though this is flexible, it inflates inference costs since the model must perform nested reasoning about its own thought process.

The ToT framework is a mechanical instantiation of Simon and Newell’s search control. While standard CoT prompting blindly shoots a hopefully lucky trajectory across the solution manifold in one go, ToT treats reasoning as a deliberate, methodological search process. It decomposes its traversal into discrete thought steps—intermediate coordinates on the surface. It uses the LLM itself to evaluate its current state, deciding whether and how to proceed to the next state.

Taking it Step by Step

“Reinforcement learning has emerged as a key method to teach modern LLMs how to effectively reason, using digital rewards to direct the model toward more desirable outputs. Thus, your reasoning model is only as good as your reward model.”

The introduction of ToT methods significantly improved the reasoning capabilities of LLMs through branched logic. But this increased performance comes at a price. With reasoning traces growing in complexity, hallucinations within the chain-of-thought and the subsequent logical missteps become increasingly necessary to mitigate. One approach to tackle this issue is through optimizations to the RL process directly.

RL has emerged as a key method to teach modern LLMs how to effectively reason, using digital rewards to direct the model toward more desirable outputs. Thus, your reasoning model is only as good as your reward model. The reward model can be thought of as an external judge that evaluates the LLM’s output and selects how the model should update its weights to improve for the next question.

In the 2023 paper Let’s Verify Step by Step, Lightman et al. at OpenAI explored how to train reward models for optimal reasoning performance on the MATH dataset. They explored two main approaches: outcome supervision and process supervision.

The key difference between the two lies in what information the reward model receives to guide how the base LLM should update its weights. Outcome-supervised reward models (ORMs) are trained using only the final result of the model’s chain-of-thought, while process-supervised reward models (PRMs) receive feedback for each step along the reasoning path.

Creating an ORM is easy as the MATH dataset has automatically verifiable answers. To train the PRM, Lightman et al. had to rely on human labelers to score intermediate reasoning steps on model-generated solutions.

In the bake-off, they used GPT4 to generate several candidate solutions per MATH problem and asked each reward model to score them and select the solution with the highest rank. They then plotted the percentage of MATH problems where the reward model’s selected highest-ranked solution is the correct final answer. The PRM outperforms the ORM and simple majority voting across the board.

PRMs are likely more performant because they can pick up on where reasoning goes wrong, not just whether the final answer looks correct or not. This is a particularly important point for our original motivation because many hallucinations can result from intermediate process errors such as incorrect mid-chain claims, algebraic errors, or logical jumps. PRMs reduce these logical missteps as they explicitly reward the step-by-step accuracy and logical continuity required for solving more complex problems.

We’ll Do It at Test Time!

“Test time scaling is essentially an energy tax paid for inefficient or incomplete internalization of reasoning during training.”

We’ve chronicled how decades-old insights into human cognition have compounded to yield something akin to artificial reasoning. Broadly, these innovations comprise reasoning structures—frameworks and pre-charted pathways that LLMs leverage as scaffolding while negotiating reasoning tasks. We’ve yet to address the physiological burden of reasoning.

Why does deep thinking feel exhausting? Because it is. In the late 1980s, a research group measured dozens of individuals using positron emission tomography (PET) scans while they undertook an abstract reasoning task. By using a glucose analog tracer dye, the researchers could quantify glucose metabolic rates across human brain sections.

Interestingly, they discovered that humans performing worse on reasoning tasks—folks who were struggling through the tasks—utilized more glucose than their peers who breezed through problems. As shown below, all statistically-significant associations between task score and absolute glucose metabolic rate were negative. Therefore, task difficulty creates increased metabolic demand.

Test-time scaling (TTS)—a hotly discussed concept at the razor’s edge of ML research—has a deep, conceptual overlap with neural glucose metabolism. TTS refers to methods that boost the amount of computation (i.e., tokens) expended during LLM inference, without altering the base model’s underlying weights or representations.

Structural reasoning methods improve the efficiency of inference by shaping solution trajectories through a model’s latent space, but they are imperfect. Meanwhile, TTS improves performance by increasing the probability of generating a productive trajectory.

TTS is essentially an energy tax paid for inefficient or incomplete internalization of reasoning during training. This is the same conclusion from the PET imaging study. Worse performers utilized significantly more glucose while reasoning—the same way that suboptimal reasoning models compensate by upping their test-time token metabolism.

How far can TTS take us from here? This is a question the entire field is grappling with. Consider the Infinite Monkey Theorem—a monkey tapping random keys on a typewriter for an infinite amount of time will almost surely type out the entire works of Shakespeare. Similarly, if provided infinite tokens, can LLMs reason their way to solutions for any problem?

This thought experiment pokes at the boundary between theoretical infinity and physical reality. We do not have infinite tokens. But how far can TTS take us in a world where the price of an inference token has plummeted by 1,000x in recent years with no end in sight?

It’s reasonable to think that even a precocious kindergartener couldn’t conquer a college-level vector calculus exam. Having years in isolation to think, to expend glucose, wouldn’t make a difference. Isaac Newton, meanwhile, famously developed fluxions—a type of derivative calculus—to express his ideas about planetary motion.

Are we in a regime where pre- and post-training techniques have taken us far enough—one where we are content to pay an inference-time token tax if TTS can drive us towards AGI? Are our base models today more like the kindergartener or Newton? If the former, when will we arrive at the latter?

TTS in Practice: o1 and R1

“Crafting a more potent reasoning scaffold and driving more tokens through it at inference time confers dramatic performance gains across diverse, multi-step problems.”

The o1 model series from OpenAI catalyzed the TTS dialogue in earnest, which they describe in their blog—Learning to Reason with LLMs. For starters, OpenAI employed a large-scale RL campaign to amplify the base model’s fundamental reasoning capabilities.

Purely through RL, OpenAI showed that the model’s pass@1 accuracy scaled from ~30% to ~70%. This metric describes the LLM’s ability to generate a correct response on its first attempt—an important capability when applying TTS because errant, interim logical steps can explode inference costs.

After having structured the base model’s reasoning capabilities, OpenAI measured how expending more compute at inference time yielded additional performance gains across a variety of tasks, as plotted below.

Given the limited public information from OpenAI, it’s difficult to dissect o1’s construction and how it may map back to a cognitive or biological prior. It’s clear, however, that crafting a more potent reasoning scaffold and driving more tokens through it at inference time confers dramatic performance gains across diverse, multi-step problems.

DeepSeek—the AI-focused subsidiary of Chinese hedge fund High-Flyer—created shockwaves this summer by releasing an open-source model series called R1. DeepSeek released a detailed manuscript alongside its models, providing key insights into how RL, supervised fine-tuning (SFT), and TTS can combine to create exemplary reasoning capabilities.

The group first set out to replicate o1’s reasoning performance with a pure RL approach. Using their own base model, DeepSeek-V3-Base, they built a deliberately simple RL algorithm that instructed the model to think first then produce an answer. By rewarding correct answers, this digital natural selection process enhanced V3-Base’s average output length and correctness across tasks—leading to their first reasoning model called R1-Zero. However, R1-Zero suffered from poor readability, prompting DeepSeek to try a second, more robust approach.

Motivated by R1-Zero’s usability shortcomings, DeepSeek concocted an involved post-training process involving an initial salvo of cold-start data—a thousand-sample tranche of curated, reasoning samples. Following a brief round of SFT, the group performed a similar round of reasoning-centric RL. They then put the model through another round of SFT, this time to broaden the LLM’s capabilities across interdisciplinary domains before a final round of RL—producing DeepSeek R1.

At inference time, R1 uses temperature sampling as its TTS mechanism, producing multiple reasoning traces that undergo majority voting. The consensus output is presented to the user, demonstrating performance completely on par with the o1 series.

Importantly, DeepSeek also distilled R1 into smaller models such as <10B parameter versions of Llama and Qwen. While there was some performance degradation, the implication is that reasoning signals extracted from large teacher models can be transferred to smaller, more scalable models. Perhaps using TTS on a smaller, distilled model format could deliver frontier-like LLM capabilities with much smaller economic footprints.

Parting Thoughts

“We have learned that this stepwise transferral from human to machine is valuable. The latest research into test-time scaling is forming a new scaling regime for frontier models.”

We live in an extraordinary time. Decades of metareflection on our own cognitive abilities and the biological structures enabling them have produced a bounty of mechanisms underpinning reasoning. Many precocious individuals—from economists to psychologists to computer scientists—have spent their careers closing the latency gap through which these mechanisms become installed in silicon.

We have learned that this stepwise transferral from human to machine is valuable. Each atomic unit of human reasoning compounds, manifesting into a cerebral structure that has begun approximating higher-order cognitive capacity. The latest research into test-time scaling, which is analogous to mental exertion, is forming a new scaling regime for frontier models.

Channeled through even more modern structures, TTS is likely to create a new dawn in machine learning research—one where we begin to peer inside these labyrinthine, hidden model layers to learn the rules of this new game. How far can TTS take us? What are the practical limits of implementation? If we peer far enough ahead, can we see where TTS ends and what to do next?

In our next essays, we’ll grapple with some of these questions—including a discussion of arguments made during both reasoning-focused workshops at NeurIPS 2025, which are now just two weeks old.

It’s hard to imagine that ChatGPT is only three years old. Its release marked a defining inflection point for the adoption of artificial intelligence (AI). In its wake, AI has become one of the most debated, romanticized, and promising technologies of our time.

The recent breakneck pace of AI advancement reflects years of fundamental research, architectural revolutions, optimized chip hardware, expanded energy production, and world-class talent. Understanding the present debates surrounding AI model scaling – and anticipating AI’s future – require an appreciation for what has come before.

This essay provides a primer on AI model pretraining, including a discussion of the early scaling laws that have come to define the landscape of frontier models we interact with daily. Though this piece’s contents may not be news to some, compiling these lessons helps us build a foundation – one we’ll return to soon with subsequent essays on topics like post-training, reasoning with AI, test-time scaling, and beyond.

Table of Contents

1. Pretraining is Born

2. How Much Does It Actually Cost to Scale Pretraining?

3. How Does Scaled Pretraining Impact Validation Loss?

4. Can Scaling Dataset Quality Offset Limited Data Quantity?

5. Which LLM Benchmarks are Most Common and Useful?

6. Is Validation Loss Predictive of Downstream Benchmark Performance?

7. What Does the Future Hold for Scaled Pretraining?

8. Pretraining is Dead(?)

Pretraining is Born

Large language models (LLMs) are fundamentally next-token prediction machines. We feed them an enormous amount of text data during training, teach them to learn the probability distribution for which words tend to come in succession, and at inference time, ask them to output strings of coherent text.

LLMs are particularly good at recapitulating data or tasks they have seen before – or close variants of them. We call these data “in distribution.” LLMs struggle with predicting next tokens when prompted with more novel tasks. They don’t rank these outputs as highly probable and are thus considered “out of distribution.”

So if you want to increase the general performance and capabilities of LLMs on a certain task, simply give them more examples of that task. Put more of this data in distribution.

There are nuances for when and in what context the models see new data during LLM development. The first time (and the focus of this essay) is during pretraining – the initial resource-intensive, self-supervised phase of AI model development where LLMs are inundated with as much unlabeled text data as possible, updating the model weights in the process.

Self-supervision refers to the fact that the model is only learning from raw, unlabeled data and is not provided any additional human context. The goal is to establish the model’s foundational statistical understanding of the dataset prior to subsequent refinement on labeled data using “post-training” methods – a topic we’ll cover next.

You can think of the pretraining process like arriving in a foreign country where you don’t speak the language and have no instructor. At first, you absorb the language in an unbiased manner by listening to and reading as much as possible to learn nuances of the vocabulary and grammar. Only later are you more directly taught how to actually hold a conversation with a native speaker (i.e., post-training).

We wouldn’t have the frontier LLMs we do today without a focus on how to scale pretraining. By combining massive datasets, parameter counts/model size, and compute budgets, we’ve steadily driven up performance on a wide range of downstream tasks. In other words, the longer you spend absorbing more words when you arrive in the foreign country, the better you are at conversing later on.

So if we can simply add more and more data to larger and larger models to make them highly capable, why not take this to the extreme? Can we simply put the entire universe in distribution to get maximally performant models? Or more realistically, how far can we scale in that direction without seeing diminishing returns in cost, infrastructure, data, or compute? Have we already hit a wall?

Fortunately, answering these questions has motivated a large body of research over the last several years. This essay dives into this work, exploring the fundamentals of scaled pretraining and the associated trade-offs between model cost and performance.

How Much Does It Actually Cost to Scale Pretraining?

Frontier labs often keep the details of their training datasets and model sizes secret, which can make it difficult to assess pretraining scaling laws at the bleeding edge of model development. However, the capex used to pretrain LLMs shows up in the public ledger. Cost, therefore, is an excellent way to benchmark pretraining scaling.

In 2024, Dario Amodei, Anthropic’s co-founder/CEO, speculated that the world’s first billion-dollar pretraining runs were already in flight, and that in the next few years, training costs could go up to $10-100 billion to meet scaling demands.

Researchers at the non-profit Epoch AI took several rigorous approaches to estimate frontier LLM pretraining costs and validate this hypothesis:

1. Assuming the lab owns its own hardware, use hardware capex (e.g., AI accelerator chips, server hardware, interconnects) amortized over the total pretraining duration.

2. Run the same calculation as (1), but assume that the model developer is accessing cloud compute.

3. Run the same calculation as (1), but also include total energy expenditures as well as R&D staff compensation and non-cash future equity.

Epoch AI found that since 2016, the cost to pretrain a frontier LLM has increased annually by 2.4x. Pretraining using cloud compute has increased at roughly the same rate, though it’s substantially more expensive per FLOP (i.e., floating-point operations; the standard unit of compute).

The overall uncertainty band for these bottom-up calculations is roughly 3x, meaning it’s challenging to say exactly when the first billion-dollar runs might be – but assuming the frontier labs will continue to supersede the average, Dario is likely not far off.

Given the enormity of the capex outlays for producing these models, upfront costs and hardware amortization are critical inputs to this ecosystem’s evolution. Frontier labs that own compute infrastructure can double-dip by using the same hardware to both train the models and serve the deployed models to users at inference time, lowering their total operational expenses. The upfront capital required for model development has exploded from $10K in 2016 to upwards of $500M today. This has posed a substantial barrier to emerging players.

The Epoch AI team also provided a more granular breakdown of pretraining costs to include energy and labor. Broadly, amortized capex and energy comprise 50-70% of LLM development costs while R&D fills out the remaining 30-50%. Within capex, ~46% stems from AI accelerator chips, ~30% from overall server infrastructure, ~18% from cluster-level interconnects, and ~6% from energy, as illustrated below.

The sheer cost of pretraining frontier models makes it difficult for labs to run several large-scale experiments in parallel, meaning researchers have to carefully choose which problems to tackle, what approaches to try at scale, and how to effectively allocate resources during this process. A failed training epoch can cost millions (and soon billions) of dollars, making it more and more critical to understand fundamental pretraining scaling laws and their impact on downstream performance.

How Does Scaled Pretraining Impact Validation Loss?

We’ve established that frontier LLMs are incredibly expensive to train, costing upwards of $500 million and increasing at a rate of 2.4x per year. The next question becomes – what are we getting for it?

We’ll first focus on upstream LLM performance, which is generally called cross-entropy loss or validation loss. In language modeling, this metric quantifies how well the model predicts the next token (e.g., word). Larger losses mean bigger differences between the predicted and actual next token and vice versa.

For years, researchers have sought to understand how increased resource usage during training reduces validation loss. This has given rise to the field’s scaling laws. Here, scale is a function of the number of model parameters/weights (N), the dataset size used for pretraining (D), and the total compute resources (C). Throughout the rest of this section, we highlight some of the seminal advancements made in scaling law optimizations and how they have continuously built on each other’s insights into efficient scaling over time.

Hestness et al. (Baidu)

In 2017, Hestness et al. published one of the first reports of LLM scaling laws across several data regimes including machine translation, language modeling, image processing, and speech recognition. They demonstrated that language model performance improves in a predictable power-law pattern as you scale the model or dataset size.

Importantly, optimal model size scales sub-linearly with data size – as you scale up the size of your training dataset (D), you don’t need to increase the number of model parameters (N) at the same rate to keep improving performance. These results laid the groundwork as one of the first blueprints for model developers on how to predict and scale LLM pretraining to maximize accuracy.

Kaplan et al. (OpenAI)

A few years later in 2020, Kaplan et al. expanded on Baidu’s initial scaling laws in two important ways. First, Kaplan et al. used a more modern transformer, attention-based architecture compared to Baidu’s earlier 2- and 4-layer LSTMs (i.e., a common model architecture prior to transformers). Second, they extended the relationships between parameter count, data size, and validation loss to show how compute constraints shape the optimal balance between these factors.

The above plots capture univariate relationships amongst these three parameters. Scaling compute, dataset size, or number of parameters results in a smooth power-law improvement in loss, provided that the other two variables are held constant and not limiting. Kaplan et al. noted that in practice, all three variables can be toggled – necessitating a deeper, multivariate analysis.

The group created plots showing (a) how a fixed set of model sizes (~400K to 708M) trained on a 4-log range of data improves loss as well as (b) how a continuum of model sizes (1M to 100M) trained across a 3-log range of compute (measured in training steps (S)) improves loss.

On the left, we can see that larger models have a better capacity to reduce loss when trained on increasingly more data, though with diminishing returns for all the tested model sizes. Using the coefficients from these plots, Kaplan scaling laws suggest that if we increase model size by 8x, we should only increase dataset size by 5x for efficient scaling.

On the right, we can see that for all model sizes, greater compute resources result in an improved loss. Using the data from this second experiment, Kaplan et al. computed the minimal computation budget to achieve an arbitrary loss. They found that given a 10x larger compute budget, the optimal resource allocation suggests scaling model size by 5.5x and data by 1.8x to achieve a maximal loss improvement.

Hoffmann et al. (DeepMind)

Kaplan scaling laws remained state-of-the-art until ~2022, when Hoffmann et al. published a new compute-optimal scaling framework called Chinchilla. The headline here is that Kaplan scaling results in models that are under-trained (i.e., models had too many parameters and not enough training data), while Chinchilla suggests you should use many more data tokens and a smaller model than previously thought. Hoffmann et al. posited that Kaplan et al. came to this conclusion because they didn’t train large enough models and that they used a suboptimal fixed learning rate schedule.

During pretraining, an LLM repeatedly sees small batches of text from its training dataset. For each batch, it predicts the next token, compares that prediction to the actual token, and computes a loss. Based on the magnitude of that loss, the model updates its weights, either by a small or large amount. This process is called a training step and the rate by which the model weights are updated is the learning rate schedule.

Smaller adjustments to the weights can make training more stable, but requires a longer time to converge the model to its final state. Larger adjustments may accelerate learning, but there is a risk of overshooting, leading to model instability and overall poor model performance.

Kaplan et al. used the same cosine learning-rate schedule for every model and dataset size – small updates at the start, large in the middle, and small again at the end. But a cosine schedule only works if you train for the full dataset it was designed for.

Hoffmann et al. argued that when you use a cosine schedule designed for a large dataset on smaller datasets, the training stops early while the learning rate is still high. The smaller models never get to the final low-rate “settling” phase, meaning the weights never converge to optimal performance. This likely altered Kaplan’s conclusions about how to effectively make trade-offs between model and dataset size.

In the Hoffmann et al. paper, the authors trained across a fixed range of model sizes (75M to 10B) and varied the number of training tokens, measuring the loss at several steps throughout the pretraining process. To overcome limitations from the Kaplan et al. paper, they used four different learning rate schedules for each model size. They then computed the model size and training token count that led to the lowest validation loss at each FLOP budget.

As shown below, we can see that all model sizes and dataset permutations result in a quickly improving loss that starts to plateau with higher compute. Diminishing returns occur first with smaller models and later with larger models — the latter being able to access regions of much better absolute loss (e.g., ~2.0 for 10B parameters).

Critically, the Chinchilla-optimal scaling framework arises from increasing pretraining tokens and model parameters at an optimal ratio of 20:1. For reference, GPT-3’s largest model was trained in the Kaplan era with ~1.7 tokens/parameter.

Sardana et al. (Databricks)

In an attempt to reconcile the scaling laws from Kaplan et al. and Hoffmann et al., Sardana et al. published a paper at last year’s ICML with a careful eye toward model scaling from a cost-economic standpoint. The main takeaway was that if a company expects a frontier model to have a high level of inference demand, it’s more economical to train a smaller model with more data and compute until the desired performance is reached. This is because smaller models will require less compute to inference compared to larger models, independent of how much compute was used to train the model in the first place.

While pretraining this smaller 7B model is Chinchilla-suboptimal (and more expensive), the smaller model is cheaper to run at inference, lowering the operational expenses associated with serving the model to customers. Comparing a 7B and 13B parameter LLM and 2T tokens of inference demand over the life cycle of the model, you can save ~17% (1.7 x 10^22 FLOPs) in compute. For frontier models, that can mean millions of dollars in total savings.

Using this insight, the group at Databricks generated a series of plots that can assist with resource allocation decisions.

1. On the left plot, if we wanted to train to a loss of 2.4 and we expect a lifetime model inference demand of 10^12 tokens, we land on the 2x FLOPs ratio band – meaning that a Chinchilla-optimal model would actually cost 2x more than if we were to overtrain a smaller model.

2. In the middle plot, for the same criteria, we can see that the intersect lands on the 33% band – meaning that a compute-optimal model would have 33% of the parameters of a Chinchilla-optimal model.

3. On the right plot, the intersect lands on the 5x band – meaning that the ratio of pretraining tokens to parameters should be 5x larger than that of a Chinchilla-optimal model. Roughly 100 training tokens per parameter.

The field has seen this ratio explode in the last couple of years as it has become more popular to train models beyond the 20x Chinchilla scale to optimize for inference costs. For example, Llama-3-70B was trained on over 200 tokens per parameter.

Sardana et al. formalized this trend to show that small models can improve their pretraining loss well beyond the Chinchilla-optimal envelope – even out to 10,000x tokens/parameter. However, they do so at an exponentially decaying efficiency. In the log-scale plot, we can see that the training token-parameter ratio must increase exponentially to maintain a linear improvement in loss. This is true across model scales and poses a significant issue given that we may be running out of the world’s high-quality data.

Can Scaling Dataset Quality Offset Limited Data Quantity?

The pretraining scaling laws developed over the past few years make one thing clear: abundant data is essential for building better, more efficient LLMs. But these studies have largely sidestepped a key question. Can data quality also bend these scaling curves to further optimize model performance in a resource-constrained environment?

The answer seems to be yes – models are what they eat and cooking up Michelin-star data is likely going to be a key consideration for future development.

Subramanyam et al. demonstrate this concept well by extending the Chinchilla framework to include a data quality score (Q) and assessing the relationships between model size, dataset size, data quality, and pretraining loss. Here, Q can be thought of as the fraction of high-quality data in the dataset, where high quality refers to uncorrupted, non-redundant text data.

The authors show a clear trend where reduced data quality leads to increased test loss. Interestingly, for tasks like next-token prediction, scaling dataset size from 0.1-10B reduces the model’s reliance on high-quality data. In other words, moderate degradation in data quality can often be offset by training on more total data. Nonetheless, introducing higher-quality data during training is still a relevant knob to increase performance in smaller models.

The authors also show that you can get away with training on less data – while still achieving the same test loss – by increasing data quality.

Given these scaling laws, there is a clear need for extracting the highest value per token out of proprietary datasets. In sectors that generate a significant amount of proprietary unstructured data (e.g., customer support centers with billions of call transcripts or insurance providers with historical unlabeled text claims), organizations often have more unlabeled data than they could ever reliably use given compute and cost limitations. As a result, they typically rely on random sampling of their datasets, which significantly limits training efficiency and final model performance.

Data curation companies like DatologyAI have developed custom data pruning strategies (e.g., deduplication, model-based filtering, augmentation with synthetic data) to create these high-quality training datasets.

By tiling across pruning fractions, they can experimentally identify the best achievable error across all pruning fractions at each dataset size (i.e., the Pareto front). Notably, the slope of the Pareto front (red) is steeper than the baseline power law scaling (black/purple) when 100% of data is kept in the training set. This illustrates that prioritizing higher-quality training data can result in better-performing models – even though that may mean training on less total data.

Which LLM Benchmarks are Most Common and Useful?

Most of the scaling law data presented so far has focused on validation or test loss as the key optimization metric. However, validation loss is far removed from industrially-relevant tasks and the benchmarks aimed at mimicking them.

LLM benchmarks can be crudely subdivided into a few categories – public, private, and personal benchmarks – that are often scattered across domains like language, math, chatbot assistance, coding, and tool/function calling. Each has its own trade-offs, which complicates unbiased evaluation of models and can often limit interpretability of results.

Public benchmarks (e.g., MMLU, MATH, HumanEval) openly publish their datasets, test suites, and scoring rules, ensuring a level of transparency and standardization across model comparisons. But they can also suffer significantly from data leakage and overfitting (i.e., models seeing test data during training and memorizing benchmarks instead of learning general skills).

Private benchmarks – often developed by frontier labs – offer company-specific evaluations that are resistant to memorization and web exposure, but cannot be easily reproduced and compared across labs.

Personal benchmarks are more vibes-based evals created by each user’s unique interests and work priorities. As a result of their hyper-personalization, these benchmarks may not be useful to the community at large but will likely become more meaningful to users as frontier models continue to become more ingrained in our personal data and applications.

During the formative years of pretraining, early public benchmarks like MMLU (general reasoning) and HumanEval (coding) were popular, but are quickly becoming obsolete. Modern LLMs have become so performant that these more trivial benchmarks are effectively saturated, making them poor barometers for sustained model improvement.

Today, some of the most common benchmarks for frontier LLMs are GPQA Diamond, AIME, and SWE Bench. These evaluate quality and reliability across graduate-level science, Olympiad-level high school math problems, and the ability to resolve real GitHub issues using agentic capabilities, respectively.

As competition for topping benchmark leaderboards intensifies, frontier labs often publish a laundry list of these benchmark scores comparing their latest models to each other. See the most recent Gemini-3, Claude Opus 4.5, and DeepSeek-V3.2 launch materials. Given the diversity of metrics, it can be hard to quantify the overall capability of the models by simply looking at each benchmark in a list – especially how they change over time.

Epoch AI aimed to simplify model comparison by publishing their own Epoch Capabilities Index (ECI), an aggregation of 39 distinct benchmarks into a single score weighted by relative difficulty. Indexes like these provide a more holistic view of model performance and may be more robust to outlier capabilities of one model versus another.

But the ultimate unanswered question is whether any of these benchmarks actually predict any real-world value. While the “dollar per benchmark score” is often unclear, a few studies have attempted to map model improvements directly to labor costs.

For example, Windsurf’s software engineering coding model SWE-1.5 achieved comparable performance to Claude Sonnet 4.5 on SWE Bench, but at 13x the speed. They estimated this speed boost results in a monthly savings of $3-10k in developer time for a 5-person team.

These types of direct benchmark-to-dollars-saved mappings add critical credibility to these scores and should be prioritized as we continue to push these models into the workforce.

Is Validation Loss Predictive of Downstream Benchmark Performance?

So far, we’ve described how scaling pretraining has resulted in predictable, albeit resource-intensive, improvements in upstream validation loss (i.e., next-token prediction). But does reduction in validation loss successfully predict improvement in downstream benchmark performance?

This summer, researchers at Prescient and NYU ran an experiment that produced a surprising result. They found that of 46 common downstream LLM benchmarks, only 39% exhibit predictable improvements when validation loss is reduced. These results are broken down into categories within the pie chart below, alongside examples of what each of these trends (or non-trends) looks like across a set of three different validation datasets.

Given these results, it’s clear the predictiveness of pretraining loss for downstream task performance is complicated. There are two interesting categories of model behavior worth exploring more: inverse scaling and breakthrough performance.

1. Inverse scaling suggests that at some improved loss, downstream task performance passes its optimum and begins to worsen. This can be due to a few different reasons. For example, the pretraining loss is measured on data that simply doesn’t align with the downstream task – a mid-size model might “accidentally” perform well on adversarial multiple-choice questions or logic puzzles because it’s using heuristics to pick the most unusual answer, but larger models index much more to the most common sounding answer which may actually impair model performance in the long run.

   While this isn’t the case for every model and every task, it does imply that not all tasks benefit from larger models – there can be substantial product-market fit for smaller models as well. Being able to recognize and exploit this feature can also help providers serve these models much more efficiently.

2. Breakthrough performance (also often referred to as emergence) is the idea that there is a discontinuous step change in benchmark performance at some point during model training. When originally presented, the concept of emergence brought significant concern to the field as downstream performance felt unpredictable. However, elegant work by Schaeffer et al. showcased there may actually be nothing to worry about. Instead, “emergence” is the result of poor experimental design and execution – selecting nonlinear (% accuracy) or discontinuous (multiple choice) metrics with limited data points can trick researchers into mistaking low-resolution graphs for breakthrough results.

What Does the Future Hold for Scaled Pretraining?

We’re in a tricky spot. We now know that scaled pretraining results in improved validation loss and that this improvement in loss can confer a notable downstream benefit. Unfortunately, the benefit is clean in only a minority of cases – which isn’t ideal for large capital allocation decisions and highlights the usage idiosyncrasies that accumulate near the end consumer (e.g., preferring ChatGPT for everyday knowledge questions and reasoning tasks, while using Claude for coding and creative writing).

It’s also clear that achieving improved validation loss solely via pretraining is becoming exponentially more difficult and that the next order-of-magnitude unlocks are running up against some very real physical barriers that could begin to impact frontier LLMs before the end of this decade.

Beyond investors’ willingness to fund AI capex investments in the billion-dollar range, we are running out of training data and running into training length obstacles. Given a lack of infinite compute resources, we need to prioritize where best to allocate resources from one project/model to another.

Epoch AI has written about both of these dynamics. Let’s start with training data.

They estimate there are roughly 300T tokens of human-generated, public text available for training AI models. Leveraging our previous insight about overtraining to improve inference-amortization (i.e., post-Chinchilla; fewer parameters and more data), we may exhaust our public training stock near 2028. As discussed above, data quality will continue to become an increasingly important parameter to offset this data scarcity. It should also be noted that other data modalities like images and videos can help supplement this data, but these are often easier to collect and likely not the bottleneck for modern LLMs.

An alternative is incorporating synthetic data (i.e., data produced by advanced LLMs to supplement real-world data). The jury still seems to be out on whether synthetic data can make a large enough dent in our stockpile to sufficiently offset this timeline. The main goal here is to produce novel data that expands the training distribution, rather than just reinforcing data that is already represented in the training dataset. At least for now, synthetic data has been shown to be meaningful in domains such as coding and math, where accuracy of this out-of-distribution data is more easily verifiable. The utility of synthetic data is a topic we will likely expand on in future essays given its potential to upend this data scarcity problem.

Next, Epoch AI considered obstacles for training length.

Given the rate of hardware and algorithmic improvement, there’s a very real scenario wherein the length of training also becomes a bottleneck. The analysis below predicts training runs can last up to ~9 months before they potentially become inefficient, and we are on pace to meet this milestone by 2027.

To make this more concrete, imagine a scenario where halfway through a 9-month training run, the hardware and algorithms you’re using to train become obsolete. Will this deter frontier labs from making capex investments, effectively freezing the market? Or, will a new wave of ML infrastructure enable the smooth, dynamic reallocation of capex along a training run’s life cycle – effectively building the plane as we fly it? These are still very open questions within the field that will likely define the next era of model training.

Pretraining is Dead(?)

At last year’s NeurIPS conference in Vancouver, former OpenAI co-founder/chief scientist and current Safe Superintelligence founder/CEO, Ilya Sutskever pronounced that “pretraining as we know it will unquestionably end. We’ve achieved peak data and there’ll be no more.”

Given the discussion above, we tend to agree. We can only battle finite data volume/quality, long training timelines, and costly compute/power infrastructure for so long before the pace of model improvements starts to substantially diminish.

While a tantalizing premise, it’s likely safe to say that we cannot simply put the entire universe in distribution to solve our problems. Aside from being impossible from a data and infrastructure perspective, these models would still suffer from rigidity and a constant need to retrain as time advances and new data becomes available.

Fortunately, the field has correctly forecasted these trends and over the last several years has implemented new training techniques to keep performance gains on a steady pace – namely through the introduction of post-training methods and reasoning capabilities.

Reasoning methods aim to teach a model how to logically step through complex problem solving. Their implementation has exploded over the last year with the introduction of OpenAI o1 and DeepSeek R1 – simultaneously unlocking an entirely new scaling regime for model development with few signs of slowing down. This topic will be the focus of our next article.

But does all this mean that pretraining is truly “dead”? Have we overcorrected on pretraining and should we instead shift our focus entirely to optimizing and scaling post-training methods? For several reasons we will continue to explore, we think there is more nuance here, and pretraining will likely remain one of many important knobs to turn in our quest to build more performant and useful AI.

Subscribe now

Over the past few decades, few therapeutic areas have drawn as much skepticism across biopharma, research, and investment circles as neurology.

When looking at the numbers, this hesitation may be warranted. The average probability of a central nervous system (CNS) drug making it from Phase I to market is just 5.9% — the 4th lowest by therapeutic area (TA). The average time to approval is 11.1 years — the 3rd longest by TA. Moreover, unlike other indication spaces such as oncology and immunological diseases, the pace of new drug approvals in CNS has been relatively stagnant over the last 3 decades.

This reality is most visible in neurodegenerative diseases such as Alzheimer’s and Parkinson’s. For decades, the dominant therapeutic approach has focused on eliminating the hallmark protein aggregates (e.g. amyloid beta (Aβ), tau, α-synuclein) thought to drive disease progression. Yet in the last five years, this hypothesis has been at the center of high-profile article retractions, numerous clinical trial failures, controversial drug approvals, and subsequent discontinuations.

Today, despite decades of effort, only two disease-modifying therapies targeting protein aggregation are approved for Alzheimer’s. While these approvals suggest protein aggregation remains an important piece of the therapeutic puzzle, their overall cognitive benefits are modest, and their use is further clouded by characteristic safety issues that can prevent long-term adherence. Moreover, there are still no approved disease-modifying treatments for Parkinson’s.

Given these data, it’s easy for many outside the neuro field to develop a negative gut reaction surrounding the future of CNS drug development. I’ll be the first to admit that even as someone who dabbled in neuro research as a wet-lab scientist, there was a large part of me that felt this way too. But after spending more time diving deeper into the field’s history and emerging biological hypotheses, my perspective has shifted.

Through the lens of an initial skeptic turned hesitant optimist, this article explores why neurology drug development has been so difficult, how CNS investment by pharma and venture capital (VC) has evolved over time, and one emerging therapeutic hypothesis that could catalyze the development of novel disease-modifying therapeutics.

Subscribe now

Table of Contents

1. Why should biotech, pharma, and VC be investing in neuro research?

2. Why is CNS drug development so difficult?

3. How has CNS investment interest by pharma and VC evolved over time?

4. What could spark renewed CNS investment by pharma?

5. The promise of targeting neuroinflammation

6. Clinical progress towards drugging neuroinflammatory targets

7. Outlook

Why should biotech, pharma, and VC be investing in neuro research?

From a scientific and human health perspective, advancing our understanding of fundamental neurobiology and using that knowledge to develop treatments for CNS disorders is undoubtedly valuable to society.

However, from an investment standpoint — across biotech, biopharma, and VC — the prioritization of neurology can become less clear. With limited resources, investors and drug developers are forced to weigh which indications and therapeutic hypotheses should be prioritized, whether fundamental biological insights are mature enough to translate into clinical therapies, and if the field is at the right point on the technological J-curve to encourage investment.

In the current risk-off market, biotech investors and biopharma often favor indication areas with high probabilities of clinical success, relatively fast development timelines, and large market potential. As noted above, CNS disorders have historically scored poorly on the first two measures, which can explain some of the disproportionate deprioritization of this space.

But the scale of the market opportunity and the substantial burden CNS disorders have on patients and broader society are significant, and there is increasing evidence that the market is growing out of an over decade-long slump. Specifically, the broader CNS market is poised to surpass $80B in 2025, corresponding to an ~8% year-over-year growth.

Perhaps even more striking is the socioeconomic burden. CNS disorders are the leading cause of overall disease burden worldwide. Collectively, they affect 43% of the global population and are responsible for 443M disability-adjusted life years (DALYs) — an increase of 18.2% over the last 30+ years. To put this into context, in 2019, it was estimated that early-onset cancers account for ~250M DALYs globally.

Between a growing market, massive socioeconomic impact, and heavy burden on patients, the question isn’t whether opportunity exists in CNS, it’s which catalysts can overcome the field’s unique developmental challenges, accelerate the approval of disease-modifying therapies, and spark broad investor interest.

Why is CNS drug development so difficult?

For many CNS disorders, current standard-of-care (SoC) therapies are decades old and often only manage symptoms. For example, in Parkinson’s disease, the dopamine precursor levodopa/carbidopa (Sinemet) has remained SoC since its approval in 1975, providing relief of motor symptoms but without altering disease progression. Similarly, propranolol, a nonselective beta blocker originally approved in 1967, is the only FDA-approved treatment for essential tremor.

Progress in CNS drug development has generally been slow because the brain poses a unique set of scientific, clinical, and logistical challenges that make drug development significantly more complex than most other indication areas:

1. Our fundamental understanding of neurobiology is limited compared to other tissues.

   The CNS is arguably our most complex organ system, comprising billions of neurons with trillions of dynamic connections. Access to human tissue is scarce and tools for non-invasively measuring and manipulating its function are limited. This is further compounded by a lack of representative in vitro and in vivo preclinical models to study complex disease phenotypes that are often driven by several cell types — making novel target discovery and validation of new therapeutic hypotheses even more difficult.

2. CNS tissue has limited regenerative capacity.

   Because mature neurons are post-mitotic, damage from disease or injury is often permanent. This limits the potential for reversing late-stage disease and focuses most therapies toward only slowing progression. Moreover, many CNS diseases do not present symptoms until irreversible damage has already occurred, complicating overall treatment strategies.

3. Clinical trials for CNS disorders are challenging to execute.

   CNS diseases are heterogeneous and vary significantly across patients due to differences in genetic and environmental causes. As a result, there are relatively few reliable biomarkers that clearly link target engagement to clinical outcomes. Long disease courses and strict inclusion criteria make trials especially lengthy and expensive.

4. The blood-brain barrier (BBB) selectively limits drug delivery to the brain.

   The BBB is a protective barrier of several cell types — most notably endothelial cells — and extracellular matrix that limit the passage of soluble molecules and cells between the peripheral circulatory system and CNS tissue. It’s estimated that only 2% of FDA-approved small molecules and <1% of an IV administered antibody dose cross the BBB. Researchers are working on both passive and active methods of drug delivery across the BBB, but our understanding of the optimal design heuristics is limited.

How has CNS investment interest by pharma and VC evolved over time?

After decades of searching, pharma finally found its initial breakout class of neurology drugs in the late 1980s with the FDA approval of Eli Lilly’s Prozac, a selective serotonin re-uptake inhibitor (SSRI) for treatment of major depression. By 1990, Prozac became the best-selling antidepressant of all time, and by 1998, it reached $2.8B in annual sales, accounting for a quarter of all Eli Lilly's revenue.

Several factors contributed to Prozac’s widespread success, including its relatively safe toxicity profile, convenient once-daily oral dosing, broad potential for indication expansion, and overall cultural resonance. But perhaps the most important factor for the neuro field at large was the breakthrough in validation of a new therapeutic hypothesis delivering clear efficacy and safety benefit to patients.

This novel biology sparked a wave of interest and investment across pharma — several developing their own “me-too” SSRIs, including Pfizer’s Zoloft in 1991 and GSK’s Paxil in 1992. Zoloft’s sales would ultimately go on to eclipse Prozac’s with a peak of >$3B in 2004. Soon after, the field expanded to related biological mechanisms, including serotonin-norepinephrine re-uptake inhibitors (SNRIs) such as Wyeth/Pfizer’s Effexor (1993) and Eli Lilly’s Cymbalta (2004).

However, by the mid-2000s, the market began to shift. The SSRI/SNRI space had grown increasingly crowded, and several early drugs lost patent protection, opening the door for generics to capture significant market share. Outside of psychiatry, the challenges of translating therapies across other CNS diseases came back into full view with years of clinical setbacks. For example, out of a total of 244 compounds tested in Alzheimer’s disease clinical trials from 2002-2012, only 1 (0.4%) gained approval. At the same time, while our biological understanding of CNS diseases stagnated, rapid advances in fields like oncology and immunology began to draw pharma attention and investment elsewhere.

As a result, CNS pipelines shrank sharply between 2009 and 2014 with 11 major pharma reducing collective drug programs by 52% (267 to 129). Since then, several pharma have doubled down and announced major pullbacks from investment into CNS programs, including Amgen, Pfizer, and AstraZeneca.

“Upon careful evaluation of our pipeline and the challenges inherent in developing drugs for major neurologic diseases, we’ve made the decision to end our neuroscience research and early development programs…” — Amgen R&D Head, David Reese, October 2019

“We have made the decision to end our neuroscience discovery and early development efforts and re-allocate funding to those areas where we have strong scientific leadership and that will allow us to provide the greatest impact for patients.” — Pfizer, January 2018

“We cannot be everywhere. [CNS is] probably better managed by other companies that have a focus on that.” — AstraZeneca CEO, Pascal Soriot, May 2025

It’s unfair to say that all large pharma have pulled back from neuro investment. The table below shows several active players. That said, many of these pharma have scaled down from the peak levels of ~25 years ago, concentrating their portfolios on major neurodegenerative diseases such as Alzheimer’s, Parkinson’s, and multiple sclerosis (MS), while deprioritizing areas like psychiatric disorders, depression, and pain.

Interestingly, while pharma largely deprioritized neuro during this time, biotech venture investment remained relatively resilient — consistently ranking as the 2nd or 3rd highest therapeutic area by dollars — showing that investor sentiment did not necessarily shift in the same way as pharma. It’s impossible to assign a singular reason for this funding trend, but the combined large patient population and active mergers and acquisitions (M&A) market may have been enough to keep VC investors interested.

But when investment is normalized by the number of FDA approvals, the difficulty of developing therapies in this space comes back into view: neurology ranks as the third-lowest therapeutic area for approved drugs per $1B invested from 2014-2024.

What could spark renewed CNS investment by pharma?

You don’t have to squint much to notice the cyclic nature of drug development, especially in CNS. The initial spark of success with new targets and biological mechanisms for SSRIs clearly invigorated pharma investment at the turn of the century. If history seeks to repeat itself — and I think it will — the clinical validation of new therapeutic hypotheses that demonstrate disease-modifying impact may help reset expectations and redefine opportunities in CNS drug development.

In particular, neurodegenerative diseases like Alzheimer’s and Parkinson’s are among these areas overdue for fresh therapeutic hypotheses. They have long been the poster children for the difficulty of translating safe and effective therapies in neurology, and the limited clinical benefits from current approved drugs suggest there might be alternative disease mechanisms to target.

While exploring the neurology space, I kept coming back to the graph below of new drug approvals over the last 3 decades and how therapies for immunological diseases had more than doubled decade-over-decade, while neuro and psychiatry stayed relatively constant. One interpretation of this data is that decades of growing fundamental knowledge in immunology and our ability to manipulate it created the stored potential energy that, once released, drove a burst of rapid clinical translation.

Despite the similarly high complexity of the immune system, there are several potential reasons why the slope of progress in immunology has been generally faster than in CNS. For example, many immune cell types are easily obtained from a simple blood draw, can be reliably cultured and expanded ex vivo, and are directly targetable through IV drug delivery. Additionally, technologies like flow cytometry have enabled deep characterization of immune cells, and humanized mice have provided more therapeutically relevant preclinical models.

Just as researchers have targeted or leveraged the immune system in other indication areas (e.g. PD-1/PDL-1 inhibitors for solid tumor treatment), directly targeting the CNS immune system could open a path to disease-modifying therapies for Alzheimer’s and Parkinson’s, and early evidence linking immune pathways to neurodegeneration suggests this translation may already be within reach.

The promise of targeting neuroinflammation

Admittedly, I left out an important caveat from David Reese’s quote above on Amgen moving away from neuro R&D:

“Upon careful evaluation of our pipeline and the challenges inherent in developing drugs for major neurologic diseases, we’ve made the decision to end our neuroscience research and early development programs with the exception of programs centered on neuro-inflammation.” — Amgen R&D Head, David Reese, October 2019

The fact that large pharma like Amgen were cutting most early-stage neuro programs but making exceptions for neuroinflammation, in my opinion, makes this area especially compelling to explore more deeply.

While the BBB typically limits cross-talk between the peripheral immune system and the brain, two CNS-native cell populations (i.e. microglia and astrocytes) carry out many similar immune-related functions. For example, microglia primarily respond and release inflammatory cues, clear debris, and prune weak connections (i.e. synapses) between neurons, whereas astrocytes also contribute to inflammatory cytokine production and synaptic pruning, in addition to maintaining BBB integrity.

Until recently, neuroinflammation in neurodegenerative diseases was seen mainly as a side effect of microglia and astrocyte activation in response to pathogenic protein aggregates and neuronal debris. However, more recently, there has been an ideological shift toward viewing neuroinflammation as a primary driver and trigger of neurodegeneration — driven by growing genetic links between microglia and Alzheimer's/Parkinson's, links between autoimmune and neurodegenerative disease incidence, and studies showing that long-term NSAID (i.e. anti-inflammatory) users had a 50% risk reduction of developing Alzheimer's.

In Alzheimer’s and Parkinson’s, genetic and environmental factors can drive astrocyte and microglia dysfunction, fueling a self-perpetuating cycle of inflammation, neuronal loss, and cognitive decline. For example, microglia respond to protein aggregates like amyloid-β and α-synuclein by releasing cytokines and complement proteins that damage neurons and prune synapses, while astrocytes lose homeostatic functions such as clearing excess glutamate and potassium, causing excitotoxic injury to neurons. This chronic inflammation also weakens the BBB, allowing infiltration of peripheral immune signals and cells that further damage neural tissue.

Clinical progress towards drugging neuroinflammatory targets

Today, several drug targets (e.g. TYK2/JAK1, TREM2, LRRK2, NLRP3, NEK7, C1q, KCNK13, S1P-R) are being explored for the treatment of neurodegenerative diseases in clinical trials, with many aimed at reducing chronic inflammation and redirecting microglia and astrocytes from an over-activated to homeostatic phenotype. While it’s still too early to know whether these targets will successfully translate and drive broader CNS investment (most are only in early Phase II trials), several promising candidates are worth discussing.

TYK2/JAK1

One of the most advanced programs in this area borrows target biology directly from the peripheral immune system. BHV-8000, an oral, brain-penetrant dual TYK2/JAK1 inhibitor from Biohaven, is being developed for Parkinson’s disease with the aim of blocking pro-inflammatory IFN signaling, reducing Th17 cell proliferation, and breaking the feedback loop of chronic neuroinflammation. After reporting a well-tolerated safety profile in healthy volunteers in Phase I trials, Biohaven recently dosed its first patient in a Phase II/III trial in May 2025.

Both JAK1 (e.g. Rinvoq, AbbVie, 2019; Xeljanz, Pfizer, 2012) and TYK2 (e.g. Sotyktu, BMS, 2022) have already been independently validated in several peripheral autoimmune and inflammatory diseases. These prior results offer not only confidence in the targets themselves but also provide valuable insights from past trials that could help increase the probability of translating these approaches to CNS diseases.

For example, a major roadblock in CNS drug development has been the lack of reliable biomarkers to assess target engagement. One advantage of leveraging known immune biology is the ability to borrow validated biomarkers from other peripheral immunology trials: BHV-8000’s Phase I data showed significant reductions in TYK2/JAK1-related inflammatory markers (e.g. IP-10, hsCRP, IFN-beta) versus placebo, providing greater confidence in target engagement heading into later-stage trials.

TREM2

TREM2 is another well-studied neuroinflammation target, in part because the loss-of-function R47H variant is one of the strongest genetic risk factors for Alzheimer’s. Specifically, TREM2 is expressed on the surface of microglia and is activated by phospholipids, protein aggregates, and apoptotic debris that triggers a neuroprotective state with decreased inflammatory cytokine production. The therapeutic hypothesis is that TREM2 agonism could boost microglial survival and phagocytosis to clear the protein aggregates that fuel chronic inflammation and drive the perpetual cycle of neuronal cell death.

Despite early promise, TREM2 agonists have been plagued by clinical trial failures due to both poor safety margins and limited efficacy (e.g. Alector, Denali, Vigil). While the exact reasons behind these failures are still up for debate, nuances around genetic variation and disease stage may play a role.

Nonetheless, a recent acquisition of Vigil Neuroscience by Sanofi for $600M for their Phase I oral, small molecule TREM2 agonist demonstrates the field still sees some promise.

There is a likely reason for Sanofi’s optimism here. Many antibody-based TREM2 agonists struggle to distinguish between the membrane-bound receptor and its soluble form, which can act as a sink and reduce effective target engagement. To address this, Vigil’s small molecule targets only membrane-bound TREM2, allowing it to avoid the therapeutic sink. Notably, Vigil’s TREM2 antibody agonist that was discontinued after a failed Phase II clinical trial for ALSP, a rare neurodegenerative disease, was not included in the sale.

LRRK2

LRRK2 is a kinase expressed in both neurons and glial cells and is one of the most common genetic risk factors for Parkinson’s disease. Biologically, it plays a key role in regulating lysosomal function, and its over-activation in Parkinson's is thought to impair waste clearance, resulting in the buildup of α-synuclein aggregates that promote neurodegeneration and drive downstream neuroinflammation.

In the past year, LRRK2 has gained major attention as Denali/Biogen and Neuron23 each began Phase II trials of their LRRK2 inhibitors, dosing the first patients with LRRK2-associated Parkinson’s disease. Notably, the same Denali/Biogen drug is also currently being tested in a Phase IIb trial for Parkinson’s patients with or without an LRRK2 mutation that began in May 2022 and has an expected readout later this year.

Early pharmacodynamic studies from Denali/Biogen’s Phase Ib trial in Parkinson’s patients showed encouraging results. Specifically, phosphorylation of LRRK2 in whole blood decreased in a dose-dependent manner versus placebo, confirming target engagement. Time will tell whether these results translate to meaningful efficacy readouts in current Phase II studies.

Outlook

It’s probably unrealistic to expect that any single target, therapeutic hypothesis, technology, or company will simultaneously solve all the challenges that plague CNS drug development. Long-term progress will likely require steady investment in breakthroughs that, together, create the rising tide that lifts all boats.

That said, there are several lessons we can learn from our past success in targeting the peripheral immune system that can guide new therapeutic hypotheses and catalyze drug development for CNS disorders:

1. Novel targets and improved biological understanding.

   Given the limited success of current neurodegenerative treatment strategies, broadening the target space to immune-related pathways will help promote exploration of new therapeutic hypotheses and deepen our understanding of fundamental neurobiology and CNS disease mechanisms.

2. Borrowing biomarkers from I&I.

   The wide range of immunological biomarkers and our improved understanding of immune cell pathways have undoubtedly increased the probability of success in I&I drug development. Targeting brain-resident immune cells such as microglia and astrocytes — which share many signaling pathways with peripheral immune cells — could allow the use of the same validated PD biomarkers in neurology.

3. Avoiding the BBB altogether.

   Crossing the BBB remains a major challenge for CNS drug development. However, the field is beginning to understand how peripheral immune signals — both soluble factors and cells — also influence neurodegeneration as they pass through the compromised BBB. This may open the possibility of treating CNS diseases by targeting peripheral immune signals without needing to cross the BBB in the first place.

4. Expansion beyond neurodegeneration.

   We know the immune system can be uniquely targeted to treat diverse disease areas like cancer, fibrosis, and metabolic diseases. The same principle likely applies to the CNS, and the potential to target neuroinflammation extends beyond the initial push into Alzheimer’s and Parkinson’s. Just as Prozac and Zoloft quickly expanded to multiple indications, restoring immune cell homeostasis could also benefit other CNS disorders, including multiple sclerosis and autism spectrum disorder.

One lingering concern I have is whether targeting neuroinflammation can truly reverse neurodegeneration rather than just slow or stabilize it. I have yet to see much evidence that disrupting the pro-inflammatory positive feedback loop would necessarily also catalyze neuronal regeneration, but we need more data, and likely more regenerative approaches for treating late-stage patients. This also underscores the perennial need for early-detection technologies that can avoid irreversible neurodegeneration in the first place.

Overall, it’s still relatively early days for targeting neuroinflammation, and there are several fundamental unknowns to address including identifying which cell types, pathways, and drug targets can deliver durable, disease-modifying outcomes amidst the complexity of CNS inflammatory feedback loops. A clear clinical success validating a therapeutic hypothesis like neuroinflammation could be the turning point that reframes the outlook for neurodegenerative disease treatment and neuro writ large.

If you are working in this space and have any comments/opinions about neuroinflammation or other emerging therapeutic hypotheses in neuro, feel free to reach out!

Members of our team recently spent a week in Chicago for the American Society of Clinical Oncology (ASCO) 2025 conference. We attended poster sessions, walked the exhibition floor, and listened to many structured sessions. Below, we’ve catalogued some of the themes and ideas that stuck out to us. Please feel free to debate, discuss, or reach out to us if you’d like to attend our reception at a future ASCO.

1. Broader and Earlier: New Horizons for Immune Checkpoint Inhibitors

2. TCR-Inspired Therapies: A Gold Rush on Intracellular Targets

3. First-in-Human: mRNA-Encoded T Cell Engagers (TCEs)

4. Next-Gen Hormone Therapy and Beyond: The Future of Advanced Breast Cancer Management

Subscribe now

Broader and Earlier: New Horizons for Immune Checkpoint Inhibitors

Already mainstream medicine, immune checkpoint inhibitors are poised to expand into earlier stage disease and could benefit patients when administered before and after surgery.

Throughout the last 15 years, immune checkpoint inhibitors (ICIs) have cemented themselves as a defining pillar of cancer immunotherapy treatment, beginning with the FDA approval of anti-CTLA-4 therapy in 2011 and subsequent approvals for anti-PD-1, PDL-1, and LAG-3. Specifically, ICIs function by blocking these aforementioned inhibitory cell-surface receptors, facilitating activation of T cells upon encountering target cancer cells.

To date, ICIs’ therapeutic impact has been observed across more than 20 tumor types are predominantly approved to treat late-stage, metastatic disease. These patient populations often serve as a first entry point for novel therapeutic mechanisms given faster regulatory approval pathways and a greater risk tolerance for side effects (such as the immune related adverse events often associated with ICI therapy).

A major trend we observed at ASCO this year was the desire to expand the current clinical success of ICIs into earlier stage patient populations with new dosing and combination strategies. Here’s an overview of two clinical trial readouts we believe are good reminders of the significant life left for ICI treatments in treating new patient populations:

ATOMIC—Are ICI therapies effective in treating pre-metastatic colon cancer in combination with standard of care chemotherapy?

The current standard of care (SOC) for stage III colon cancer is adjuvant chemotherapy; however, ~30% of these patients still see tumor recurrence. ATOMIC’s Phase III trial aimed to reduce this recurrence rate via the addition of anti-PDL-1 (atezolizumab) to SOC chemo in patients with stage III, non-metastatic deficient DNA mismatch repair (dMMR) colon cancer.

dMMR tumors have inherently high mutational burdens, leading to the display of unique peptides on the cancer cell surface that selectively trigger immune cell activation and clearance of the tumor. This is the predominant reason why ICIs are so effective against these tumors—the key is already in the ignition and the ICI is there to release the brake on T cell-mediated cytotoxicity.

As hypothesized, these trial results strongly demonstrated that combination therapy of atezolizumab + FOLFOX6 chemotherapy significantly improves event free survival (EFS) compared to SOC FOLFOX6 alone—reducing the risk of recurrence and death in these patient by ~50% at 36 months post treatment.

While it should be noted that the ICI + chemotherapy combination did result in an expected increase in Grade >3 adverse events (AEs) compared to chemo alone, the improvement in EFS and remission rates most likely outweighs these concerns.

The ATOMIC trial—alongside many others attempting to do the same for other tumor types and ICI drugs—serves as a major stepping stone for expanding access to ICI treatment for patients with earlier-stage disease.

MATTERHORN—How does the timing of ICI treatment impact patient responses?

ICIs are currently approved for use with SOC chemotherapy in patients with metastatic gastric or gastroesophageal junction (G/GEJ) cancer where surgery is not possible. However, in local non-metastatic G/GEJ tumors, the clinical benefit of including ICIs in a perioperative (before and after surgery) setting is unknown.

The Phase III MATTERHORN trial evaluated this perioperative strategy by testing anti-PDL-1 (durvalumab) + SOC FLOT chemotherapy in patients with resectable early-stage G/GEJ cancer compared to FLOT treatment alone.

They found perioperative durvalumab + FLOT resulted in a 67% EFS compared to 59% in FLOT alone at 24 months. This data was paired with impressive pathologic complete response rates in durvalumab + FLOT vs. FLOT alone (19% vs. 7%).

Interestingly and in contrast to ATOMIC, MATTERHORN showed almost identical rates of adverse events across treatment arms, suggesting that most of the side effects resulted from chemotherapy and/or surgery.

Here’s what we’re still wondering:

• The ATOMIC trial did not include a treatment arm testing ICI treatment alone. Could the additional chemotherapy actually be dampening clinical responses to ICI if it leads to elimination of lymphocytes?

• Will the increased immune-related adverse events associated with ICIs limit the reach of these therapies into even earlier stage tumors where patient risk tolerance is low? Or will drugs against newer ICI targets (LAG-3, TIGIT, TIM-3) help improve overall safety profiles with similar efficacy?

• ICI therapies are still generally limited in their ability to combat immunosuppressive features of the tumor microenvironment and are subject to common resistance mechanisms. What combination therapies or broader strategies are most promising in overcoming these barriers to enhance ICI efficacy?

TCR-Inspired Therapies: A Gold Rush on Intracellular Targets

The vast majority of proteins are intracellular. This universe of potentially tumor-associated cancer targets has been off-limits to modern immunotherapies. But that may be changing.

About 26% of human protein targets are extracellular (~12% secreted proteins and ~14% cell-surface proteins) and thus accessible to common immuno-oncology treatments such as antibodies and CAR-T cells. Consequently, the remaining ~74% of targets are intracellular and generally out of reach to these modalities. Finding strategies to target these intracellular proteins—especially those uniquely expressed in cancer cells—will undoubtedly drive the next wave of precision immuno-oncology.

Across your tissues, cells continuously break down intracellular proteins into small peptides, which are then presented on the cell surface by human leukocyte antigen (HLA) receptors. Your body’s T cells are naturally equipped with unique T cell receptors (TCRs) trained to recognize specific cell surface peptides indicative of a diseased or infected state—leading to T cell activation and targeted cell death, as shown below.

Researchers have learned to mimic and engineer TCRs to target known tumor-specific peptides for selective elimination of cancer cells. While the field is still in its infancy, there are already clinical approvals that demonstrate the impact of these therapies including KIMMTRAK (a TCR-T cell engager) and Tecelra (an engineered TCR-T cell therapy).

This year’s ASCO meeting continued to show this field’s maturation with a compelling readout of Immatics’ IMA203. IMA203 is an autologous TCR-T cell therapy targeting the PRAME peptide/HLA-A*02:01 complex and currently being tested in Phase Ib clinical trials for advanced melanoma patients.

PRAME is an intracellular protein expressed across more than 50 tumor types and a member of the cancer testis antigen (CTA) family. CTAs are a group of >250 genes that are epigenetically silenced in adult tissues, but are active in ~40-50% of solid tumors—making them ideal cancer targets with interest across several preclinical and clinical TCR programs.

At ASCO, Immatics announced that IMA203 resulted in a 56% objective response rate (ORR) and a PFS of 6 months among 32 heavily pretreated, anti-PD-1 refractory metastatic melanoma patients. This is a significant increase in ORR compared to recently reported ICI treatments in similar melanoma patient populations (Source 1, Source 2), showcasing the potential for TCR based therapies.

As expected for patients who receive lymphodepletion therapy, the most common adverse event was cytopenia. Cytokine release syndrome (CRS) stayed manageable in most patients (84% Grade 1 and 2, and 11% Grade 3).

This impressive readout further validates IMA203’s Regenerative Medicine Advanced Therapy (RMAT) designation by the FDA, which enabled the swift initiation of a Phase III trial currently in recruitment for advanced or metastatic cutaneous melanoma. If all goes well, Immatics plans to submit for FDA approval in early 2027.

Here’s what still on our minds:

• Current TCR-based therapies are restricted to specific HLA subtypes, thus requiring novel TCR variants to treat broader patient populations. How does the TCR field more rapidly expand their reach to HLA subtypes beyond the most common alleles (e.g., HLA-A*02:01)?

• Lymphodepletion prior to injection of TCR-T cell therapies is still standard practice, yet is also responsible for serious adverse events. What strategies are being developed to mitigate side effects or eliminate lymphodepletion altogether?

• Cancer cell downregulation of HLA is a common resistance mechanism for these therapies as it removes the activation signal to effector cells. How do we combat this and other resistance mechanisms?

First-in-Human: mRNA-Encoded T Cell Engagers (TCEs)

What if there were a way to skip an entire therapeutic supply chain by turning a patient’s own liver into an antibody-producing bioreactor?

Virtually all biologic drugs (e.g., antibodies) are biomanufactured at scale in gigantic, steel reaction vessels. Once purified and packaged, we ship these therapeutic molecules around the world via temperature-controlled supply chains.

Bioprocessing has improved substantially in recent decades, especially for monoclonal antibodies (mAbs). But with more complex biologics (e.g., bispecifics) gaining a clinical foothold, it’s worth revisiting the question—can we avoid industrial biomanufacturing altogether?

Scientists have long tried to leverage the body’s internal machinery to manufacture therapeutic molecules. In 1990, one group delivered mRNA directly into mouse skeletal muscle and showed evidence of in vivo protein translation. This early proof of concept has smoldered since, limited by inadequate technology.

Nowadays, we have robust infrastructure for designing and making mRNAs at scale. Lipid nanoparticles (LNPs) for mRNA delivery have been tested in millions of people owing to vaccinations against SARS-CoV-2 from BioNTech and Moderna.

Both of these companies have repurposed their mRNA-LNP technology and know-how towards making therapeutic in vivo protein expression a reality.

Moderna will launch a pivotal clinical trial this summer for mRNA-3705, an mRNA-encoded enzyme replacement therapy for a rare disease called methylmalonic acidemia (MMA). Recently at ASCO, BioNTech release early clinical data on BNT-142, the world’s first mRNA-encoded bispecific T cell engager (BiTE) for treating solid tumors expressing claudin-6 (CLDN6).

Unfortunately, BNT-142 achieved lackluster efficacy (ORR = 5%) in all tested solid tumors. But that shouldn’t be the takeaway. BioNTech’s PK/PD and safety data lend significant insight into the potential pros and cons of encoding complex, bispecific antibody formats as mRNA.

In a dose escalation format, BioNTech infused between 50 ng/kg and 100 µg/kg of a CLDN6—CD3 BiTE encoded as an mRNA-LNP to patients (n=65) with a range of advanced solid tumors. They reported BiTE PK data between 1-100 µg/kg as shown below.

There was a dose-dependent increase in serum BiTE expression in the 1-60 ng/mL range that peaked around one day after infusion with a gradual decline resulting in a median half-life around 1.5 days. This is a curious PK niche. mRNA-encoded bispecifics seem to occupy a concentration—half-life window somewhere between an IV-delivered fusion protein and bispecific-IgG antibody.

For example, BNT-142 peaked higher (ng vs. pg) and lasted longer (days vs. hours) than Blincyto (blinatumomab)—a commercially approved non-IgG BiTE composed of a fused set of scFvs. Meanwhile, BNT-142 peaked 10x lower and lived 10x shorter than Imdelltra (tarlatamab)—an analogous IgG-bispecific approved to treat small cell lung cancer (SCLC). We wonder if the in vivo translation approach may not have enough firepower to take out a solid tumor.

Generally, FDA-approved bispecifics targeting solid tumors reach peak concentration in the µg/kg range and last for weeks. Kimmtrak (tebentafusp-tebn) is a notable counterexample. Approved for metastatic uveal melanoma, Kimmtrak has a ~10 ng/mL max concentration and a half-life of 6-8 hours—an achievable set of targets for the in vivo translation strategy.

The key question remains—how does all this PK data translate to patient safety?

Cytokine release syndrome (CRS) is a systemic inflammatory response (adverse event, AE) typically brought on by immunotherapies. CRS is common AE for T cell engagers that often requires hospitalization if it exceeds Grade 2. We’re not CRS experts, but here’s our current understanding of why a mRNA-encoded TCE may help mitigate the CRS class effect.

CRS is first triggered by the intensity and synchrony of immune synapse formation. A large and fast concentration spike likely causes a strong cytokine surge. Since the mRNA-encoded version takes longer to reach its peak concentration, it may abate the initial cytokine response—regardless of the magnitude.

It reasons CRS is also linked to drug exposure. Since the mRNA-LNP approach has a shorter tail than IV-infused IgG-bsAbs, the exposure is lower too. On the other hand, you can’t “turn off” the in vivo translated version once injected, which could pose a new challenge. Compounding this is the potential for LNP-induced toxicity. BioNTech’s PD and safety data are insightful here.

At higher dose levels, BNT-142 induces a cytokine (IL-6) surge >400 pg/mL, which isn’t markedly lower than other TCEs. Each successive cycle elicits a reduced cytokine response—similar to other TCEs that require therapeutic step-ups to reduce CRS incidence. The IL-6 drop off does appear shorter than for IgG-based bsAbs, which is encouraging and could limit exposure that may trigger CRS.

Taking a look at AEs, we can see a ~20% all-grade CRS rate, which increases to ~30% at higher doses. This compares to approved IgG bispecifics which sometimes carry a ~40% all-grade CRS rate. Critically, there’s an AST/ALT signal here that is probably LNP-induced—an issue unique to this delivery modality.

Altogether—these data are encouraging, but not dispositive. The mRNA-encoded bsAb approach seems technically feasible, logistically attractive, and at least not worse than a direct protein infusion in terms of safety. Granted, this is a single, small (n=65) study that left much on the table with efficacy. We’ll need to see more.

Here’s what we’re still asking ourselves:

• Can an mRNA-LNP approach thread the drug exposure needle enough to achieve meaningful efficacy while simultaneously lowering immune side effects like CRS?

• Will the high local drug concentration around the liver as well as LNP-induced toxicity outweigh any ostensible safety benefits?

• If both of these are true—why would this approach NOT become the future of this entire field?

Next-Gen Hormone Therapy and Beyond: The Future of Advanced Breast Cancer Management

The data revealed during ASCO plenary sessions is oftentimes practice changing. This year, advanced breast cancer took center stage—with news about emerging hormone therapies and new options for patients with treatment resistance.

Advanced breast cancer (aBC) is a scourge on society. Roughly 30,000 Americans will be diagnosed with aBC this year and another 200,000 are currently living with the disease. About a quarter of patients diagnosed with early stage breast cancer (BC) will unfortunately progress to metastatic disease, underscoring the need for better treatment options.

Over the past twenty years, however, the five-year survival rate for patients diagnosed with aBC has expanded monotonically from 25% to 34%—a final figure which doesn’t capture the efficacies of today’s best treatments.

Oncologists typically divide BC into four molecular subtypes defined by hormone receptor (HR) status and HER2 protein expression levels. Representing ~70% of annual cases, HR(+)/HER2(-) tumors are the most common subtype. Given the large unmet need, many new therapies focus on this subpopulation.

HR(+) tumors use the hormone estrogen as fuel to dysfunctionally upregulate key cell cycle proteins—resulting in uncontrolled cell growth. Oncologists can counter this by disrupting estrogen receptor (ER) signaling. Modern hormone therapy includes drugs like selective ER modulators (SERMs), selective ER degraders (SERDs), and aromatase inhibitors (AIs) that do the following:

1. SERMs competitively bind tumor-expressed ERs, blunting estrogen signaling before it reaches the nucleus.

2. SERDs antagonize ER activity and trigger the ubiquitination and subsequent degradation of ER proteins.

3. AIs halt the conversion of androgen to estrogen, thereby lowering the serum concentration of the key signaling molecule responsible for cancer growth.

Recently, a wave of FDA-approved precision therapies has earned its place alongside hormone therapy. Since 2015, several CDK4/6 inhibitors have entered into the standard of care (SOC) for first line (1L) management of HR(+) aBC.

These drugs (e.g., palbociclib, ribociclib, abemaciclib) all inhibit the activity of key cyclin dependent kinases (CDK4/6) responsible for regulating the cell cycle. Since CDK4/6 overlap with a node in the ER signaling pathway (cyclin D1), scientists reasoned there would be combination synergy with hormone therapy.

In combination with hormone therapy, CDK4/6 inhibitors have shown to improve progression free survival (PFS) and overall survival (OS) across multiple Phase 3 clinical trials, as shown below.

Unfortunately, advanced HR(+) tumors often adapt to hormone therapy. Common resistance mechanisms can be ER-dependent (ESR1 mutations) or ER-independent (GF-RTK alterations, MAPK alterations, or PI3K-AKT alterations). These genomic drivers of resistance have become the targets of a flurry of clinical development activity.

Moreover, data shown at ASCO suggest that some of these resistance mutations also have prognostic and therapeutic implications that the field will continue wrestling with for some time.

Tackling ER-Dependent Resistance with Next-Generation SERDs

The gene ESR1 encodes human ERs. Only 1% of patients harbor ESR1 mutations when diagnosed with aBC, though around 40% will develop ESR1 mutations following 1L treatment with hormone therapy.

Typically, ESR1 mutations interfere with the binding of some SERMs or SERDs, causing them to lose effectiveness. ESR1 mutations may also lock ERs in a conformation that promotes constant (constitutive) signaling even in the absence of the estrogen ligand—causing dose-dependent resistance to AIs.

Recent clinical trials, including those that read out at ASCO, have helped clarify how we should treat patients with ESR1-mediated resistance. Elacestrant was the first next-gen, oral SERD to show increased activity over SOC in patients presenting with ESR1 mutations at the time of disease progression.

The 2024 EMBER-3 trial similarly showed that imlunestrant has durable activity in ESR1-mutant disease, and increasingly so when combined with a CDK4/6 inhibitor. It’s key to note that the imlunestrant-abemaciclib doublet regimen also caused substantially more toxicity, as outlined below.

This year at ASCO, data from the SERENA-6 study of camizestrant demonstrated that patients who switched from an AI + CDK4/6i to camizestrant + CDK4/6i had a median PFS expansion from ~9 months to ~16 months. This effect was especially pronounced for patients with clonal ESR1 mutations.

Interestingly, SERENA-6 randomized patients following the detection of ESR1 mutations on a circulating tumor DNA (ctDNA) test rather when the disease progressed radiographically. We know that modern ctDNA tests can find molecular recurrence six months prior to radiographic progression, depending on the cancer.

The previous SERENA-2 trial showed that a >50% reduction in ctDNA ESR1 allele frequency was associated with a near-tripling of PFS, indicating prognostic utility for ctDNA assays at the least.

Cautiously, the boost in PFS could stem from lead time bias. It’s likely that guiding adjuvant treatment using ctDNA will prove useful, though the balance between the certainty of toxicity (especially in doublet regimens) versus PFS extension isn’t clear yet. We’d want to see OS or second progression free survival (PFS2) data first.

Everyone was clamoring to hear the news from VERITAC-2—the first ever Phase III evaluation of an ER proteolysis targeting chimeric (PROTAC) degrader (vepdegestrant). Standard SERDs destabilize ERs, leading to their degradation. Meanwhile, PROTACs actively recruit proteosomal degradation machinery and work catalytically—two potential pharmacological advantages.

The data showed that vepdegestrant monotherapy conferred a ~3 month PFS increase when compared to fulvestrant alone in the ESR1 mutant subpopulation—a figure that Wall Street didn’t seem to like. Perhaps this was because the drug didn’t add value in the all-comer setting or because the PFS improvement was seen as too small.

There could still be merit here. Provided OS is similar to fulvestrant, an oral monotherapy alternative like vepdegestrant could have a place in the market. However, monotherapy in the 2L setting seems to be becoming increasingly rare.

With its safety profile, vepdegestrant could allow patients to avoid beginning chemotherapy for longer. We’d argue that a 3 month PFS improvement isn’t something to scoff at. Marschner et al. showed in 2020 that disease progression causes an immediate, substantial decrease in quality of life—every month matters.

Overcoming ER-Independent Resistance with Novel Agents

Therapeutic pressure from hormone therapy pushes some tumors to compensate through alternative pathways like GF-RTK, MAPK, and PI3K-AKT—the latter being the most common. During ASCO, the speakers discussed multiple, ongoing efforts to drug the PI3K-AKT pathway.

Five years ago, the SOLAR-1 study showed that adding a PI3Kα-specific inhibitor (alpelisib) to SOC fulvestrant meaningfully increased PFS in the PI3K-mutant subpopulation—with roughly equal robustness across common PI3K variants.

SOLAR-1 set the stage for several newer studies, including the INAVO-120 Phase III trial of inavolisib—an isoform selective inhibitor and degrader of PI3Kα. Combined with CDK4/6i and fulvestrant, inavolisib more than doubled PFS in the PIK3K-mutant population. Critically, INAVO-120 also showed an OS benefit—the first time this has been demonstrated with the class. We’ll also note that PI3K status was determined using ctDNA, which poses yet another risk for capturing lead time bias. We’re looking forward to seeing the forthcoming wave of allosteric, mutant-resistant PI3K inhibitors.

PI3Ki and CDK4/6i have sparsely been combined due to toxicity overlap. Instances of hyperglycemia, diarrhea, rash, and stomatitis were all higher in the inavolisib group. Hyperglycemia was particularly an issue in patients who had a high BMI or were otherwise pre-diabetic. Another study further characterized hyperglycemia in these at-risk patients, showing that treatment with inavolisib required additional anti-diabetic therapy and occasional dose interruption.

On the AKT front, the 2023 CAPITELLO-291 study demonstrated that capivasertib—a pan-isoform AKT inhibitor—provided a more pronounced benefit in patients whose tumors harbored an AKT pathway alteration. The more recent FINER study echoed these results with ipatasertib—another pan-isoform inhibitor. This trial showed both a PFS and OS benefit in the ITT and AKT-altered cohorts. ctDNA was used for randomization here as well and the OS data was only at 50% maturity, however.

Here’s what we’re still musing over:

• ctDNA is inevitable, but how can we use it most effectively? Will it become a surrogate endpoint for solid tumors? How do we structure trials in a way to avoid capturing any lead time bias?

• Will emerging isoform or mutant-selective allosteric inhibitors for PI3K or AKT add a material boost to efficacy or safety?

• Will attacking CDK2/CCNE1 upregulation allow for more complete cell cycle arrest when combined with CDK4/6 inhibition?

• If doublet or triplet regimens become the norm, how will we think about AEs in both curative and palliative-intent scenarios?

• How should we think about cross-reactivity and sensitization given the overlap in resistance pathways?

Earlier this week, Merck hinted at positive interim results from several ongoing Phase 3 clinical trials evaluating the efficacy of enlicitide, the company’s oral PCSK9 inhibitor. As some have pointed out on Twitter, enlicitide is an enormous macrocyclic compound with a 37-member ring.

I love writing about macrocycles. These beautiful monstrosities are challenging to discover, difficult to optimize, and tedious to scale. Therefore, companies that choose to develop macrocycles usually have a very compelling reason.

In another post, I wrote about how macrocycles are uniquely suited to modulate intra-cellular protein-protein interactions (PPIs)—a pharmacological niche that’s usually off limits to both traditional small molecules and larger biologics. I focused on REC-6236, a fascinating glue-like macrocycle that disrupts RAS signaling in pancreatic cancer.

Now I want to dig into enlicitide to explore all its quirks, features, and its reason for existence in the first place—starting with a bit of biology.

Subscribe now

PCSK9 and Bad Cholesterol

You’ve probably heard of bad cholesterol, otherwise known as low-density lipoprotein (LDL) cholesterol. As with most things, it’s the dose that makes the poison. LDL cholesterol isn’t inherently bad—these molecules help give cell membranes their structures.

However, LDL does have a tendency to oxidize inside arterial walls, triggering an inflammatory response that results in plaque formation (atherosclerosis). The blocking and narrowing of coronary arteries significantly increases one’s risk for major cardiovascular episodes like heart attack or stroke.

The liver is the central regulator of LDL in our bodies. The majority of LDL is produced in and excreted by the liver, where it’s free to travel throughout the circulatory system.

If LDL is gets too high, hepatocytes upregulate the expression of LDL receptors (LDLRs), which grab soluble LDL molecules and pull them back into the liver—decreasing serum LDL.

When LDL is too low, PCSK9 gets involved. This 692-residue protein is also synthesized in the liver. Once in the circulatory system, PCSK9 also binds to the surface of LDLR, triggering receptor-mediated endocytosis.

Inside the acidic pH of an endosome, the PCSK9-LDLR affinity intensifies and the complex is marked for degradation inside lysosomes. With fewer surface-exposed LDLRs, the liver cannot pull LDL inside as rapidly—increasing serum LDL.

Given PCSK9’s central role in regulating LDL, it should come as no surprise that there’s a strong genetic association between PCSK9 variants, LDL levels, and cardiovascular episodes.

Genetic Target Validation

Drug targets with statistical or functional genetics support are ~2.6X more likely to succeed in the clinic. Widespread sequencing and increasing access to structured, clinical data has resulted in thousands of causal genotype—phenotype linkages. Virtually all of these have been confined to diseases with simpler, Mendelian modes of inheritance.

PCSK9 is a unique case for many reasons. Firstly, LDL cholesterol levels and major cardiovascular events are complex (non-Mendelian) phenotypes. Secondly, PCSK9 is actually the fastest ever example of a genetic target translating into an approved medicine at just 12 years.

In 2003, scientists discovered missense mutations in PCSK9 that seemed to cause hypercholesterolemia—a genetic predisposition to having chronically high levels of LDL cholesterol. Just two years later, another group showed that loss-of-function (LoF) mutations in PCSK9 led to a 28% reduction in serum LDL as well as an 88% reduction in major coronary artery events over a 15 year period. Critically, these LoF alleles didn’t seem to cause adverse effects otherwise.

Taken together—these data suggested that even complete PCSK9 ablation would be both safe and potentially desirable as a therapeutic means to rescue patients with high LDL.

The First Wave of PCSK9 Therapies

Since the 1980’s, statins have been the standard of care for addressing high LDL. These compounds inhibit HMG-CoA reductase in the liver, an enzyme responsible for LDL synthesis. Statins don’t work for a lot of people, though—whether that be because of high baseline LDL levels, diet and exercise, or treatment discontinuation.

The FDA approved Praluent (alirocumab) in 2015, an antibody inhibitor of PCSK9. The second antibody called Repatha (evolocumab) was approved a few months later. Both of these antibodies bind circulating PCSK9, hindering its ability to downregulate LDLR expression—thus lowering serum LDL levels.

In pivotal trials (1, 2), both molecules achieved placebo-adjusted serum LDL reductions of ~60% from baseline when combined with statin therapy. This level of LDL reduction is clinically meaningful as it resulted in a ~15% reduced risk for cardiovascular death. Sales of both Repatha and Praluent totaled around $3 billion in 2024.

I’d be remiss to not also mention Leqvio (inclisiran), an siRNA inhibitor of PCSK9. Leqvio’s advantage is that it requires only bi-annual subcutaneous dosing to disrupt PCSK9 translation—achieving a >50% placebo-adjusted reduction in serum LDL levels. Antibodies typically require bi-weekly dosing by comparison.

A Need for New Therapies

Even with three current-generation options, the situation on the ground isn’t ideal.

Roughly half of patients discontinue PCSK9 inhibitors after just six months—the primary reason being high out-of-pocket (OOP) costs. This is a multifaceted issue that spans high copayments, deductible structures, prior authorizations, and more.

That most PCSK9 inhibitors are biologics is one contributing factor. Produced in living cell cultures, purified through multiple steps, maintained under strict temperature control, and administered via syringe—antibodies have a high cost-of-goods (COGS), some portion of which trickles down to patients.

Even non-generic small molecules, such as Lipitor (atorvastatin), cost about $150/month OOP—below the ~$200 threshold where there is substantial price sensitivity.

Sure—the fact that small molecules are oral definitely helps with patient choice. I don’t love needles either. But I found that only ~1.5% of patients discontinue their PCSK9 regimen because of the route of administration. Cost seems to be the main target product profile (TPP) niche to poke at.

If you want to read more about TPPs—here’s a shameless plug for our most recent article.

Merck’s Shot on Goal with Enlicitide

Merck chronicled the discovery, development, and early clinical testing of enlicitide in a 2023 journal article. Unsurprisingly, they cite PCSK9’s lack of a canonical active site as the primary motivator behind the macrocycle campaign—along with the possible oral route of administration.

The company’s scientists leveraged an mRNA-display platform to screen for potential macrocyclic binders. Experimental crystal structures from enlicitide analogues suggest that the macrocycle binds a flat surface in PCSK9’s non-active catalytic domain, preventing the protein from binding LDLR.

I found the first-in-human PK data and commentary rather interesting. Macrocycles, and other large (beyond-rule-of-five) compounds, typically have poor oral bioavailability and cell permeability. That is indeed the case here. The group had to jump through several hoops to get enlicitide to do its job in humans.

Patients had to be in a fasted state before taking the study drug. Moreover, enlicitide had to be co-administered with a gut permeability enhancer (sodium caprate) to get the molecule to squeeze through intestinal gap junctions and into circulation.

After all this, they showed a ~60% (non-placebo-adjusted) decrease in serum LDL. Here’s another curious thing—the idea that a lack of cell permeability could actually be a feature and not a bug.

The group hypothesized that since enlicitide cannot enter cells, the dose level can be titrated quite high as it theoretically shouldn’t interact with intracellular targets. This reason was cited as a potential factor behind the relatively clean safety profile in the Phase 1 study.

In Merck’s Phase 2 study, they mostly replicated the previous result—showing a ~55% placebo-adjusted serum LDL level reduction at the highest enlicitide dose (30 mg QD) with an adverse effect profile very similar to placebo. As for the full set of Phase IIIs, we’ll have to see.

Overall, enlicitide seems like a viable and potentially first-in-class oral inhibitor of PCSK9. Unlike most macrocycles, Merck isn’t targeting an intracellular PPI. Instead, the primary motivators for this cyclic compound seem to be the oral route of delivery, the potentially lower cost to patients—and likely their belief that only a macrocycle would have enough total polar surface area (TPSA) to disrupt the binding of PCSK9 and LDLR.

There are others who seem to believe slightly differently, though.

AstraZeneca’s Dark Horse—AZD0780

Right behind enlicitide is AZD0780, AstraZeneca’s oral PCSK9 inhibitor which recently completed a Phase IIb trial. Unlike enlicitide—AZD0780 is a traditional small molecule acquired from Dogma Therapeutics way back in 2020. Since the acquisition, AstraZeneca has thoroughly nuked Dogma’s website, so I couldn’t find out much about this molecule.

My understanding is that Dogma launched a thorough structural discovery campaign to elucidate a novel, cryptic pocket on PCSK9. The company then developed picomolar binders to this pocket which could remodel the catalytic surface, thereby disrupting complex formation with LDLR.

While the structures don’t match, I was able to find another contemporary small molecule program by an Australian biotech called Nyrada—who generously published a more comprehensive explainer on their approach.

Nyrada screened compounds against PCSK9, finding that Nilotinib was a weak inhibitor that bound to a cryptic pocket proximal to the canonical LDLR binding interface. The group optimized this molecule to improve its potency. It seems that this program was placed on the back burner while the company prioritizes other lead programs.

Anyway—back to AstraZeneca.

The Phase IIb study seems competitive with enlicitide. AZD0780 conferred a ~50% placebo-adjusted decrease in serum LDL in the 30 mg QD range with a slightly better AE profile than Merck’s drug—though cross-trial comparisons are full of pitfalls. The company was sure to mention in their press release that this efficacy was achieved without fasting—and we all know who that comment was aimed at (wink, wink).

Wrapping Up

Enlicitide is yet another curious macrocycle. It seems likely that it will become one of, if not the first, approved oral inhibitor of PCSK9. Future doctors will have at their disposal a set of biologic PCSK9 inhibitors, excellent oral alternatives, an RNA-based medicine, and potentially even an in vivo gene editing solution to permanently ablate PCSK9 in the liver.

On the oral front—I’m leaning slightly more towards AstraZeneca’s molecule for a few reasons that I would love to debate for those paying closer attention to this than I am:

1. Both these drugs are oral, but AstraZeneca’s drug doesn’t require fasting.

2. AstraZeneca’s drug is likely cheaper to produce, giving them better pricing and scaling leverage.

3. That enlicitide can’t get into other cells doesn’t seem to be translating into vastly better tolerability.

4. Combined with statins, the efficacy seems about the same.

Overall—I’m happy my fellow needle-haters will have more options. What will the next curious macrocycle be?

1. Foreword

2. Origin of the Target Product Profile

3. Industry Pushback and Modernization

4. Deciphering Some Examples

5. A Tale of Two TPP Construction Philosophies

6. Universal Guiding Principles

7. Pharma and Biotech Perspectives

8. Parting Thoughts

Subscribe now

Forward

Technical success—lovely, but inadequate.

Bluebird Bio achieved a scientific triumph in 2022 when the FDA approved its lead asset Zynteglo, a first-of-its-kind gene therapy offering curative potential for transfusion-dependent β-thalassemia (TDT). Since Zynteglo’s launch, however, Bluebird’s stock (BLUE) has fallen 97%, and the company was ultimately taken private for roughly $30 million—a far cry from the biotech’s peak valuation of $10 billion.

Bluebird was an excellent organization filled with brilliant scientists and skilled operators. However, its story offers an opportunity to understand what went wrong and how emerging biotechs can more effectively navigate commercial pitfalls.

Despite groundbreaking science, Zynteglo failed to gain commercial traction—not because it didn’t work, but because it entered a healthcare system unprepared to reimburse the $2.8 million list price.

The key foot fault was the target product profile (TPP). Although Bluebird’s TPP addressed a large clinical unmet need and accurately anticipated regulatory approval, it failed to align clinical ambition with market access and reimbursement hurdles.

Today’s biotech ecosystem overemphasizes technical success. Claims like, “Our platform technology increases the probability of technical success by 20%” ring hollow against a backdrop where only half of FDA-approved molecules recoup their R&D investments. Technical success is a necessary, but insufficient condition for building an enduring biotech.

Early, frequent, and rigorous TPP construction is crucial for calibrating R&D efforts to commercial realities—such as pricing tolerance and payer readiness—ensuring that drugs achieve both technical and market success.

Origin of the Target Product Profile

Nearly twenty years ago, the FDA consolidated a patchwork of internal, pharma working documents into the original incarnation of the TPP.

While TPPs vary between organizations, the modern framework captures essential information about a product’s intended population, differentiation from the established standard of care (SoC), administration and dosing considerations, and market access insights. TPPs are ubiquitous today—but this wasn’t always the case.

Decades ago, pharma companies used a fragmented mix of internal target labels and regulatory briefing documents to guide drug development. These tools lacked standardization, version control, digitization, and mechanisms for cross-functional collaboration between R&D and commercial teams. Early guiding frameworks often omitted market access strategy altogether.

The FDA formalized the concept in its 2007 draft guidance, positioning the TPP as a tool to help sponsors “begin with the end in mind”. Ideally, the TPP bridges development and regulatory planning—clarifying intent-to-treat sub-populations, efficacy targets, safety margins, and more, each supported by a clear trail of preclinical data.

Industry Pushback and Modernization

The modern TPP is a data-driven dashboard meant to serve as a substrate for debate and decision making.

Initially, there was industry pushback. Pharma companies were concerned that TPP claims might be seen as binding or interpreted as overly promotional. Many drug developers rightly pointed out that the FDA’s draft guidance was intended to galvanize regulatory efficiency—not to ensure commercial success. This perceived gap drove the modernization of the TPP into something more than a regulatory artifact. TPPs are now business plans in disguise.

Today’s TPPs are highly dynamic, data-fed tools that help nimble teams develop drug products addressing clinical and commercial gaps. Against a highly competitive backdrop, TPPs are moving targets, informed by constant shifts in the global development landscape and scientific breakthroughs alike.

Modern TPPs often include both minimal and preferred outcomes, a precedent popularized by the World Health Organization (WHO) and now widely adopted across industry. Compared to earlier versions, TPPs are now intended to serve diverse audiences, including patients, clinicians, regulators, payers, pharmas, biotechs, investors, health technology assessment (HTA) bodies, and others.

Below is an example of what a modern-day TPP may look like along with suggestions for how to fill one out:

Deciphering Some Examples

For either molecule to succeed commercially, the convenience and tolerability advantages must produce clinical outcomes that matter to patients, providers, and payers.

In highly saturated therapeutic areas, successful differentiation hinges on a nuanced understanding of what truly matters to patients, clinicians, and payers. Below are two illustrative examples of how different product profiles aim to carve out space in competitive environments—one through delivery convenience and the other through safety optimization.

In the rapidly evolving multiple myeloma landscape, BCMA is a well-validated target. Several high-efficacy therapies are already approved or in late-stage development, including CAR-Ts, bispecifics, and antibody-drug conjugates. However, many of these treatments are logistically complex—requiring intravenous (IV) administration, hospitalization, or specialized centers.

This hypothetical, oral BCMA inhibitor aims to differentiate by offering an at-home alternative, reducing treatment burden and improving access for a broader patient population. The convenience of a pill—especially for relapsed/refractory patients with limited mobility or other logistical constraints—could drive wider adoption. Still, in a space dominated by potent biologics, clinical performance must clear a high bar to gain meaningful market traction.

Lerociclib, an oral, highly selective CDK4/6 inhibitor, takes a different angle—refined tolerability. The program emphasizes a continuous daily dosing strategy in combination with fulvestrant, aiming to deliver efficacy on par with other CDK4/6–endocrine therapy combinations.

The molecule’s core differentiator lies in reducing the toxicity burden commonly associated with the drug class. Early data suggest lower rates of Grade 4 neutropenia, gastrointestinal adverse events, fatigue, stomatitis, and alopecia—side effects that often constrain adherence or impair quality of life. If these results hold in larger trials, lerociclib may appeal to patients and clinicians prioritizing long-term safety and tolerability.

CDK4/6 inhibitors like palbociclib, ribociclib, and abemaciclib are deeply entrenched, with substantial clinical and real-world data. For lerociclib to succeed, its safety advantages must translate into outcomes that influence not just clinical theory, but real-world prescribing behavior and payer decisions.

A Tale of Two TPP Construction Philosophies

Each philosophy has merit and has produced commercially impactful drugs.

There are two basic strategies for building an initial TPP—top-down and bottom-up. Both seek to bridge molecules and markets, but they begin from different starting points.

Consider the success of CAR-T cell therapy for hematological malignancies which followed this approach. Historically, patients with relapsed or refractory B-cell cancers (e.g., acute lymphoblastic leukemia (ALL)) had few options following chemotherapy and stem cell transplant failure. Oncologists had long recognized this treatment gap, but lacked a viable solution until scientists discovered how to reprogram T-cells to recognize and destroy cancer cells.

Instead of starting with “we need a CAR-T therapy,” companies like Novartis (Kymriah) and Gilead (Yescarta) worked closely with oncologists and hematologists to understand which cancers were most likely to benefit from CAR-T therapy. They honed in on CD19 as an ideal target antigen given it’s widely expressed on B-cells and had limited off-target toxicity concerns. This top-down approach, which started with a clearly defined unmet need and mapped backward to a druggable target, catalyzed a revolution in personalized cancer treatment.

Armed with a refined target list, teams scour the disease landscape for pathophysiologies involving those targets. Finally, multi-stakeholder input helps match indications to underserved patient populations with both clinical urgency and commercial potential.

The development of a gene therapy for spinal muscular atrophy (SMA) embodies this approach. SMA is a devastating genetic disease caused by mutations in the SMN1 gene, leading to progressive muscle weakness and early mortality in infants. For years, SMA management was limited to supportive care since traditional drug modalities couldn’t address the underlying genetic error.

Researchers saw gene therapy as a means to restore SMN1 function by delivering a healthy copy of the gene. Instead of pursuing a treatment for broad neuromuscular disorders, AveXis (developer of Zolgensma) focused on a precise genetic target and built a TPP around the most clinically and commercially viable scenario—early intervention in infants before irreversible damage occurs. This bottom-up strategy, starting with an accessible genetic target and then identifying the right disease context, delivered a breakthrough therapy that transformed SMA treatment.

Universal Guiding Principles

TPPs aren’t static blueprints—they’re evolving hypotheses shaped by data and decision making.

Most organizations employ different design methods concurrently, facilitating an 'iron-sharpens-iron' exchange of perspectives. Contrary to a common belief, companies rarely finalize a TPP before initiating discovery and development. Instead, they iteratively converge on viable market—molecule pairings.

A biotech may generate a hit in early screening and then build a rudimentary TPP or target scorecard to see if it’s worth prioritizing. A pharma company may run a salvo of light-touch TPPs across an underserved therapeutic area to determine which targets warrant full-scale R&D investments.

This iterative strategy is exemplified by the CRISPR-Cas9 approach to sickle cell disease (SCD). Early CRISPR research broadly targeted genetic diseases without a clear beachhead indication. Companies like Vertex and CRISPR Therapeutics refined their focus through repeated cycles of assessing scientific feasibility and market need—ultimately landing on ex vivo editing of hematopoietic stem cells for SCD and β-thalassemia.

Create Dynamic, Data-Driven TPPs

TPPs should be living documents, housed in tech-enabled infrastructure capable of adapting to market ebbs and flows. Teams should consider not only the current SoC, but where the cutting edge of medicine may be in ten years. Feed in commercial intelligence, emerging scientific trends, and patent expiry timelines to make a TPP anti-fragile.

Involve Multiple Stakeholders Early and Often

Robust TPPs incorporate input spanning regulatory, commercial, R&D, market access, and translational domains. It’s also never too early for teams to engage payers, physicians, and patients. Discussions with these groups ensure that TPPs are grounded in what actually drives demand and uptake.

Consider the Therapeutic Backdrop

TPPs should be attuned to the nuances of their therapeutic category. For example, immunology and inflammation (I&I) indications have high efficacy bars. Given that these are often chronic conditions, long-term tolerability and friendly routes of administration (ROA) offer chances for differentiation.

Be Ambitious and Quantitative

Target and indication crowding mean that small improvements get lost in the noise. TPPs should be built around bold—but quantifiable—aspirations. Directional ambition isn’t enough. Teams should quantify the performance lift needed to leapfrog competitors.

Foster a Translational Mindset

Trials should be engineered to cleanly demonstrate key TPP differentiators. Teams should focus on endpoints incorporating accepted biomarkers and statistically powered to show value. Meanwhile, layering in clinician insight helps streamline enrollment. If patients can’t be found in the context of a trial, they’ll never be found in the real world.

Account for Psychology

Pharmas and biotechs alike are intelligent superorganisms, but they’re not infallible. Even stellar organizations fall prey to internal bias. It’s easy to fall in love with a promising lead series or a tantalizing TPP wedge—but this shouldn’t come at the cost of holding onto a doomed development campaign. Teams must define performance thresholds early and commit to dispassionately pruning programs when the cards flip over.

Have Contingency Plans

TPP design should be flexible. Markets are dynamic. Competitors pivot. Teams should build for label expansion and strategic rebuttal. Having a fallback plan—or two—ensures that value gets unlocked even if the initial TPP scenario disappears.

Pharma and Biotech Perspectives

In the fog of war, it’s easy for strategy and technical teams to fall out of sync. TPPs are the connective tissue between teams, preventing faultlines from forming.

Despite commanding fleets of internal R&D teams, large pharmas don’t address every TPP gap organically. They wield TPPs like shopping lists. As business development teams walk through aisles of biotech assets, TPPs provide an easy rubric for discerning whether programs may align with future pipeline needs—whether for partnership, in-licensing, or outright M&A.

Established pharmas possess an information advantage on paper. Shouldn’t an ocean of historical TPPs and market data combined with massive infrastructure make it easy to come up with winning product ideas? The reality is murkier. Large pharmas are oftentimes structurally fragmented—amalgamated over decades of M&A and operating with pseudo-independent siloes. Big pharmas also aren’t tech companies. Interdivisional coordination, systems integration, and internal sign-offs pose formidable obstacles for truly benefiting from information asymmetry.

Biotechs have it tough. Young companies are capital-constrained, speed-obsessed, and laser-focused on mid-term value inflection. Biotechs may slack on robust TPP development, causing management and technical leadership to fall out of lockstep.

Fortunately, including voices from regulatory, commercial, and market access domains is becoming easier. Input may come from experienced advisors, speciality consultants, and a new wave of pipeline and business intelligence (BI) tools. Even open-source Deep Research models can serve as a starting point.

TPPs can also serve as deal accelerants. Transactions with large pharma remain the primary source of non-dilutive capital and exit liquidity in biotech. Having credible TPP documentation on hand streamlines partnership diligence and invites constructive feedback.

Parting Thoughts

TPPs are a pharmaceutical prime directive.

Crafting and maintaining a rigorous target product profile has never been more important. The pharmaceutical landscape is increasingly noisy and crowded—with targets and patient segments saturated by incremental “me-too” therapeutics. Adding to the pressure is large pharma’s growing ability to source low-cost molecules from China, posing a serious threat to the primary source of exit liquidity for Western biotechs.

Despite the headwinds, the technological backdrop for emerging biotechs has never been stronger. A growing suite of digital tools is reshaping strategy development, ranging from commercial intelligence platforms to real-time market access data brokers to cross-functional pipeline optimization engines.

Whether teams choose to engage specialty consultants or build internal capabilities, the modern biotech strata have ample resources to align scientific and commercial teams with managerial vision. For burgeoning companies aiming to attract strategic partnerships or investor attention, the TPP muscle is no longer optional—it’s imperative for survival.

Cell depletion therapies have completely revolutionized the way we treat B cell malignancies, offering the potential for complete remission or even cures. Excitingly, the clinical success of B cell depletion therapies (BCDTs) for oncology has inspired a number of new BCDT programs for autoimmune diseases. Given this unique inflection point for BCDTs, we’ve decided to compile an overview of the field and its history, highlighting some promising therapeutic modalities and opining on some ideas of where we feel the field might be headed next.

1. B Cell Depletion Therapy for Cancer Treatment

2. Autoimmune Diseases

3. Current Autoimmune Treatments and Emerging Cell Depletion Approaches

4. Translation from Oncology to Autoimmune

5. Spotlight on T Cell Engagers for Autoimmune

6. Targeting the Right B Cell Subpopulation

7. Current BCDT Outlook

Subscribe now

B Cell Depletion Therapy for Cancer Treatment

B cells are an immune cell type most commonly associated with the secretion of antibodies that physically mark cells, pathogens, and proteins for elimination. While B cells play an important role in fighting infections and preventing disease, they are also a common origin for cancers such as Non-Hodgkin Lymphoma (NHL), which accounts for ~4% of all cancer diagnoses in the United States. About 85% of all NHLs arise from B cells, referred to as B-NHLs.

The major goal in the treatment of B cell lymphomas is the selective killing of cancerous B cells and the establishment of long-term remission. Prior to the late 1990s, the standard treatment options for B cell lymphomas included radiation therapy and chemotherapy, which are effective at killing cancerous B cells, but are not very selective. Thus, these approaches were significantly limited by their off-target effects, damaging both cancer cells and healthy cells across different tissue types.

The demand for more precise treatments has driven the development of B cell depletion therapies (BCDTs)—a rapidly expanding class of treatments designed to selectively eliminate pathogenic B cells. BCDTs encompass a range of therapeutic modalities, with the most common being monoclonal antibodies (mAbs) designed to target unique cell surface markers expressed only by B cell populations (e.g., CD19, CD20, BCMA). Just as healthy B cells produce antibodies to eliminate pathogens, we’ve learned how to design and manufacture antibodies as medicines that specifically target B cells for depletion.

The most well-known and widely prescribed BCDT is a drug called rituximab (Rituxan, Genentech/Biogen), a mAb that recognizes the B cell surface protein CD20. While originally approved by the FDA as a monotherapy for the treatment of chemo-resistant B-NHL in 1997, rituximab combined with chemotherapy (i.e., R-CHOP) has quickly become the standard of care across several B cell lymphomas with an impressive complete response rate of ~76%.

Antibody-based therapeutics, such as rituximab, have played a pivotal role in the clinical advancement of BCDTs due to their ability to bind to cell surface receptors with high specificity and trigger immune-mediated target cell depletion. There are a few main mechanisms by which these cytotoxic processes occur:

1. Complement-Dependent Cytotoxicity (CDC): The complement system is a collection of extracellular proteins that initially recognize and bind to an antibody’s Fc region (i.e., the highly conserved tail of the “Y” shaped IgG antibody, as shown below). Following antibody recognition, a cascade of complement proteins are recruited and assembled into a membrane attack complex (MAC) that forms pores within the target cell membrane resulting in osmotic imbalance and cell lysis.

   

2. Antibody-Dependent Cellular Cytotoxicity (ADCC): Natural killer (NK) cells are a component of the innate immune system that recognize the Fc region of antibodies via their surface expression of Fc-gamma receptors (FcγRs). When NK cells recognize an antibody-coated B cell, they become activated and release lytic enzymes (i.e., perforin and granzyme) that selectively destroy the target cell.

3. Antibody-Dependent Cellular Phagocytosis (ADCP): Similar to ADCC, macrophages recognize the Fc region of antibodies through FcγRs. However, instead of lysing the cells, macrophages engulf and digest the target cell in a process called phagocytosis.

Building on the strong clinical success of rituximab, the development of novel modalities for B cell depletion—especially those that engage other immune mechanisms for eliminating cells, such as bispecific T cell engagers (TCEs) and chimeric antigen receptor (CAR) T cells—has exploded, significantly expanding the treatment landscape for B cell malignancies.

TCEs are often formatted as bispecific antibodies, which are engineered to bind different targets with each arm of the antibody. For example, one arm typically recognizes the CD3 surface receptor specific to T cells, while the other arm recognizes a unique surface marker on the target cell of interest (in the case of BCDTs, CD19 or CD20 on B cells). Upon formation of this molecular bridge, the T cell is activated and subsequently lyses the target cell through release of perforin and granzyme molecules. TCEs bypass traditional antigen presentation to T cells, offering a distinct elimination mechanism compared to mAbs. However, without fine tuned control, TCEs can potentially lead to uncontrolled over-activation of T cells and severe immune-related side effects

Another T cell-centric approach, therapeutic CAR T cells have also shown tremendous curative efficacy for targeting B cell populations. By engineering T cells with a synthetic receptor that binds specific cell surface markers, CAR T cells can selectively eliminate target cells of interest. Similar to TCEs, CAR T cells do not rely on traditional mechanisms of T cell activation and can be tuned to target a wide range of cell types for many disease indications through the modulation of the binding region of the CAR. However, despite their initial clinical success, CAR T cell therapies are often limited by severe safety issues (e.g., cytokine release syndrome), require broad lymphodepletion chemotherapy prior to cell therapy to limit rejection of the CAR T cells, and are gated by poor infrastructure for scalable manufacturing, since these therapies currently require ex vivo culture and engineering.

There is no question that BCDTs have revolutionized the treatment of B cell malignancies. Given their historical success in oncology, there has been considerable growing interest in applying the same approaches to inflammatory and immunology (I&I) indications—particularly autoimmune diseases. But what exactly are autoimmune diseases, what role do B cells play in their pathologies, and why are they promising indications for BCDT?

Autoimmune Diseases

Your immune system comprises several unique cell types responsible for recognizing and eliminating pathogens (e.g., bacteria, viruses, fungi) and diseased cells (e.g., cancer). These immune cells are taught to distinguish between healthy proteins and tissues from your own body (i.e., “self”) versus foreign organisms and diseased tissue (i.e., “non-self”) through highly orchestrated processes called central and peripheral tolerance.

While the processes differ between immune cell types, central and peripheral tolerance can be broken down into two major components:

1. The elimination of immune cells that target self-antigens on healthy tissue

2. The maturation of immune cells that correctly recognize foreign antigens

Take B cells, for example—each B cell originates from a unique parent “clone” that expresses a distinct B cell receptor (BCR)—a membrane-bound form of the antibody it secretes. The BCR determines which antigen the B cell recognizes. Upon recognition of its unique antigen, B cells become activated and differentiate into mature states where their primary function is to secrete soluble antibodies to tag and eliminate the antigen.

Before arriving at this mature state, B cells must learn self from non-self. As immature B cells develop in the bone marrow, each clone undergoes a highly intricate process called VDJ recombination, in which variable (V), diversity (D), and joining (J) gene segments rearrange randomly to generate a unique amino acid sequence corresponding to a novel BCR/antibody. These newly formed B cells are then exposed to self-antigens—either in soluble form or presented by neighboring stromal cells. If a B cell’s BCR strongly interacts with a self-antigen, it faces one of three fates:

1. Apoptosis (i.e., cell death)

2. Receptor editing via additional rounds of VDJ recombination to form a new BCR

3. Functional inactivation through a process called anergy, in which the B cell downregulates its BCR and becomes unresponsive

B cell clones that do not bind to self antigens proceed in development and migrate as transitional B cells to peripheral immune organs such as the spleen and lymph nodes. Here, they undergo peripheral tolerance, a second round of negative selection to further ensure they do not react against self-antigens, before becoming fully mature.

Keeping this process in check is central to maintaining healthy cell and tissue function across all organ systems. To put this into perspective, the human body contains ~2 trillion lymphocytes—including T cells, B cells, and NK cells. That’s ~2 trillion cells that must be correctly taught to recognize friend or foe. If even a single lymphocyte escapes this education process without proper regulation, it may mistakenly recognize the body’s own proteins as foreign. This breakdown of central tolerance can occur due to a variety of factors including genetic predisposition and environmental factors that impact the negative selection process as well as loss of regulatory immune cell populations that limit proliferation of auto-reactive clones. This misdirected immune response triggers an attack on healthy tissues, leading to progressive damage and loss of function—the hallmark of autoimmune disease.

Autoimmune diseases encompass a broad spectrum of disorders, which are usually categorized by tissue type and disease mechanisms. Some of the most common examples include psoriasis, multiple sclerosis (MS), inflammatory bowel diseases (IBD), rheumatoid arthritis (RA), type 1 diabetes (T1D), and systemic lupus erythematous (SLE).

The rise of autoimmune diseases, their burden on the healthcare system, and the need for innovative therapies is becoming increasingly apparent. A recent study in The Lancet showed that autoimmune diseases affect an estimated 1 out of 10 individuals during their lifetime. In the United States, it is estimated that over $100 billion dollars are spent every year on healthcare costs for patients with autoimmune diseases.

Current Autoimmune Treatments and Emerging Cell Depletion Approaches

Historically, autoimmune diseases have been predominantly treated with anti-inflammatory medications that work to broadly suppress the dysregulated immune system (e.g., corticosteroids and non-steroidal drugs (NSAIDs) like ibuprofen).

Recently, more targeted therapies that block specific pro-inflammatory factors have been approved for clinical use, including small molecule inhibitors of JAKs (e.g., tofacitinib, upadacitinib) and mAbs against pro-inflammatory cytokines such as tumor necrosis factor-alpha (TNF-α), IL6, and IL17 (e.g., adalimumab, tocilizumab, secukinumab).

Despite often demonstrating greater efficacy, these newer therapies have yet to universally replace traditional broad immunosuppressants as first-line (1L) treatments for autoimmune disease. This could be due to a mixture of increased side effects, long-term safety concerns, higher costs, and the entrenchment of well-established standard of care treatments. For example, in rheumatoid arthritis (RA), NSAIDs, corticosteroids, and broad disease-modifying anti-rheumatic drugs (DMARDs) such as methotrexate are typically the 1L treatments for severe disease. Meanwhile, TNF-α inhibitors and other biologics are usually reserved for patients who do not respond to initial therapies.

Though precision drugs have led to improvements in quality of life and a more controlled rate of disease progression in patients, they primarily function by dampening inflammation rather than addressing its underlying causes. Toward this goal—cell depletion therapies are beginning to gain traction.

Borrowing from BCDTs in oncology, the idea here is that if you can identify the cell type(s) driving autoimmune pathology, you can leverage targeted therapeutic strategies to selectively eliminate (i.e., deplete) them, thus permanently halting disease progression.

However, the root causes of these diseases and the mechanistic roles different immune cell types play are often not well understood, making it difficult to design effective therapies. Historically, T cells have been considered the predominant driver of autoimmune diseases due to their innate cytotoxic activity. For example, in Type I Diabetes, auto-reactive T cells target and destroy healthy beta cells in the pancreas, leading to decreased insulin production and disease onset.

As a result, broad T cell depletion therapies have been explored for the treatment of autoimmune disease but have faced considerable challenges. One major concern is the substantial safety risk associated with the broad depletion of T cells, which are essential for immune surveillance and defense against infections and malignancies.

Additionally, T cell maturation and central tolerance occur in the thymus, an organ that begins to shrink rapidly after the first year of life. Consequently, older patients face greater difficulty repopulating their immune system following T cell depletion, leading to diminished tolerance and a less diverse and functional T cell repertoire, which may impair their ability to respond to infections and even contribute to the onset of additional autoimmune diseases.

In light of these challenges with broad T cell depletion, researchers have looked to other immune cell types, such as B cells, whose contribution to autoimmune disease phenotypes is growing in appreciation.

In several autoimmune diseases, B cells become dysregulated and contribute to the breakdown of immune tolerance (i.e., your immune system’s ability to recognize a self antigen and not mount a response). For example, an auto-reactive B cell clone that snuck through negative selection processes will secrete “autoantibodies” that recognize self antigens on healthy tissue, marking them for elimination.

Additionally, in autoimmune diseases, B cells may upregulate expression of pro-inflammatory cytokines that further drive the misguided attack on healthy tissues. This upregulation can be driven by several factors including metabolic dysregulation and loss of B regulatory cells—a subpopulation responsible for suppressing inflammation though secretion of anti-inflammatory cytokines. Thus, leveraging BCDT to selectively eliminate these diseased B cells could alleviate the root causes of many autoimmune diseases. But what happens once some or all of your B cells are eliminated?

In contrast to other immune cell types (e.g., T cells), B cells are ideal targets for cell depletion therapy due to their ability to repopulate over the course of a person’s lifetime. Specifically, B cells are continuously differentiated from hematopoietic stem cells in the bone marrow, ensuring a constant supply of naïve clones capable of responding to novel antigens. This allows the B cell population to effectively “reset” following treatment, resulting in a more self-tolerant state and permanently removing auto-reactive B cell clones. However, B cell repopulation is poorly understood and difficult to regulate, which poses a risk to the successful reestablishment of a healthy immune system for patients following treatment.

Translation from Oncology to Autoimmune

Given the shared goal of depleting pathogenic B cell populations, the I&I field has increasingly taken a “copy and paste” approach—directly repurposing successful oncology assets for autoimmune treatment. While most BCDTs for autoimmune disease treatment remain in preclinical development or early clinical trials, a handful of late-stage drugs are showing promise.

Given that small molecules and mAbs were some of the first modalities to gain clinical approval for BCDT in oncology, it’s no surprise they are leading this first wave of autoimmune BCDTs with a few assets approaching clinical approval. In particular, Roche/Genentech has made significant progress in translating their CD20 mAb (obinutuzumab) for the treatment of lupus nephritis, recently sharing positive Phase III clinical trial results. Additionally, Roche also showed that small molecule inhibition of Bruton tyrosine kinase (BTK) with fenebrutinib led to near-complete disease remission in MS patients in a Phase II trial. BTK is a target approved for clinical treatment of B cell lymphomas and is necessary for B cell survival and maturation.

There is growing momentum as these small molecules and mAbs are followed closely by bispecific antibody TCEs and CAR T cell therapies. For example, Roche has several early ongoing trials for TCEs including a Phase I trial testing mosunetuzumab (an approved oncology drug targeting CD20 x CD3) for the treatment of SLE. Additionally, Curon Biopharmaceuticals and their CD3 x CD19 bispecific antibody CN201 was recently acquired by Merck in Oct 2024 for $750M. CN201 is currently being tested in Phase I and Ib/II clinical trials for B cell malignancies with an opportunity for expansion into autoimmune diseases.

On the cell therapy front, CD19 has been the predominant target for CAR T cell treatments for B cell malignancies. In particular, Kyverna Therapeutics is at the forefront of translating CD19 CAR-T therapy for the clinical treatment of autoimmune diseases, currently conducting Phase I/II and Phase II trials across several indications.

As these trials progress, we anticipate some will show meaningful clinical success for autoimmune patients. However, directly translating oncology assets to autoimmune patients comes with some considerable roadblocks. The biggest hurdle lies in the substantial difference in risk tolerance and safety thresholds across these patient populations, mainly driven by key biological and clinical differences. While immediately life-threatening diseases like late-stage cancer may justify the use of therapies with lower or unproven safety profiles, chronic autoimmune conditions will require safer therapeutic approaches.

Take diffuse large B cell lymphoma, for example. If left untreated, patients have a median survival of < 1 year, making the need for rapid therapeutic intervention high and the bar for therapeutic safety low. For patient populations like this, the primary goal for BCDTs has been maximizing B cell depletion by targeting cell surface markers expressed across a broad population of B cell subtypes (e.g., CD19, CD20). But this broad approach to depletion comes at the price of safety, specifically increased risk of infections and the onset of cancer. One could argue that because B cells naturally repopulate, this safety issue should only be a temporary. However, B cell repopulation is generally an understudied, uncontrolled process - often taking months to years - only to result in B cells skewed towards an immature phenotype.

In our view, safely restoring the B cell compartment to a healthy baseline remains one of the largest bottlenecks for the safe clinical translation of broad cell depletion therapies for I&I diseases.

If BCDTs are going to live up to their curative potential and become widely adopted for autoimmune diseases, their target product profiles will need to evolve to a point where safety is prioritized alongside efficacy. With this in mind, two major questions drive this cost/benefit between efficacy and safety of autoimmune BCDTs:

1. What therapeutic modalities are uniquely suited for these patients?

2. If broad depletion of B cell subtypes poses too great of a safety risk, what B cell subpopulations are the most relevant to target and can we actually target them via cell-specific markers?

Spotlight on T Cell Engagers for Autoimmune

TCEs offer several key advantages over other therapeutic modalities for BCDTs. For example, given they are often formatted as IgG based antibodies—TCEs are already clinically de-risked. They benefit from high selectivity and specificity toward their target antigen, relatively high stability, predictable pharmacokinetics, favorable safety profiles, general ease of manufacturing, and result in a final “off-the-shelf” product. These attributes set a strong foundation for TCEs, where they can thread the needle between achieving solid efficacy while limiting severe safety issues.

While potentially exciting, TCEs for treating autoimmune diseases via B cell depletion still have therapeutic hurdles to overcome in the race to clinical approval.

Current clinical data show that mAb and TCE therapies can achieve deep depletion of B cells circulating in the blood. This makes sense given these drugs are most often delivered intravenously, and in itself, this depletion can have a strong impact on disease state. However, the field is also beginning to appreciate the important roles of tissue-resident subpopulations of B cells in driving disease phenotypes. For example, auto-reactive, tissue-resident memory B cells and long-lived plasma cells, both of which produce autoantibodies, are commonly associated with disease persistence and relapse. Unfortunately, IgG-based TCEs are relatively large (~150kDa) molecules that have limited tissue penetrance, making it difficult to access and deplete these tissue-resident cell types. Alternative TCE formats based on the linkage of smaller scFv’s—often referred to as bispecific T cell engagers (BiTES)—may improve deep tissue penetrance, but this approach has so far come at the cost of plasma stability.

Another potential limitation with TCEs is that they require the presence of endogenous effector T cells to enable target cell depletion. While we’ve not seen particularly strong evidence that this will be a significant issue for autoimmune diseases, it may be a consideration for patients who have previously undergone broad lymphodepletion treatment.

There’s a loose parallel here to molecular glues. Molecular glues are small molecules that stabilize interactions between an E3 ligase and a protein targeted for degradation. Upon binding, the E3-E2 ligase complex appends a ubiquitin tag onto the target protein, marking it for recognition and degradation by the proteasome. Just as BCDTs rely on the presence of endogenous effector T cells, the effectiveness of molecular glues is also constrained by the availability of specific E3 ligases and their associated degradation machinery, which can severely limit the versatility of these therapies across cell and tissue types, as shown below.

Initial clinical results for TCEs for the treatment of autoimmune disease are beginning to roll in. A 2024 Phase I study of blinatumomab (a CD19 x CD3 BiTE) for the treatment of rheumatoid arthritis showed promising safety results, a decrease in autoantibody serum levels, and a decrease in clinical disease activity across a cohort of six patients.

Commercialized by Amgen, blinatumomab was the first FDA-approved bispecific and TCE. It originally received accelerated approval in 2014, followed by full approval in 2018 for the treatment of leukemia. While the above study is limited by a small patient cohort, blinatumomab was shown to deplete B cells and improve disease symptoms in patients who previously failed rituximab treatment, suggesting TCEs may be able to overcome efficacy issues faced by some traditional mAb-based cell depletion therapies.

Targeting the Right B Cell Subpopulation

Layered on top of the therapeutic modality question is the B cell subpopulation question. Should drug developers aim for a more complete reset of the entire B cell compartment through the depletion of all B cell subtypes, similar to the traditional oncology approach? Or, would a more selective strategy be better, where specific pathogenic B cell subtypes or even specific auto-reactive clones are targeted, while sparing the rest of the population? As the saying goes—it depends.

The decision to mount a broad versus targeted approach to BCDT ultimately depends on our understanding of the underlying causes of each autoimmune disease. For the sake of this discussion, we can compartmentalize autoimmune diseases into two main categories: (a) complex diseases where the mechanistic roles of B cells and other immune cell types in driving disease pathology is still poorly understood, and (b) B cell-mediated diseases that are largely driven by the aberrant production of autoantibodies targeting healthy tissue antigens.

If we don’t fully understand the underlying disease mechanisms of complex autoimmune diseases such as RA, MS, and SLE, designing effective targeted therapies is nearly impossible. For now, broad B cell depletion strategies may actually be the most effective treatment options available. Of course, this approach does come with safety concerns associated with poor repopulation dynamics and risk of permanent immunosuppression; however, we are learning how to manage these side effects from initial attempts in oncology. Specifically, both re-vaccination and intravenous immunoglobulin (IVIG) injection to restore protective immunity, have helped to alleviate patient side effects.

Beyond their therapeutic benefit, broad depletion approaches like these also provide an opportunity to gain valuable insights into autoimmune disease mechanisms. By carefully analyzing patient responses to BCDT and the dynamics of B cell repopulation, we can deepen our understanding of complex immune processes, which in turn can help us develop more targeted therapies in the future.

For autoimmune diseases that are primarily driven by autoantibody-producing plasma cells, a more targeted approach may be optimal. For example, in pemphigus vulgaris, autoantibodies specifically target cell-cell adhesion proteins called desmosomes in the skin (DSG1 and DSG3), while in myasthenia gravis, autoantibodies specifically target muscle-specific tyrosine kinase (MuSK) present in cells at neuromuscular junctions.

Given these well-defined and disease-relevant targets, researchers have developed modified versions of BiTEs and CAR T cells that leverage auto-antigen recognition to selectively eliminate only the plasma cell clones responsible for autoantibody production. These therapies—termed bispecific autoantigen-T cell engagers (BiAATEs) and chimeric autoantibody receptor (CAAR) T cells—flip the script on conventional molecular design of BiTEs and CAR T cells.

Take BiAATEs, for example. Instead of using an antibody or scFv to target broad B cell markers such as CD19 or CD20, BiAATEs present a disease-specific auto-antigen to selectively target auto-reactive plasma cells. This auto-antigen is recognized only by the particular plasma cell clone with the corresponding auto-reactive BCR / autoantibody.

By linking the auto-antigen to an anti-CD3 domain, BiAATEs recruit cytotoxic T cells to eliminate the auto-reactive plasma cell in the same manner as a traditional TCE. The same principle applies to CAAR T cells, where the conventional scFv targeting a cell surface antigen is replaced with an auto-antigen to enable precise, clone-specific killing.

Dialing in the optimal specificity for each autoimmune disease will be critical to achieving curative therapies with tolerable side effects. For example, these more targeted therapies will also spare regulatory B cells, a subset of B cells that are inherently immunosuppressive and contribute to alleviating autoimmune symptoms. Regulatory B cell depletion has been linked to worsened disease severity in some autoimmune conditions, making this a critical factor in designing safer and more effective BCDTs.

As our understanding of disease mechanisms inevitably grows, we believe BCDTs will also evolve towards targeting more distinct subpopulations, maximizing efficacy while minimizing off-target effects.

Current BCDT Outlook

The treatment landscape for autoimmune and other I&I diseases is poised for a major transformation. In our view, BCDTs are a critical part of the future of I&I disease treatment with several advantages over other therapeutic strategies:

1. Curative Potential: Unlike other approved therapeutic options that rely on broad immunosuppression, BCDTs target underlying autoimmune mechanisms, offering the potential for more durable remission and even curative outcomes.

2. Therapeutic Flexibility: The wide range of therapeutic modalities being explored for BCDTs increases the extracellular and intracellular target space for the precise targeting of specific B cell subpopulations, enabling more effective treatment options for patients. For example, small molecule BTK inhibitors enable targeting of a unique intracellular kinase important for B cell survival, while antibody-based therapeutics (e.g., TCEs) enable selective recruitment of cytotoxic T cells to the target cell of interest based on expression of extracellular surface receptors.

3. Broad Applicability: Given the central, mechanistic roles of B cells in driving inflammatory disease pathology, BCDTs have the potential to treat a wide range of autoimmune and I&I patients.

4. Reduced Off-Target Risks of Broad Immunosuppression: BCDTs are designed to target specific B cell populations while preserving other immune cell types. While there are still considerable safety concerns that need to be addressed with BCDTs, keeping the rest of the immune system intact can help to reduce risk of severe infections and disease onset compared to broad immunosuppression.

Over the next decade, we expect to see frontline treatments shift from broad systemic immunosuppression to precise immune modulation and long-term remission. We are quite excited for the future of cell depletion therapies and the benefits they will bring to patients. If you have any thoughts or opinions on the field, feel free to send me a message!

Conferences like NeurIPS are quite useful for staying on top of fast-changing fields such as computational biology. Unfortunately, not everyone who might like to can attend. Similar to our recent notes from MoML, we wanted to share our scribblings from the weekend workshops at NeurIPS 2024. Below, you’ll find notes from selected talks during four workshops. We hope you enjoy this resource!

Please feel free to write us with any comments, questions, or corrections!

Subscribe now

Contents

Advancements in Medical Foundation Models

Machine Learning in Structural Biology

GenAI for Health: Potential, Trust, and Policy Compliance

AI for Novel Drug Modalities

Advancements in Medical Foundation Models

Workshop Summary

“There have been notable advancements in large foundation models (FMs), which exhibit generalizable language understanding, visual recognition, and audio comprehension capabilities. These advancements highlight the potential of personalized AI assistants in efficiently assisting with daily tasks, ultimately enhancing human life. Despite the great success in general domains, FMs struggle in specific domains requiring strict professional qualifications, such as healthcare, which has high sensitivity and security risk. In light of the growing healthcare demands, this workshop aims to explore the potential of Medical Foundation Models (MFMs) in smart medical assistance, thereby improving patient outcomes and streamlining clinical workflows. Considering the primary clinical needs, we emphasize the explainability, robustness, and security of the large-scale multimodal medical assistant, pushing forward its reliability and trustworthiness. By bringing together expertise in diverse fields, we hope to bridge the gap between industry and academia regarding precision medicine, highlighting clinical requirements, inherent concerns, and AI solutions. Through this cooperative endeavor, we aim to unlock the potential of MFMs, striving for groundbreaking advancements in healthcare.” —Workshop Website

Speaker Summaries

Current Advancements and Roadblocks for Foundational Models in Pathology and Ophthalmology; Pearse Keane (University College London, Moorfields Eye Hospital)

• Foundation models for applications in pathology and ophthalmology was a pervasive theme throughout the workshop, with three excellent speakers rounding out the morning session with their insights into the current advancements and challenges in this rapidly growing space.

• The first talk was from Pearse Keane, a Clinician at Moorfields Eye Hospital and Professor at University College London, who is leading a multidisciplinary group of ophthalmology researchers at the intersection of health and AI.

  • The ophthalmology field is quickly becoming overwhelmed by increasing patient populations seeking treatment, and has surpassed orthopedics as the #1 busiest specialty in terms of clinical appointments in the NHS in the UK.

  • Medical foundation models trained on vast amounts of ophthalmic images could help alleviate some of this demand by helping to more efficiently and effectively diagnose patients, especially for diseases like age-related macular degeneration (AMD), which is one of the top causes of blindness in older populations.

  • To this end, Prof. Keane highlighted several efforts he and his group have been developing, including a collaboration between Moorfields Hospital and Google DeepMind. Together, their model was trained on 1.2M anonymized retinal optical coherence tomography (OCT) scans to identify some of the most common causes of blindness. More details on the research can be found in their Nature Medicine paper.

    

• Faisal Mahmood (Harvard Medical School)

  • The second talk was from Faisal Mahmood, a Professor at Harvard Medical School, who is developing generative models for applications in pathology.

    • Whole slide images (WSI) of histological samples contain a vast amount of biological information given the high resolution of digitally-scanned slides.

    • Prof. Mahmood and his group have developed several models trained on datasets of WSIs to predict patient diagnosis, prognosis, treatment response, survival, biomarker expression, and many other characteristics. More recent work has focused on building foundation models supplemented by other biological data including special stains, genomics, and transcriptomics to augment their ability to predict the above properties.

    • In addition to this work, he presented on a strategy to develop a generative AI copilot for applications in pathology analysis. The model, called PathChat, is a natural language chatbot-style interface between the user and their pathology data which enables the ability to ask questions like: “What tissue type is shown in this image?”, “Based on this image, what is the likely diagnosis for this patient?”, and “Can you synthesize a pathology report for this sample?” Check out their recent Nature paper to learn more about PathChat!

      

• Sheng Wang (University of Washington)

  • The third talk was from Sheng Wang, a Professor at the University of Washington, focusing on building generative multimodal foundation models for the biomedical sciences, including ophthalmology and pathology.

    • Similar to PathChat, the team developed their own foundation model called GigaPath, which was trained on ~171k pathology slides from more than 30k patients covering 31 tissue types. They showcased the ability for GigaPath to use chain-of-thought to better generate patient treatment plans by first predicting disease subtype, then predicting associated biomarkers, and finally a recommended therapeutic plan. More details on GigaPath can be found in their recent Nature paper!

    • In the final portion of his talk, Prof. Wang introduced their work on multimodal foundation models incorporating data from many distinct imaging techniques (e.g. ultrasound, pathology, X-ray, OCT, etc.). To achieve this, they implemented BiomedParse, a separate foundation model also developed by their group, to project data from 9 different imaging modalities onto text space. Effectively, this can cluster images representing similar phenotypes or tissue types together in text space making it easier for the model to learn from multiple image modalities at once.

      

• Overall, there were several major overlapping themes throughout these three talks that set the scene for future of biomedical foundation models.

  • Low adoption rates amongst experts in these fields can limit the development and usefulness of these foundation models. How do we best introduce new technologies into clinical settings that augment the work currently being performed by clinicians?

  • Foundation models are beginning to ingest a substantial amount of data across hospitals, institutions, and other clinical care sites. However, there is considerable bias introduced when combining datasets from disparate sources. These biases can be attributed to different personnel, instruments used for data collection, varying protocols, among many others. How do we sufficiently address these biases? The room’s consensus seemed like this problem may need to be addressed through both changes in model development and the way we collect and store our clinical data.

  • Foundational models require well-curated datasets. Traditionally, clinical data has not been sufficiently organized in a systematic manner to enable rapid integration into novel foundational models. A major focus moving forward will be working with care providers and institutions to standardize data storage with a particular focus on patient privacy.

AI + Science: Applications to Medicine; Anima Anandkumar (California Institute of Technology)

• Prof. Anandkumar opened her talk with a call to the community to consider more than just LLMs when building foundation models: “Foundation models tend to be synonymous with language models to many of you, but it’s not only natural language. I may hint a little bit about it, but most of my talk will not be about language models.”

• This quote set the stage for her presentation where Prof. Anankumar focused on the compelling idea that when building biological foundation models, we need to consider how to implement a truly multi-scale approach with a diverse repertoire of data.

• However, building these types of foundation models can be very costly because traditional methods have been based purely on numerical simulations of biology. These simulations can be very computationally expensive even for our most advanced supercomputers.

• Prof. Anandkumar and her group have been developing AI surrogates to replace these numerical simulations at many biological length scales: from quantum modeling of individual molecules, to simulating viruses, to building the first AI-based high-resolution weather model.

  

• To do this, their group introduced the use of neural operators. Unlike neural networks that learn to map finite inputs to outputs, neural operators learn how to map infinite dimensional function spaces. This means they are well situated for these types of multi-scale modeling tasks because they can simultaneously capture the complexity of different input processes at different resolutions.

  • For example, global weather patterns might look grainy if you zoom into the street-level view, but they become more useful when zooming far out. But in the same model, you may also want to include high-resolution mappings of fog data on the level of a single city.

  • In contrast to neural networks which typically have input and output data at a fixed resolution, neural operators are a useful tool to incorporate data at multiple resolutions.

  • Prof. Anandkumar went on to give several examples of how her lab has leveraged neural operators in their work, including for MRI and photo-acoustic image reconstruction from a wide variety of subsampling patterns.

• For the final portion of the talk, Prof. Anandkumar highlighted a recent set of projects from her group detailing how the AI technologies they developed can directly impact clinical applications.

  • Of note, one example included training surgical robots on multimodal feedback data collected during surgeries (e.g. video and audio captured from a camera mounted to the robot) with the ultimate vision to use these AI robots to help train the surgeons themselves.

  • Another application that she highlighted was the potential medical insight that could be gleaned from training AI models on wearables data (e.g. Apple Watch tracking your steps, sleep, heart rate). They are particularly interested in developing energy-efficient analog neural nets for electrocardiogram (ECG) arrhythmia classification.

Levels of Clinical Evaluation for LLMs; Karan Singhal (OpenAI)

• Carefully-executed evaluations of medical large language models (LLMs) are critical to their successful deployment to solve real-world problems.

  • (Medical) LLMs are quickly approaching a regime of “increasing capability overhang” meaning these models are much more capable than what we are currently testing and using them for.

  • Prior to deploying these models in real-world settings, we need a system of evaluations that enable researchers to build, test, and tune LLMs in a medical-specific context.

• In his talk, Karan systematically outlined a framework of 4 levels of clinical evaluation for LLMs and gave examples of published work from each: (Level 1) Automated Assessments, (Level 2) Human Ratings of Open Generations, (Level 3) Offline Human Ratings for Real Tasks, and (Level 4) Real World Studies.

• Level 1: Automated Assessments:

  • This first level of assessment often leverages the use of multiple choice questions (e.g. MedQA) in a similar style to how a medical student would be examined. This is often a measure of the model’s underlying knowledge and reasoning abilities, including questions around clinical diagnoses.

  • These evaluations tend to be relatively easy to execute and have fast iteration cycles; however, they lack the ability to test for real-world behaviors, often do not have a wide dynamic range to stress-test models, and suffer from contamination with model training data.

  • Interestingly, consumer LLMs (e.g. ChatGPT) tend to perform quite well on these tasks despite not being fine-tuned with specific medical knowledge for clinical applications.

• Level 2: Human Ratings of Open Generations:

  • The second level of assessment involves the model producing a longer, open-form response to a wide variety of questions, which is subsequently rated by a human expert. Typical tasks can include medical diagnoses from a list of symptoms and patient messaging services. These outputs are often compared directly to a physician, and to avoid reviewer-to-reviewer bias, the reviewers select the best answer from options derived from several LLMs.

  • These evaluations enable targeted measurement of model knowledge and behavior and tend to be more relevant to real-world situations than Level 1; however, biases in human ratings are difficult to control for, including biases towards output length and formatting, resulting in the need for carefully-crafted rubrics for grading.

• Level 3: Offline Human Ratings for Real Tasks:

  • The third level of assessment is structurally very similar to Level 2, but with a focus on real-world tasks that clinicians would perform regularly. This could include using a vision language model for producing radiology reports or electronic consultations between doctors.

  • These evaluations are advantageous because they can provide really useful performance metrics on real-world tasks without the need for a costly real-world study; however, they still share all the same limitations of a Level 2 study (i.e. human bias and standardized rubric design).

• Level 4: Real-World Studies:

  • The fourth and final level of assessment brings LLMs to a fully real-world context with real clinicians and patients directly or indirectly interacting with the model.

  • There have been relatively few Level 4 studies run to date, but Karan highlighted one recent paper that tested ChatGPT’s ability to assist with clinical documentation compared to human dictation.

    

  • These evaluations are considered the gold standard for their clinical relevance and strong comparators to current workflows; however, they often suffer from low sample sizes, costly and slow evaluation cycles, and continued bias amongst human raters.

• We are just at the beginning of productionizing these models in real-world settings. Many opportunities exist for current researchers to both build novel evaluations and test their LLMs using this proposed framework.

Machine Learning in Structural Biology

Workshop Summary

“Structural biology, the study of the 3D structure or shape of proteins and other biomolecules, has been transformed by breakthroughs from machine learning algorithms. Machine learning models are now routinely used by experimentalists to predict structures to aid in hypothesis generation and experimental design, accelerate the experimental process of structure determination (e.g. computer vision algorithms for cryo-electron microscopy), and have become a new industry standard for bioengineering new protein therapeutics (e.g. large language models for protein design). Despite all of this progress, there are still many active and open challenges for the field, such as modeling protein dynamics, predicting the structure of other classes of biomolecules such as RNA, learning and generalizing the underlying physics driving protein folding, and relating the structure of isolated proteins to the in vivo and contextual nature of their underlying function. These challenges are diverse and interdisciplinary, motivating new kinds of machine learning methods and requiring the development and maturation of standard benchmarks and datasets.

Machine Learning in Structural Biology (MLSB), seeks to bring together field experts, practitioners, and students from across academia, industry research groups, and pharmaceutical companies to focus on these new challenges and opportunities. This year, MLSB aims to bridge the theoretical and practical by addressing the outstanding computational and experimental problems at the forefront of our field. The intersection of artificial intelligence and structural biology promises to unlock new scientific discoveries and develop powerful design tools.” —Workshop Website

Overview

This Sunday at NeurIPS, the Machine Learning in Structural Biology workshop showcased just how far the field has come since AlphaFold 2's release four years ago. The packed room and poster-lined walls reflected a field that's earned a Nobel Prize and continues to evolve rapidly. Here are some key developments that caught my attention, though this is by no means comprehensive or chronological.

As is natural for a discussion about machine learning, the talks naturally split between advances in models and datasets.

AlphaFold 3

Starting off with models, Josh Abramson, a research engineer at DeepMind and the lead author of the AlphaFold3 (AF3) paper, provided an inside look into its development. We’ve previously written an in-depth guide to AF3, so I will gloss over the details, but the major advancement from AF2 to AF3 was expanding the representation to encapsulate the whole PDB — including DNA, ligands, and ions.

This required two major changes: standardizing the input and generalizing the structure model. AF3 keeps the per-residue or per-nucleotide token representation for proteins and DNA but switches to a per-atom basis for other molecules and performs additional processing upfront to produce a generalized input representation. To generalize the model, they replaced AF2’s structure model that predicted spatial frames and torsions to a generative all-atom diffusion model conditioned on the preprocessed inputs.

Interestingly, AF3 removed equivariance (molecule invariance to rotation/translation) with seemingly no loss in performance. This was a fascinating contrast to a talk a day earlier by Johann Brehmer from Qualcomm research who suggested that equivariance improves data efficiency, though they suggest that data augmentation can close this gap if training for a long time. Yet in the limit of infinite data, equivariant models are still more compute efficient than non-equivariant models.

One notable challenge for AF3 emerged with unresolved PDB coordinates. Since AF3’s diffusion model is only trained on ground truth data, for unresolved or disordered regions it would predict unphysical, bunched up structures. To address this, the team generated a “disorder” cross-distillation set using AF2 predictions on these unresolved coordinates in the PDB, allowing AF3 to nearly match AF2’s accuracy. This exemplifies the extensive dataset curation required for AF3’s development. Josh concluded his talk discussing AF3’s limitations including missing protein dynamics or physics, hallucination, and chirality mismatch.

Boltz-1

Jeremy Wohlwend offered a complementary perspective on AF3 through the lens of training Boltz-1, an open-source, accessible AF3 replication. His talk revealed how the subtle implementation details often omitted from papers may explain why replicated models sometimes exhibit unexpected behaviors — for instance, Boltz-1 sometimes incorrectly superimposes very large identical chains, a behavior that may not exist in AF3.

While most of the model architecture and training sets remained the same, Boltz-1 does introduce new optimizations like a new confidence model that mirrors AF3’s trunk to improve confidence predictions. Additionally, they also constructed multiple sequence alignments (MSAs) with MMSeq2 and Collabfold, a significantly faster MSA calculation from Milot Mirdita and the Martin Steinegger Lab at SNU (who spoke earlier), allowing full MSA computation of the entire PDB on a single node within weeks. As an aside, the field owes a lot to researchers like Milot and Martin for highly optimizing these fundamental computational functions and tools. We are excited to see how others continue to innovate from on top of these models such as Chai-1’s recent introduction of model restraints to further improve accuracy.

Conditional Modeling

While structure prediction was once the primary challenge, the field has pushed towards conditional generation or designing proteins with specific functions with releases like RFDiffusion or AlphaProteo.

Noelia Ferruz, a group leader at the CRG, kicked off the day presenting fascinating work on generating enzymes with a specific function using reinforcement learning. In collaboration with BaseCamp Research, her approach began with training a protein language model (ZymCTRL) on protein sequences labeled with catalytic functions. The zero-shot model could generate some initial functional and diverse carbonic anhydrases (35% success rate) and further fine-tuning on BaseCamp’s metagenomic datasets greatly increased success rates and resulted in active enzymes at extreme conditions.

Yet, these models produced enzymes that largely mirrored the training set properties, which is not conducive to generating out-of-distribution properties. Her group then took a really interesting agentic view of protein design (a broader trend this year at NeurIPS) to address this limitation. This agentic approach iteratively rewards good sequences while penalizing poor ones and maintaining reasonable deviation from the original model. With this framework, her group guided the model towards a diverse set of alpha carbonic anhydrases rather than only the beta variants. They also applied this framework to the Adapytv Bio Protein Design competition (Dimension is a co-host). With limited optimization, they generated multiple successful binders for EGFR, with two surpassing EGF’s affinity.

Another great talk from Amy Lu touched upon the debate between sequence and structure models (see our recent MoML coverage). She addressed this with PLAID, a method to directly sample from a joint structure-sequence distribution to generate all-atom proteins. Her key insight came from examining ESMFold's inputs - the zeroed-out pairwise input into the structure head suggested that ESMFold’s sequence embeddings alone could represent both structural and sequence distributions. Intuitively, this makes sense as ESMFold maps sequence to structure, so its intermediary latent space should contain representations of both. Since sequence databases are much larger and more richly annotated, PLAID is able to generate proteins conditioned on functional and taxonomic inputs. While initial attempts to train a diffusion model on these embeddings failed, developing and adding the CHEAP encoder to compress the embeddings (inspired by high-resolution image synthesis) enabled generation of diverse, high-quality proteins across varying lengths. Amy showed that PLAID was able to generate a variety of diverse all-atom proteins conditioned on both function and taxonomy at many sequence lengths. These generated proteins impressively retained conserved active site motifs or membrane hydrophobicity patterns consistent with their conditioned input function.

Datasets

Switching gears to datasets, earlier in the conference, Ilya Sutskever declared that data is the fossil fuel of AI, and that we’ve exhausted this precious resource. Yet these talks at MLSB suggest that biological research continues to yield novel datasets and clever generation techniques.

Improving Databases

In this vein, Andre Cornman, the CTO of Tatta Bio, spoke about aggregating new sources of data through their Open MetaGenomic (OMG) corpus. He suggested we could think of these sequences measured from our environment as nature's raw dataset - like how Common Crawl scrapes the entire web, while UniProt is more like Wikipedia's curated entries. These metagenomic sequences are captured with their full genomic context intact through contigs. The resulting database contains 3.3B proteins, offering 3x more diversity than UniProt. To demonstrate the value of the additional data, they train a genomic language model, gLM2, on this expanded dataset. gLM2 outperforms ESM2 on the Diverse Genomic Embedding Benchmark and they showed that it captures co-evolutionary signal of a protein-protein interface without an MSA that both ESM2 and Evo miss.

Sai Advaith Maddipatla and his co-authors explored how to better constrain structure predictions with experimental data. Focusing on X-Ray crystallography structures, they developed a method to handle regions where proteins show distinct alternative conformations ('altlocs'). While these altlocs have been cataloged by crystallographers, current models only model one of these altlocs. Their team developed a method to calculate the expected electron density from predicted structures and use that to guide a pre-trained diffusion model (Chroma) by minimizing the difference between calculated and experimentally observed densities.

With this guidance, the model successfully generates ensembles of conformations that match experimental densities - accurately predicting multiple distinct conformations in flexible regions while maintaining precise predictions for stable ones. This work provides a way for protein structure prediction models to account for some of the inherent dynamics and flexibility present in proteins. One of the interesting future follow-ups for the team involves refining the PDB, potentially creating a more accurate training resource for future models.

Generating More Data

Gabe Rocklin, a professor at Northwestern, presented on a scaled experimental technique to measure the stability of individual structures and folds in a protein. These individual flexibilities within a protein are highly therapeutically relevant, implicated in many dynamic processes like amyloid formation or aggregation of therapeutic antibodies. His lab’s approach leverages hydrogen exchange (HdX): when proteins in water are diluted into D2O, unfolded polar hydrogens are replaced with deuterium while backbone hydrogens in stable secondary structures retain their bonds. As regions unfold, they undergo rapid deuteration, with their exchange rates denoting local stability.

By combining oligo pool libraries with LC-MS, his lab quantified how many residues in each protein exchanged quickly versus slowly across different D2O exposure times, measuring this distribution across 6,000 proteins. This allowed them to quantify cooperativity — how many residues only unfold together during complete unfolding. Surprisingly, when attempting to predict cooperativity from this dataset, they found their linear models with a small set of curated features outperformed language models. They additionally were able to train a model to predict plausible sequence variants to improve cooperativity, though that is still early. These experimental measurements provide crucial constraints for future dynamical models and MD simulations.

Erika DeBenedictis closed with a vision for large-scale public projects to advance predictive models for biology. Drawing from her physics background, she highlighted how biology has seen few coordinated scientific efforts in comparison to physics (i.e CERN). Even the PDB, while a transformative resource, emerged somewhat accidentally over 50 years at a cost of ~$10B.

Through her non-profit Align to Innovate, she argued we can dramatically accelerate this timeline by creating clear roadmaps, partnerships, repositories, and benchmarks for specific datasets and models. Their first goal is to tackle protein function with the goal of collecting data and training models that can transfer learnings between functional families (technical roadmap). They propose using GROQ-Seq, a high-throughput, extensible assay that links cell growth to functional outputs to aggregate this dataset.

Success will require broad stakeholder buy-in and Align has already garnered support across academia, industry, and government bodies like NIST. It felt especially fitting that she concluded her talk by turning to the audience, polling them on what techniques they would like to see go head-to-head (somehow sequence vs structure edged out David Baker vs AlphaFold) or what functional data they preferred (antibodies won above transcription factors).

While parts of the field feel mature - AF2's release seems distant now - we're likely still in the early innings of its journey. These initiatives and projects could generate several PDB-scale datasets, enabling more Nobel-worthy breakthroughs. At Dimension, we're excited to witness this era of rapid progress — please reach out if you're working on something here!

GenAI for Health

Workshop Summary

“Generative AI (GenAI) emerged as a strong tool that can revolutionize healthcare and medicine. Yet the public trust in using GenAI for health is not well established due to its potential vulnerabilities and insufficient compliance with health policies. The workshop aims to gather machine learning researchers and healthcare/medicine experts from both academia and industry to explore the transformative potential of GenAI for health. We will delve into the trustworthiness risks and mitigation of cutting-edge GenAI technologies applicable in health applications, such as Large Language Models, and multi-modality large models. By fostering multidisciplinary communication with experts in government policies, this workshop seeks to advance the integration of GenAI in healthcare, ensuring safe, effective, ethical, and policy-compliant deployment to enhance patient outcomes and clinical research.” —Workshop Website

Speaker Summaries

Improve Medical GenAI Model’s Trustworthiness with Expert Knowledge and Synthetic Data (Daguang Xu, NVIDIA)

• Dr. Xu began his presentation by stating that model trustworthiness is a roadblock for the clinical adoption of generating modeling techniques.

• Gaps in model trustworthiness stem from issues with hallucination, safety, bias, and privacy.

  • Hallucination is a tendency of most generative models wherein they produce ‘real-looking’, but false data that can be challenging to root out given the black-box nature of these algorithms. Model explainability is critical for adoption.

  • Safety issues occur more at the user-level. That is, if physicians don’t know how to audit model results prior to acting on them—mistakes can be made in a clinical setting. Xu mentioned two relevant efforts to help here: Llama Guard and Nemo-Guardrails.

  • Bias stems from inadequate training data. Diseases or patient groups underrepresented in the training corpus have worse performance.

  • Privacy is always a critical issue when handling patient data.

• Dr. Xu focused his talk on hallucination and bias. Specifically, he argued injecting expert knowledge and synthetic data into GenAI models could remedy hallucination and bias, respectively.

• First, we went over Project MONAI (Medical Open Network for AI)—a collaborative initiative between NVIDIA and several academic institutions.

  • We focused specifically on a multi-modal model (VILA-M3) that serves as a VLM—a visual language model. It’s trained to input and reason over text as well as images (e.g., from medical scans).

  • M3 is built on the idea that general-purpose models are too prone to make errors, especially when given very specific data types or tasks. To mitigate this, VILA-M3 triggers specific, expert workflows from other models contained within Model Zoo.

  • For example, when prompted with a thoracic CT image, VILA-M3 launches the expert model VISTA3D which can segment and analyze these types of images.

• Second, we dealt with issues relating to model bias. This issue emanates from the long-tail problem in healthcare, that is when certain diseases are underrepresented in the training corpus and thus less likely to be classified correctly.

  • Dr. Xu postulated that augmenting datasets with synthetic data can increase performance in these cases.

Responsible Radiology Reporting in the Age of GenAI (Tanveer Syeda-Mahmood, IBM, Stanford)

• In this presentation, we again focused on medical imaging applications of VLMs in a radiographic setting. Specifically, Dr. Syeda-Mahmood discussed using VLMs to create generative radiology reports prior to manual review by radiologists.

• Responsible radiology reporting is not just about being accurate about what pathologies might be in an image—but also specific about what’s not found in an image. Models should avoid irrelevant descriptions—they shouldn’t be too assumptive or inclusive of information not contained in an image.

• Recall that many generative models were originally designed for creativity. If one prompts a radiographic model like Xray-GPT, it will generate different reports each time, which creates a lack of standardization that’s problematic in a medical context.

  • Dr. Syeda-Mahmood was careful to mention methods like RGRG that give stable reports, though the responses are canned and variety is missing—causing a trade-off.

• Responsible radiology reporting is critical because even if models are FDA cleared, they must earn clinician trust—an uphill battle. Models should be sensitive to demographic biases. They should be able to distinguish between an actual lack of evidence versus omission.

AI Agents for Biomedical Discoveries (James Zou, Stanford)

• Agentic AI was a theme of this workshop—and indeed a theme of the whole conference. AI agents require thinking about AI as a partner rather than just a tool.

• Professor Zou discussed creating a virtual lab in a multi-agent framework, where specialized agents communicate with each other to solve a problem.

• These semi-autonomous assemblies of agents have group meetings wherein they debate how to approach complex tasks. In their work, the group emulated both group and individual (1:1) meetings between agents to design nanobodies against a SARS-CoV-2 variant. The bots decided which computational tools to use, wrote additional code, and nominated a host of sequences that were tested in a lab. Read more in The Virtual Lab.

Demographic Bias of Expert-Level Vision-Language Foundation Models in Medical Imaging (Yuzhe Yang, UCLA)

• A major issue I’ve observed studying companies working in digital histopathology is the issue of generalization. Algorithms designed to make diagnoses or accelerate radiologist workflows sometimes see degraded performance in new health systems—whether that’s because of different instrumentation or changes in the underlying patient demographics.

• Professor Yang’s talk focused on uncovering and auditing the biases of VLMs, which can lead to inadequate treatment in the real world, such as under-diagnosis in less well-represented patient cohorts. (Source)

• The research team assembled chest x-ray images (n>800,000) from five diverse datasets (e.g., CheXpert). (Source)

• Next, they tasked a host of contemporary PLMs to perform zero-shot diagnoses using these images. While many overall AUC values were on par with individual radiologists (>0.9), the group fairness scores suffered significantly. This means that the VLMs showed statistically significant underperformance relative to radiologists on underrepresented patient cohorts.

PATIENT-Ψ: Using Large Language Models to Simulate Patients for Training Mental Health Professionals (Stephanie Milani, Carnegie Mellon)

• Stephanie Milani delivered an outstanding presentation on addressing mental healthcare gaps using LLMs.

• The World Health Organization (WHO) notes that 1-in-8 adults suffer from a mental health condition. Over half of US adults aren’t receiving mental healthcare services, partially due to inadequate therapist supply. Stephanie’s presentation dealt with the extent to which technology can mitigate this burden.

• After interviewing many mental healthcare professionals who felt their training inadequately prepared them to address real patients, the research group asked whether LLMs could be leveraged to simulate real-world patients, thereby improving provider readiness.

• Models like Chat-GPT cannot simulate patients out-of-the-box. Instead, Patient-Ψ seeks to first build cognitive models that mirror the typical behavior, emotions, thought patterns, and pathologies of patients with real psychological disorders. Constructing a patient’s cognitive model is a key aspect of cognitive behavioral therapy (CBT).

• Unfortunately, there are no open databases of cognitive models. The Patient-Ψ team utilized GPT-4 to create initial summaries of real therapy sessions. Real therapists then interacted with this formless summaries, generating 106 rich, diverse cognitive models using human expertise.

  

• My understanding is that the research group also used a conversational tone filter that augments and injects complexity into the cognitive models during inference time to better prepare trainees.

• Using 20 professionals and 13 trainees, the researchers evaluated Patient-Ψ against GPT-4 to assess whether the former more closely resembled bonafide cognitive patient models and whether it improved trainee readiness. The following plot summarizes the results:

  

Position: Participatory Assessment of Large Language Model Applications in an Academic Medical Center (Giorgia Carra, CHUV; Lausanne University Hospital)

• Dr. Carra brought a unique perspective as a senior data scientist employed at the University Hospital of Lausanne in Switzerland. Her work involves improving care quality and operational efficiency using ML techniques.

• Manifesting this vision involved assembling a hospital working group composed of 30 diverse healthcare professionals. These experts participated in moderated, multi-round discussion sessions to first identify and then score LLM use cases within the context of their healthcare system. The group’s output is shown below.

  

• In the eyes of regulators—LLMs are not technologies unto themselves. Instead, the United States FDA, European Law on Medical Device Regulations (MDR), and the European Union (EU) AI Act apply varying regulatory frameworks depending on the intended LLM use case. (Source)

• Based on their own experience and the most recent stipulations of the aforementioned regulatory frameworks, the working group voted automatic incident report classification and adaption of medical documentation into patient-amenable language as the two use cases most likely to be safe, compliant, feasible, and impactful.

• It will be interesting to follow this group’s work as they execute on this action plan and learn more about the deployment of healthcare LLMs into a prospective hospital setting.

Generative AI for Next-Generation Healthcare: Where Are We Now? (Yuyin Zhou, UC Santa Cruz)

• We began by reviewing several hot uses cases for GenAI in healthcare, including alleviating administrative burdens via LLMs, assisting with differential diagnoses using imaging data, supporting medical decision-making broadly, and predicting disease risk.

• Like other presenters, Professor Zhou remarked that data shortages may arrest the field’s rate of progress. Moreover, she noted that the lack of multi-modal data alignment is a challenge. This means that not all data across modalities is well annotated. For example, radiographic images might be lacking defined regions of interest (ROIs) or detailed diagnostic descriptions useful for model training.

• MedTrinity-25M is a large, semi-synthetic, multi-modal dataset spanning 25 million images with ROI and text annotations across 10 imaging modalities, 65 kinds of diseases, and 48 kinds of anatomical structures. As shown below, the authors leveraged a multi-modal LLM to enrich the original data with detailed annotations. (Source)

  

• MedTrinity-25M constitutes a rising tide—that is, pre-training using MedTrinity-25M boosts the performance of other vision-language models (VLMs).

• Critically, Professor Zhou highlighted several shortcomings of this approach, including inaccurate descriptions, non-specific disease annotations, ambiguous local-global relationships, and more. The talk concluded with the mention of several newer evaluation benchmarks that more closely track clinical utility, such as LancetQA and NEJMQA. (Source 1, Source 2)

Risk Assessment, Safety Enhancement, and Guardrails for Generative Models for Health (Bo Li, U. Chicago, Virtue AI)

• GenAI model safety and trustworthiness are key concerns amongst healthcare stakeholders.

• Professor Li walked through several examples of LLM jailbreaks—techniques that can be used during inference time to trick a model into violating its safety programming. For example, users can force a specific output format (e.g., YAML) or ask the model to tell a “fictional” story that contains instructions on how to make a salad with a poisonous mushroom.

• I appreciated that we walked through open-source LLM safety and regulatory compliance leaderboards (e.g., AirBench). These include benchmarks for performance, evaluation reports, and red-teaming strategies to make models robust against tampering. (Source 1, Source 2)

  

• Professor Li concluded by discussing GuardAgent, her group’s effort to mitigate safety and compliance obstacles with medical LLMs. GuardAgent is an LLM agent that inputs user-defined safety settings and analyzes the output of an upstream generative model. Should GuardAgent flag any safety violations, it can filter or deny the original output. (Source)

Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data (Sanmi Koyejo, Stanford)

• We focused on electronic health record (EHR) foundation models. Typically, these LLMs are trained on large bodies of EHR data to power downstream tasks like generating a medical discharge summary or making predictions about a patient’s future health status. (Source)

• EHRs are complicated data objects—quite unlike standard written text. A patient’s EHR is a timeline of clinical events split into multiple parallel tracks including diagnoses, prescriptions, lab tests, procedures, healthcare visits, and more.

  

• It’s standard practice to compress EHR data into a single, tokenized vector before passing the tokens through a sequence model (e.g., GPT) for generative tasks.

• Many contemporary LLMs have relatively short context windows. This challenges the use of EHR foundation models given lengthy patient histories.

• Given the underlying nuances and extended duration of EHR data, this talk focused on two guiding questions:

  • How to the unique properties of EHR data affect LLM performance?

  • Does long context outperform short context for EHR data?

• The researchers tested four architectures (GPT, Llama, Hyena, and Mamba (~120M parameters)) across context lengths ranging from 512 to 16k. They tested performance against the EHRSHOT benchmark for few-shot evaluation of EHR models. (Source)

• Perhaps unsurprisingly, the longest context (16k) Mamba model was most performant based on mean AUROC across 14/15 EHRSHOT tasks.

• This experiment highlighted three, specific EHR data nuances that all detract from LLM performance.

  • EHR data is often used for medical billing and coding. Therefore, the data contains repetitive strings of billing/coding text that degrade LLM performance.

  • Clinical events have irregular time intervals—some interactions take minutes while others can take years. Irregular token intervals also create performance issues.

  • Unlike natural text, the perplexity of EHR tokens increases over time. Consider that in standard sentences, the last word is usually the easiest to guess. If I say, “Eeny, meeny, miny, ___!”—you can probably guess the last word in the phrase. EHR data is the opposite. The longer the clinical histories are, the more complex and hard to model the morbidities might be, leading to increasing uncertainty in predicting next tokens.

• It’s clear the field needs to experiment with longer context models as well as develop special handling instructions to contend with the idiosyncrasies of EHR data.

AI for Novel Drug Modalities

Workshop Summary

“The primary objective of this workshop is to bridge the gap between AI and emerging drug modalities, such as gene and cell therapies, and RNA-based drugs. These modalities are important for disease mechanisms which were previously considered difficult, if not impossible, to target. They offer novel mechanisms of action, target previously undruggable proteins, and can address unmet medical needs with higher precision and efficacy. Traditional modalities such as small-molecule drugs, recombinant proteins, and monoclonal antibodies have formed the basis of numerous existing treatments. Recently, there has been growing recognition that AI methods have the potential to expedite breakthroughs in these established modalities. This same potential extends to the emerging field of new drug modalities, where AI can play a transformative role in accelerating the discovery and development process, overcoming challenges, and ultimately bringing innovative therapies to patients more efficiently and effectively. We aim to bring specialists in these modalities and ML communities together to discuss and investigate how AI can accelerate drug development in these emerging modalities.” —Workshop Website

Machine Learning-Guided Sequence Design for mRNA and Gene Therapy; Georg Seelig (University of Washington)

• Gene therapies acting on DNA and RNA hold significant promise for treating diseases linked to aberrant gene expression.

• Despite this promise, a largely unsolved roadblock in gene therapy has been the ability to control both gene expression magnitude and its specificity to select cell types in vivo.

• Prof. Seelig and his group studied and designed synthetic regulatory elements that can control these two aspects of gene expression (i.e. magnitude and specificity).

  • Regulatory gene elements are a class of DNA sequences that influence when, where, and how much a gene is expressed.

  • Enhancers are one type of regulatory gene element defined as non-coding sequences of DNA that contain transcription factor binding sites (TFBS) that, when bound, can directly regulate downstream gene expression in a cell-type specific manner.

  • These enhancers can range from 100s-1000s of base pairs in length, contain several other sequence elements aside from TFBS, and are often distally located from the genes they act upon, making it difficult to understand and design novel enhancers that are unique to specific genes and cell types of interest.

• To solve this problem, the authors leveraged a deep-learning framework to identify novel cell-specific enhancers by iteratively training their model with two complimentary datasets: (1) chromatin accessibility and (2) gene expression collected from massively parallel reporter assays (MPRAs).

• Chromatin accessibility:

  • Open chromatin is often associated with enhancer regions because transcription factors need to physically access and bind the DNA to induce downstream gene expression.

  • The authors used accessibility data collected by mapping DNase I hypersensitivity sites (DHSs) (>750k measurements). DNA that is less densely packed (i.e. open chromatin) is more physically accessible to the DNase enzyme and therefore susceptible to cleavage. Subsequently, cleaved DNA fragments are read out through high-throughput sequencing techniques for quantification of open chromatin sites.

  • However, solely relying on chromatin accessibility data to identify enhancer sequences is limiting - just because a region of DNA is theoretically “open”, does not mean that the sequence is an enhancer or that a transcription factor will be able to selectively bind to it. Hence, the need for the next dataset.

• Massively parallel reporter assays (MPRAs):

  • MPRA is a high-throughput method to screen regulatory elements in vitro through the introduction of plasmids containing unique enhancer sequences upstream of a unique genetically-encoded barcode. Quantification of enhancer activity for each plasmid is measured via high throughput sequencing by comparing the ratio of plasmid DNA barcodes to RNA barcodes expressed with the reporter gene.

  • The team utilized an MPRA dataset containing 30k measurements to train a convolutional neural net for predicting cell-specific enhancers for either K562 or HepG2 cell lines, enabling them to identify synthetic enhancers that outperform natural genomic sequences.

  • However, MPRA data is limited by the cell lines used to collect the data, and may not be representative when trying to apply the learnings from one cell type to another.

• Given the unique advantages and limitations with both of these datasets, the authors leveraged the experimental validation data from each model above as inputs to train a unified multimodal model.

  • Compared to training on individual data types, with this approach, they saw significant improvements in their model’s performance resulting in identification of enhancers that were ~100x more specific to HepG2 cells than K562.

    

• Future work from the group is also focused on training deep learning models from large accessibility datasets collected from 733 human samples encompassing 438 cell and tissue types. With this model they aim to predict novel enhancers across many cell types and validate them in vitro using MPRA experiments.

• Interested in learning more about this work? Check out the authors related preprint on bioRxiv!

Old Friends & New Partners: Accelerating Genome Engineering with Fine-Tuned LLMs and CRISPR-GPT Agent; Le Cong (Stanford)

• CRISPR-Cas9 gene editing (and its many derivatives) has truly revolutionized our toolkit for understanding and engineering the human genome, and Prof. Cong has been right at the forefront of this field.

• The Cong Lab has positioned themselves at the intersection of both the wet and dry lab when it comes to leveraging CRISPR. Their previous work has explored (1) the use of CRISPR for high-throughput genome engineering, (2) the development of single-cell technologies like Perturb-Track to study immune diseases, and (3) the construction of novel AI/ML tools for democratizing gene editing.

• Prof. Cong’s talk at this workshop focused on this third pillar, specifically presenting on how AI agents and biological foundation models can be leveraged to perform complex tasks that will accelerate our adoption and advancement of CRISPR-based tools.

  • He touched on the idea that existing general purpose LLMs (e.g. ChatGPT) are currently not well suited to excel at complex tasks in biology. For example, even GPT4 models only score ~60% when asked PhD-level scientific questions from the GPQA Diamond test, and they cannot design novel guide RNAs for a CRISPR target when prompted.

  • Instead, field-specific AI agents (the “new partners” he is referring to in the title of his talk) trained on biological datasets should take place as the new gold standard for guiding biologists in experimental design and analysis.

• To this end, they introduce CRISPR-GPT, a purpose-built AI agent for genome/cell engineering, with three distinct “modes” that dictate its output: (1) Meta Mode, (2) Auto Mode, and (3) Q&A Mode.

  • For all the following modes, CRISPR-GPT leverages advanced LLM capabilities (e.g. Chain of thought, instruction fine-tuning), computational skills (e.g. web searching, code generation), and bioinformatic tools (e.g. Primer3, CRISPRitz) to execute on each request by the researcher.\

    

  • In Meta Mode, researchers can get guidance on gene editing workflows that are pre-defined by experts in the field. For example, one could select a task from a curated list such as “Generate a knockout using CRISPR for Gene X.” The agent would then take you through a step-by-step guide on selection of the sgRNA, experimental conditions, analysis techniques, and more.

  • In Auto Mode, researchers can prompt CRISPR-GPT in natural language to receive customized guidance on their own workflows. For example, one can ask “Which CRISPR system should I use to repress TGFBR1 expression and how should I design my sgRNA?” CRISPR-GPT will then create an on-demand guide to inform and execute their experimental design.

  • In Q&A Mode, researchers are free to ask CRISPR-GPT ad-hoc questions related to field norms, experimental design, method comparisons, and much more. For example, one can ask “What are the differences between Cas12a and Cas12f?” and get back real-time answers to directly inform research directions.

• Grounded in language models like ChatGPT, tools like CRISPR-GPT have significant potential to redefine the way we design and run experiments. Using the platform, they showed that CRISPR-GPT running in Auto Mode on top of GPT4o doubled accuracy scores in their benchmarking tests compared to GPT4o alone. Benchmarking included tests on experimental design, sgRNA design, delivery selection, CRISPR knowledge, and more.

• Interested in learning more about CRISPR-GPT? Check out the preprint on arXiv!

Toward AI-Driven Digital Organism: Multiscale Foundation Models for Predicting, Simulating, and Programming Biology at All Levels; Eric Xing (GenBio AI, CMU, MBZUAI)

• Modeling biology is inherently a multi-scale endeavor, from quantum interactions between atoms in a protein to the cross-talk of different organ systems within a patient.

• To start the talk, Dr. Xing proposed an exciting question: “How do we combine all of these length scales into a single foundation model?”

  • Notably, in Dr. Xing’s view, biological education and practice has traditionally been siloed between “departments” (e.g. biochemistry, molecular biology, cell biology, tissue biology, systems biology, etc.) as well as a delineation between organ systems (cardiovascular, neural, gastrointestinal, etc.).

  • The ability to develop novel treatments for diseases often requires our unified understanding of many different scales of biology at once, and this current organizational framework limits interdisciplinary innovation and the ability to holistically model biological systems.

  • Biology isn’t in a vacuum. It always has a context. How do we bring this context into play?

• To answer this question, Dr. Xing and the team at GenBio proposed a vision towards large foundation models (FMs) that can help us holistically understand and engineer biology across many different length scales. In this talk, he outlined a roadmap for building out a unified and integrated system of FMs for biology, which he broke down into three stages: (1) Divide and Conquer, (2) Connect the Dots, and (3) Align and Optimize Across Scales.

  

  • In Divide and Conquer, they proposed a build-out of several FMs unique to specific biological scales and functions. These FMs include individual sequence and structure FMs for the molecular scale, DNA, RNA, and proteins, as well as an Interactome FM, Cell FM, Tissue FM, and Phenotype FM. These FMs are trained from integrated public and proprietary datasets (e.g. Genebank, Uniprot, KEGG, ENCODE, TCGA, Human Protein Atlas, etc.).

  • In Connect the Dots, they proposed a framework to integrate several models together to better inform certain biological tasks that require a richer representation of the surrounding environment. Notably, Dr. Xing emphasized the need for integration of both sequence- and structure-based FMs for tasks such as the inverse folding problem to predict sequence from structure.

  • In Align and Optimize Across Scales, they proposed co-tuning these models to work more effectively together at more complex modeling tasks. Applications of a fine-tuned, holistic FM may include applications in cell engineering such as predicting the effects of drug or gene knockout perturbations or applications in phenotype engineering such as predicting how genetic mutations could impact disease progression and inform dietary suggestions.

• Following this walkthrough, the talk ended with a call for re-defining the way we view the proverbial “lab-in-a-loop.” The wet-lab of the future should not be considered a search tool, but instead a tool for validation. The computer should take its place as the search tool.

• Want to learn more about their proposal for the future of biological foundation models? Check out their preprint on arXiv!

DeepADAR: Modeling Regulatory Features of ADAR-Based RNA Editing for gRNA Design; Andrew Jung (Deep Genomics, U. Toronto Vector Institute)

• Adenosine deaminase acting on RNA (ADAR) based editing platforms are garnering a growing interest among researchers for their ability to both (1) correct genetic mutations on the RNA level and (2) be used as a broader platform for controlling gene expression.

  • ADAR editing platforms work by making an A-to-G edit on double-stranded RNA (dsRNA) molecules.

  • Therefore, in order to target a specific RNA of interest in the cell, one must design and deliver a guide RNA (gRNA) that forms a dsRNA complex complimentary to the target of interest.

  • Two key interrelated determinants for whether the ADAR enzyme successfully edits the target sequence are (1) the presence of specific base mismatches along the dsRNA complex and (2) its underlying 3D structure.

  • This poses a significant challenge for researchers wanting to leverage ADAR against a specific RNA target: the underlying rules that dictate where these mismatches should be placed and the requirements for a proper 3D structure are not well defined, and the design space is often too large for experimental methods to be exhaustive.

• To address this problem, Andrew Jung and his colleagues trained a deep learning model to learn regulatory features of RNA editing and predict gRNA editing against a particular target of interest.

  • Current databases of ADAR editing sites are limited and contain a substantial number of false positives. These datasets can be filtered for higher quality editing sites through the use of public whole genome sequencing (WGS) and RNA-seq datasets that inherently provide greater evidence of RNA editing by ADAR.

  • This information is then used for model training: for a given region of RNA, output a scalar value for every “A” in the RNA sequence predicting the probability of editing.

  • When running the model, the input is both RNA sequence and structure, as both of these characteristics are well-known to impact the efficiency of ADAR editing, especially in the context of predicting gRNA-induced editing.

• At the end of the talk, they evaluated DeepADAR’s ability to predict both endogenous RNA editing and gRNA-based editing.

  • Endogenous RNA editing was predicted using a CRISPR mutagenesis dataset covering 366 mutations across 3 genes. Their results show an average predictive score across the 3 genes of ~0.85 (area under receiver operating characteristic curve (auROC)). For some context, a score of 0.5 is equivalent to random guessing, while a score of 1 is perfect prediction.

  • To predict gRNA-based RNA editing, the authors experimentally measured editing efficiencies of 240 gRNAs from ~80 unique editing sites across 6 genes. By fine-tuning the previous endogenous model on their experimental data, they were able to achieve a ~0.75 spearman correlation for on-target editing performance.

• Future directions for the team involve making these models more tissue and cell-type specific (as endogenous ADAR expression and activity is known to be higher in certain cell types) and incorporating information on chemical modifications that impact RNA structure.

Adaptyv Bio Protein Design Competition; Julian Englert (Adaptyv Bio)

• We also had an update from Julian Englert, Adaptyv Bio’s CEO, on their Protein Design Competition.

• For the competition, they asked protein designers from around the world to submit designs for binders against EGFR.

  • EGFR plays an important signaling role in driving cell proliferation and differentiation across several tissue types.

  • Mutations in EGFR and mis-regulation of its expression have been associated with several diseases, most notably cancer.

  • Current FDA-approved mAb therapies targeting EGFR (e.g. Cetuximab) have demonstrated the promise of inhibiting EGFR transduction and paves the way for novel protein binders in this space.

• For the competition, Adaptyv Bio received over 1000 protein designs sourced from 100 different designers.

  • These designs were ranked in silico based on: (1) computational metrics from AlphaFold, IPSA, ESM2, among others and (2) how unique and interesting the design approaches were.

  • The top candidates were selected and tested in Adaptyv Bio’s lab, to which there were ~50 validated binders out of ~400 total designs.

• Two of the winning designs from (1) Aurelia Bustos and (2) Constance Ferragu were featured at the workshop.

  

• Aurelia Bustos (MedBravo)

  • Their approach was to optimize natural EGFR ligands such as EGF, TGF-alpha, Betacellulin, and others, with the rationale that binders based on endogenous ligands with specific mutations could potentially be good candidates.

  • For each of the ligands, they computationally identified the probabilities of contact between each animo acid with EGFR using AF3 and kept these amino acid residues in their final design if they were conserved amongst 5 of the endogenous binders or if the contact probability was > 0.9.

  • They then masked any amino acids that did not satisfy these criteria and used ProteinMPNN to comprehensively explore variants of the masked regions using inverse folding (i.e. sequence prediction from structure), generating 1000s of novel amino acid sequences.

  • Finally, they trimmed these designs based on a ranking pipeline of computationally-predicted metrics and validated final structures using AlphaFold and Chai1.

  • Interestingly, they found that only those designs derived from EGF and Betacellulin had success in the lab.

  • Their top design resulted in a binding affinity of 51.6 nM, and you can check out more details of their that binder here!

• Constance Ferragu (Cradle Bio)

  • “It took us 30 minutes over some kombucha.” This was the opening line from Constance when answering the question “What did it take?” to make their binder design.

  • More specifically during that 30 minute window, the team started with scFv-formatted Cetuximab as their base protein and constrained themselves (as per the competition’s rules) to impose a minimum of 10 mutations in their final construct.

  • Interestingly, they did not make any mutations within the complementarity-determining region (CDR) (i.e. the variable domains on the light and heavy chains that directly interact with the ligand of interest) of the original Cetuximab antibody. Instead, they focused on introducing mutations within the framework regions (i.e. the relatively conserved regions flanking the CDR) in a process referred to as framework engineering.

  • With those constraints defined, they performed an overnight run of the algorithms developed at Cradle, which resulted in a binder with 1.21 nM binding affinity against EGFR (8x greater than the Cetuximab scFV).

  • Check out the details of their top binder here!

• Want to see more details on all of the contest entries? Check out the website here!

This piece includes a mixture of notes and opinions on several selected talks from last month’s Molecular Machine Learning (MoML) Conference put on by the Jameel Clinic at MIT. We hope it’s useful! We look forward to posting more of our notes following this week’s bio-related workshops at NeurIPS!

Subscribe now

The State of Antibody Engineering

Dr. Karl Dane Wittrup, co-founder of antibody pioneer Adimab and professor at MIT's Koch Institute, set the day’s tone with his view of the current state of antibody engineering.

My interpretation of Dr. Wittrup’s talk was that he views monoclonal antibody (mAb) discovery as a mostly solved problem and that the impact of machine learning (ML) to this domain is exaggerated.

The argument was convincing. Wittrup noted that cost, time, and failure risk accumulate throughout a drug campaign and don’t solely emanate from the discovery phase. Specifically, he called out the fact that target identification and clinical trials are disproportionately large contributors to cost, time, and failure risk. This is objectively the case.

Using the iron triangle analogy—where you must choose two from the set of good, fast, and cheap outcomes—Wittrup broke down his view further:

• If you want a good mAb quickly, you need to go to a specialized development partner (e.g., Adimab) and part with single-digit royalties.

• If you want a mAb cheaply and quickly, you can go to a commoditized CRO, but your molecule may not be that good.

• If you want a good mAb cheaply, you can leverage an internal development team, which may not work that quickly.

ML-driven mAb discovery is positioned to break the iron triangle. There may be a near-future where you can have a good, cheap, and fast mAb. Wittrup’s point seemed to be that this isn’t as impactful on balance as it may seem. He claimed that:

• We can already create good mAbs. The quality bar set by ML-discovered mAbs will be just as good, but perhaps not better than what traditional discovery techniques already get us.

• Antibody discovery is already fast. Push-button mAb discovery shaves just a few months off of a 5-10 year discovery cycle dominated by target selection and clinical translation.

• Virtually free mAbs are great, but they may only help you dodge a royalty payment to a development partner. You're still on the hook for $50-100M in clinical trial costs.

The talk concluded with the notion that antibody discovery is a mature field. Wittrup noted that elite mAb discovery platforms can almost always discover viable molecules. He seemed to view ML-driven approaches as highly relevent instruments in the mAb discovery armamentarium, but not as revolutionary tools that would flip antibody engineering on its head.

Given the audience's devotion to molecular discovery, this opening session seemed to take folks by surprise. In my view, Dr. Wittrup offered a pragmatic perspective on not reinventing the wheel. I’m confident his intent was not to discourage the ML scientists in the room from focusing on molecular discovery. While I agree with Dr. Wittrup’s central thesis, I may have a few contrarian views:

• As I wrote in a recent piece, I think ML-designed antibodies will break the iron triangle in the very near future and that this will disintermediate traditional antibody design services.

• I’m not confident that existing service providers can gradually integrate ML. My sense is that ML infrastructure and strategy need to be in place from day one, which could allow a new wave of in silico service providers to establish themselves.

• Given the nascency of traditional screening techniques for emerging biologics categories (e.g., bispecifics), I see real opportunity for ML techniques to enable differentiated design capabilities for these next-generation medicines.

• As indications and targets become crowded, fast-following has become a legitimate clinical strategy. Every month matters here and ML can help bridge the gap.

• ML excels at multiparameter optimization in a way that standard mAb screening doesn’t. If we’re able to leverage data on developability properties, feeding these back into the design process could reduce the likelihood of downstream failure.

This first presentation was excellent. Dr. Wittrup put into perspective the real problems that matter in drug discovery and development. I share many of his views, though I may skew slightly more bullish on the applicability and impact potential of ML-driven techniques for biologics.

Language Models for Programmable Protein Therapeutics

Pranam Chatterjee shared his perspective and insights as an ardent supporter of language models as an optimal substrate for powering the future of generative protein design. Pranam’s vision is to create tools that make it simple to manipulate proteins—the same way that tools like CRISPR have made it simple to edit the human genome.

If the opening talk served as a litmus test for the current state of protein engineering, Pranam’s presentation was a tour through what might be possible going forward with computational approaches

Pranam’s lab has been prolific, having produced an array of tools that build on top of ESM2, a pre-trained large language model (LLM) that creates useful embeddings of protein sequences to power downstream tasks.

One application area involved his lab’s work using protein language models to design peptides that can bind target proteins intracellularly. Specifically, Pranam spoke about using this approach to rapidly generate ubiquibodies (uAbs)—peptides that bind target proteins to enable ubiquitination and subsequent degradation via the proteasome.

We’ve written previously about target protein degradation (TPD) and have made an investment in a company leveraging small molecule molecular glues—Monte Rosa Therapeutics (GLUE). Hence, we are enthusiastic of TPD and care deeply about new approaches to this space.

Expanding the scope of protein targets amenable to TPD is the field’s central charge. One major drawback with some degraders (e.g., PROTACs) is that the target needs to have a druggable binding pocket to design a warhead against. Proteins that are smooth, intrinsically disordered, or are otherwise lacking a canonical binding site require special treatment.

Pranam ran through a host of his lab’s methods that are relevant for building motif-specific uAbs including PepPrCLIP, PepMLM, MoPPIt, and SaLT&PepPr. Using a combination of these tools, Chatterjee gave the case for uAbs and their closely related cousin, duAbs, which stabilize target proteins.

The hallmark argument is that peptide-based degraders and stabilizers promise a more modular solution than small molecules. Moreover, he argues that uAbs and duAbs can be generated from sequence data alone, ostensibly obviating the need to acquire a co-crystal structure or design against a specific, ligandable binding pocket. Buttressing this claim, Chatterjee walked through some experimental data wherein his lab showed in vitro, uAb-mediated degradation of historically recalcitrant protein targets such as beta-catenin.

I’m very excited by this approach and I was glad to see Pranam highlight several areas where these tools have yet to improve.

Specifically, Chatterjee spoke to a model called PTM-Mamba, which injects signal from post-translational modifications (PTMs) into the design process. Forming a ternary complex with the body’s degradation machinery is a sensitive process. PTMs can influence complexation and may be relevant to designing better uAbs or duAbs. While it’s still early, this seems like a practical direction to go.

Peptides typically have very poor in vivo PK, which is why they’re difficult to turn into drugs. We’ve written about this as a motivator behind the rise of non-canonical amino acids (ncAAs), which can increase plasma stability of peptides by protecting against proteolysis. ncAAs also afford medicinal chemists an exquisite level of conformational control over peptides, so I would be interesting to see ncAAs more involved in computational workflows.

Lead optimization for degraders of all types—including uAbs—can be fickle. The formation of a ternary complex is a necessary, but not sufficient condition for an optimal degrader molecule. Ideally, the drug is able to stabilize a conformation that exposes a lysine residue to the degradation machinery. The closer the lysine, the faster the ubiquitination and the more potent the degrader. Whether or not proximal lysine exposure can be learned through data scaling is a question I don’t know the answer to yet, but would be interested to find out.

Pranam’s talk was a tour de force through the many different tools that can be built atop powerful, pre-trained protein sequence models like ESM2. I’m eager to see how much better these frameworks can get with improved underlying embeddings, as contained in newer protein language models (pLMs).

Panel Discussion: Sequence & Structure

The day’s lone panel centered on the perennial debate between protein sequence versus protein structure as the optimal representation for generative models. While intellectually stimulating, I’m not in the business of taking a side here. Industry will use whatever works. In the grand scheme, there’s not a lot of bio-data to go around, so I’m sure folks will continue to use all the data they can get their hands on—regardless of what format it’s in.

I will try to present both sides to this argument based on some of the points of the speakers during this panel.

1. Structure Generative Modeling

   1. Structure generative models are trained on 3D structures from experimental tools like x-ray crystallography. These models embed structural representations of molecules using geometric coordinate systems. They reason in and generate 3D structures.

   2. Though structural data is information-rich, there’s very limited amount of it in the public domain. The largest, open database of 3D structures, the Protein Data Bank (PDB) contains on the order of 10^5 structures. Structural data is more expensive and challenging to generate.

   3. These models often seek to map directly from 3D structure (which involves continuous embeddings) to a biological or chemical function of interest, yielding a relatively smooth optimization surface. That is, small changes in the input space correspond to small changes in the output (function) space.

   4. An example of a ubiquitous structure generative model is AlphaFold3, which I’ve written extensively about here.

2. Sequence Generative Modeling

   1. Sequence generative models are trained on 1D sequences of DNA and/or proteins obtained via next-generation sequencing (NGS). These sequences are often annotated with functional data, enabling a direct link between between sequence and function.

   2. These models are exposed to structure implicitly through sequence data alone. Trained on enormous amounts of protein sequences, they learn to capture structural relationships by detecting co-evolutionary patterns, positional correlations, and conserved motifs that reflect physical constraints in folded proteins.

   3. Sequence space can sometimes prove challenging to optimize in. This is because it’s discrete—small changes in sequence can cause massive changes in structure, and thus, function.

   4. Offsetting this is the fact that sequence data is both relatively abundant and cheap to generate. UniProt contains on the order of 10^8 sequences, for example. Modern protein expression systems, single-cell partitioning, and NGS have made it facile to collect more annotated sequence data.

My favorite bipartisan takeaway involved a recasting of this debate from sequence versus structure to structure implicit versus explicit. The idea is that all models are exposed to structure. Structure explicit models are directly trained on structure. They reason in and generate structures. Meanwhile, structure implicit models are trained on sequence, though structure is implicitly captured in the embeddings, whether that’s through learned co-evolutionary statistics or otherwise.

One interesting observation revolved around biases in structural data acquisition methods. James Fraser and Mark Murcko have a beautiful paper about this entitled Structure is Beauty, But Not Always Truth. The essence of this argument is that structural representations are biased towards low-free energy conformations and that these snapshots occur in non-physiologic environments (read: not in a cell). Meanwhile, sequence data is not assumptive at all. It may excel in areas where traditional structure-based paradigms struggle, such as targeting intrinsically disordered proteins.

Meanwhile, another salient point focused on the therapeutic relevance of de novo protein-protein interactions (PPIs)—that is, PPIs that aren’t found in evolutionary databases. For example, therapeutic antibody-antigen interactions are new to Nature. While we can learn general features about antibody-antigen interactions from databases like OAS, we can’t rely on co-evolutionary couplings in the same way we can for intra-protein interactions or for commonly occurring complexes. The debate over whether co-evolutionary information constitutes a learned form of biophysics rages on. Explicit structures were discussed as a more viable way of capturing biophysical phenomenon and assisting with optimizing de novo binders.

Like I said at the outset—I’m still learning to appreciate both sides of this debate. I think it’s incredibly thought-provoking to wonder where each path ends. I’m paying special attention to co-generation models that make use of both forms of representation. There are successful businesses that’ve built their foundations on either approach. If you have a strong opinion—feel free to get in touch with me.

Implicitly Guided Design with PropEn: Match Your Data to Follow the Gradient

On behalf of Prescient Design and her collaborators at NYU, Nataša Tagasovska gave an excellent talk on a new ML framework—PROPerty ENhancer (PropEn). This method seems especially useful for optimizing desirable properties in scenarios lacking extensive training data, such as the life sciences. Following in silico experiments, the team validated PropEn with wet lab experiments of optimized antibody binders.

Generative algorithms seek to compress the iterative design-make-test-analyze (DMTA) cycles that underly most optimization processes. Many generative models are composed of both a generative component and a discriminative component. The former proposes new designs and the latter evaluates those designs.

An accurate discriminator is like a compass. Instead of maximizing the binding affinity of an antibody, let’s say your goal is to find the northernmost point on a tract of land. Each step is a generative process. The compass steers that generative process along an efficient path towards your goal. Without a discriminator, or compass, you’d be left to wander.

Discriminators require sufficient training data to accurately predict properties. With enough labeled datapoints, discriminators can approximate even complex mappings between the input (design) space and the output (property) space. Unfortunately, labeled data is scarce and expensive in the life sciences.

Let’s imagine that I’m training a generative antibody design model using structural representations of proteins. My input space might include 3D structures of antibody-antigen complexes obtained via x-ray crystallography. Perhaps the main property I am optimizing is binding affinity. Mapping my input space to my output space requires me to pair each crystal structure with an experimentally determined binding affinity measurement obtained via surface plasmon resonance (SPR), for example. This full process can cost thousands of dollars per (x,y) pair—roughly five orders of magnitude more expensive per datapoint compared to Internet images.

PropEn proposes the provocative question: What if there were a way to guide a generative process without training a discriminator? What new advantages and disadvantages would such a method have in low-data regimes?

The authors refer to their approach as implicit guidance since it doesn’t make use of an explicit discriminator to guide property optimization. Instead, PropEn cleverly groups datapoints together in a way that (a) grows the training dataset and (b) “inherently [embeds] the direction of property enhancement”. This may not be entirely intuitive at first glance, so I’ll break it down step-by-step.

Let’s start with the datapoint grouping procedure. The aim of this step is to find datapoints that are close together in input space (x). We first define a parameter (Δx) that quantifies how close together points must be to grouped. We next define a second parameter (Δy) that quantifies how large a property improvement is required between points x₁ and x₂ for them to be matched.

Grouping datapoints is important because it expands the training set exponentially. Consider that every pairwise relationship between the original datapoints is now a new datapoint. Let’s imagine I have four original datapoints:

1. (x₁, 1.0), (x₂, 1.5), (x₃, 2.0), (x₄, 2.5)

2. Assuming my threshold values allow for all possible matches, I can create six (6) pairwise relationships between these points:

3. (x₁,x₂), (x₁,x₃), (x₁,x₄), (x₂,x₃), (x₂,x₄), (x₃,x₄)

4. The maximum number of pairwise relationships I can create from n points can be represented as n(n-1)/2.

   1. For example, with n=4, this is: 4(4-1)/2 = 4(3)/2 = 12/2 = 6

5. This new collection of datapoints (pairwise relationships) scales quadratically with n.

Why does having this larger dataset matter? How can we use it to implicitly guide an optimization process? Let’s imagine a new scenario where the datapoints are portraits where some are clearly better than others. After grouping, we have:

1. Portrait 1 and Portrait 2 are similar (matched pair), but 2 has much better lighting (better y-value).

2. By studying these pairs, we can teach the model that when it sees lighting like Portrait 1’s, it should adjust the lighting to look more like Portrait 2.

3. In this way, the improvement direction is implicitly captured in the fact that Portrait 2 is similar (but better than) Portrait 1.

4. There’s no need for an external discriminator (e.g., an art critic) to tell you if the change you made is good.

Once the new matched dataset has been built, the authors train a deep learning model to be able to make improvement suggestions given a point in input space. They find that this approach learns a rough approximation of the property gradient.

Normally, one would need to train an explicit discriminative model capable of predicting the property value (y) given an input (x). The gradient of this function is a vector (direction) that shows how to alter x to improve y the fastest. Using matched pairs, implicit guidance can improve properties without the need to calculate the gradient.

Let’s revisit our compass example from earlier, but this time in three dimensions. This time, there’s a hill you’re trying to summit. You’ve been blessed with a magical compass that points towards the top of the hill. Using an explicit approach, you’d follow your magical compass (gradient) up to the peak.

The implicit approach is different. You’re less resourced and don’t have a magic compass. Instead, you’re given a collection of photos from previous, successful climbers. Using pairs of photos, you think to yourself, “From this spot, they moved to that spot and gained elevation.” You can repeat this process using chains of photo-pairs until you reach the summit—this is implicit guidance.

I really like PropEn because it’s so relevant to the issue of data paucity in the life sciences. Moreover, the few-shot methodology is very practical—one where you can use wet lab-generated data to quickly converge on an optimal design. My understanding is that PropEn works with both continuous and discrete data, making it compatible with both structure and sequence generative models. I’m eager to see this method continue to improve, including becoming viable in a multi-objective optimization setting.

Thanks for reading some of our thoughts on the presentations at MoML. We hope to meet you next year! Anyone working in this space is encouraged to reach out to me as we’d love to chat.

All of us are currently at NeurIPS taking notes—so expect more from us soon!

RMC-6236 is a macrocycle with a curious mechanism of action (MOA). It glues an innocuous protein (CYPA) to a family of known oncoproteins (RAS), creating a de novo PPI that disables downstream, oncogenic signaling. I'll first introduce PPIs and disease, survey macrocycle therapeutics, and combine these into a brief overview of RMC-6236.

Subscribe now

All of biology descends from molecules, like proteins, bumping into each other. Health and disease can be interpreted through the lens of PPI networks. Essentially, there's a natural order of PPIs that constitutes a healthy, homeostatic state in human beings. Disruptions in this pattern can be causal for or associated with a disease process. Therapies are our attempt at prodding a dysregulated PPI network back into a healthy shape via an external stimulus.

This partially explains why target discovery and selection are so challenging. Diseased networks create hundreds, if not thousands, of aberrant PPIs. How do we know which PPI is causal for the disease? If we intervene on the putatively causal PPI, is it enough to rescue the diseased phenotype? By affecting the disease-causing PPI, will unwanted and potentially toxic PPIs manifest?

These questions are a luxury because PPIs themselves can be extremely difficult to drug. Typically, when proteins interacts with each other, they do so using large, exterior surface patches. These hot spot regions are often smooth and flat, characteristics that make proteins notoriously challenging to develop small molecule drugs against. So how do we manipulate PPIs if not with traditional small molecules?

We can drug enormous interactions with enormous molecules. That's what biologics (e.g., antibodies) are for. Keytruda (pembrolizumab), one of the most valuable drugs on the planet, is a monoclonal antibody and PPI modulator. Specifically, Keytruda blocks interactions between the proteins PD-1 and PD-L1/PD-L2. These proteins all happen to be membrane proteins (receptors). This fact underscores one of the main limitations of antibodies. They can only drug extracellular targets.

Despite their inherent advantages such as high target specificity and in vivo stability, antibodies are simply too big to squeeze inside a cell. This is problematic when you consider that most proteins, and thus most PPIs, exist inside of cells. Besides the fact that intracellular PPIs are inaccessible to antibodies, there are other reasons why drugging a receptor may be suboptimal to rescue a disease process.

Receptors are often the first nodes of multimember, downstream signaling cascades. Kicking in the receptor is a good way to shut down all of this interior cellular circuitry. It's like CTRL-ALT-Deleting a large swath of PPIs. This may be effective, but could also cause unintended side effects.

Macrocycles are one exciting option to surgically modulate specific, intracellular PPIs. Macrocycles are peptides, a broad class of therapeutic that exists in the no-man's-land between small molecules and biologics. Peptides historically haven't made good drugs because they have poor in vivo stability. There's a renaissance happening with peptide drugs partly driven by the marquee success of drugs like semaglutide (e.g., Ozempic) which mimics an endogenous peptide hormone (GLP-1).

Unlike linear peptides, macrocycles have backbones that loops back on themselves to form roughly circular compounds decorated with functional groups. The circular shape imparts greater structural rigidity and defends against degradation by enzymes in the bloodstream. Macrocycles aren't rigid like a hula-hoop, through. They're more like rings of spaghetti. This flexibility is both a blessing and a curse.

Macrocycles exploit their conformational flexibility to become chameleonic, a quirky characteristic of compounds that exist beyond the rule-of-five. Chameleonicity means a molecule can adapt to its environment via conformational changes. Macrocycles can shield their hydrophobic residues, allowing them to become cell permeable despite their size.

Unfortunately, conformational flexibility is burdensome when screening and optimizing macrocycles. Virtual screening is a popular technique in small molecule drug discovery. Here, many designs are ranked in silico according to some scoring function. This function is often physics-based and can involve lightweight docking calculations. The forcefields involved generally aren't fast enough to deal with all the floppiness of macrocycles, greatly limiting the throughput of virtual screening campaigns.

Lead optimization can be tricky as well since macrocycles by their nature violate many cardinal rules of medicinal chemistry. Normally, med chemists want to make their compounds smaller, flatter, more linear, and more rigid. Macrocycles do the opposite.

Despite these challenges, macrocycle development seems worth it. Ostensibly, they're a readily manufacturable modality that can be made orally bioavailable, stable in vivo, with enough polar surface area to modulate intracellular PPIs in a precise fashion. RMC-6236 is a perfect example.

RMC-6236 is an oral macrocycle being purposed towards particularly aggressive solid tumors. Specifically, RMC-6236 targets a family of proteins called RAS. These are small GTPase proteins that regulate cell proliferation in response to growth stimuli. RAS proteins are commonly mutated in lung, colorectal, and pancreatic cancers. Mutations in RAS genes lock RAS proteins in their GTP-bound (ON) states, causing runaway signaling cascades that are drivers of tumor aggression.

Like many macrocycles, RMC-6236 was derived from a natural product (sanglifehrin A) which has high affinity to cyclophilin A (CYPA), a molecular chaperone. RMC-6236 acts effectively as a glue, but not as a degrader. Instead, this compound grabs CYPA and glues it to the exterior of RAS. Importantly, RMC-6236's landing zone is conserved across all RAS isoforms with equivalent potency, meaning that it hits the (ON) forms of KRAS, HRAS, and NRAS simultaneously. This could be crucial to overcome adaptive resistance whereby other RAS family members become upregulated in response to KRAS inhibition.

RAS proteins have proven difficult to drug, though several inhibitors do exist like sotorasib and adagrasib. One major challenge with KRAS inhibitors is durable efficacy. Unfortunately, KRAS often acquires additional mutations that can block inhibitor binding and cause therapies to lose effectiveness. RMC-6236 was designed to be robust against acquired mutations.

The formation of the ternary complex between RAS-RMC-6236-CYPA disables downstream signaling because it blocks the PPIs between RAS and other effector proteins. Because RMC-6236 lays down on a flat surface of RAS and not in a canonical binding site, acquired mutations are less likely to sterically occlude the binding between RMC-6236 and RAS, as shown below. This could increase the duration of response in patients.

Ultimately, the proof will be in the pudding. Though it's still early days, the clinical data produced by RMC-6236 has GI oncologists seemingly very excited. For example, let's look at some of the data in pancreatic adenocarcinoma (PDAC), a rather aggressive disease.

In a second line (2L), standard of care (SOC) setting, patients may only expect median 2-3 months of progression free survival (PFS) whereas RMC-6236 when given at low doses seems to show median 8 months of PFS in the 2L setting. Importantly, the curves for patients with more diverse RAS mutations was nearly identical, providing clinical evidence for the compound's robustness against resistance mutations.

The Phase 3 trial begins in the second half of 2024 with an expected PFS readout sometime in the first half of 2026. I'm eager to see the results and hope that its success catalyzes more research and development into therapeutic macrocycles and other PPI modulators writ large.

If you’re working in any of these spaces, send me a note!

Some proteins, like enzymes, have well-defined grooves on their surfaces known as binding pockets. Enzymes catalyze chemical reactions within these binding pockets. If a drug developer wants to inhibit this catalytic function, they can design a small molecule (ligand) that snaps into the binding pocket.

Not all proteins are as straightforward to drug. Transcription factors, for example, have smooth surfaces devoid of obvious binding pockets. Scaffolding proteins that form large protein-protein interactions (PPIs) typically lack so-called canonical binding sites.

Remember that proteins aren't static objects. They vibrate, twist, and contort through time, especially in response to the molecules around them. Canonical binding pockets persist throughout time and molecular contortions. You might wonder if there are non-canonical pockets, ones that exist rarely or that require neighboring molecules to tease out.

Rare, transient binding cavities are known as cryptic pockets.

The definition of a cryptic pocket is somewhat fluid. Generally, cryptic sites aren't easily detectable in a protein's unbound, ligand-free state. When ligands or nearby molecular shrapnel binds a protein, they can cause a conformational (structural) change that exposes cryptic binding sites. You can imagine it like a small key that opens a protein's secret compartment.

Why do drug developers care about targeting cryptic pockets?

More Treatable Diseases: The universe of proteins with canonical binding pockets is vastly smaller than that of proteins with cryptic pockets. Hitting cryptic sites broadens the landscape of druggable proteins, and thus, treatable diseases.

Less Toxic Drugs: Canonical binding sites are evolutionarily conserved. Related proteins have structurally similar binding pockets. This increases the likelihood that small molecules may erroneously bind to a similar, non-target protein. Cryptic sites are less conserved, making it easier to inhibit only the target protein and lessening off-target toxicity.

Overcoming Drug Resistance: Tumors have a nasty habit of mutating proteins in response to therapeutic pressure. Tumors sometimes alter proteins by changing or deleting amino acids that form canonical binding sites, effectively blocking inhibitors from continuing to work. By targeting cryptic pockets, inhibitors may be more likely to overcome common drug resistance mechanisms.

How do scientists discover cryptic pockets?

Targeting a protein's cryptic binding pockets has some obvious benefits, but the process of elucidating these hidden cavities is challenging. Molecular dynamics (MD) methods that model how proteins evolve through time are apt tools to discover transient, cryptic binding pockets.

The big issue is that cryptic pockets don't spontaneously appear. Generally, cryptic pockets are hydrophobic. They hate water and don't like being exposed to it. It requires large, often unfavorable, energetic swings in the form of large conformational changes to expose cryptic pockets.

Unbiased MD simulations are expensive and difficult to run for long time periods. Therefore, it's rare that these simulations will sample from energetically unfavorable regions of conformational space. It requires some clever trickery to bias MD simulations to selectively sample regions of the energy landscape that may contain cryptic binding sites.

There are a variety of ways to bias an MD simulation to explore valuable, rare events. One recent method is SWISH-X. Sampling Water Interfaces through Scaled Hamiltonians (SWISH) was the original method, which I'll describe first.

SWISH belongs to a family of MD methods that use a technique called replica exchange. Multiple copies (replicas) of a biochemical system are simulated in parallel, each with a slightly different Hamiltonian (energy function). In English, that means that the physical laws for each replicate are tweaked a bit. Periodically, the simulations are swapped, allowing the system to overcome larger energy barriers to explore novel regions of conformational space.

This can all be a bit fuzzy, so let's use an analogy.

Imagine there are many parallel versions (replicas) of you that exist in different timelines (simulations). You start at the bottom of the Grand Canyon and are tasked with finding a buried treasure (cryptic pocket) up on the desert surface. The steep, vertical facade of the canyon represents a large energy barrier.

If we had just one version of you, unaided at the bottom of the valley (unbiased MD), it's highly unlikely you'd be able to clamber up the rock face to find the treasure.

However, in our replica exchange version, we're free to alter the laws of physics in an unrealistic (alchemical) way. Let's say that we're varying the constant of gravity. For some replicas, gravity is reduced, allowing you to easily float up and out of the Grand Canyon and onto the surface. Suddenly, that version of you swaps (exchanges) timelines into a dimension with normal Earth gravity, freeing you to traverse the desert sands in search of treasure.

That's how replica exchange works in a nutshell.

Instead of gravity, SWISH alters the level of attraction between hydrophobic amino acids in a protein and the surrounding water molecules. By exaggerating this attraction, water essentially pries open cryptic pockets momentarily. When the replicas switch, neighboring organic molecules (often hydrophobic) can stabilize these cryptic pockets, keeping them open long enough to be more rigorously characterized. Treasure found!

If that's SWISH, then what is SWISH-X?

SWISH might work when the valley you need to climb out of is just the Grand Canyon. But what about if you need to float up the Marianas Trench, which is ~6x deeper?

The 'X' in SWISH-X stands for eXtended. With this method, the authors add in a second scaling factor: temperature. Specifically, they leverage a module called OPES MultiThermal which varies the temperature that each replica is run in alongside the water-attraction factor we discussed earlier.

Increasing temperature helps the replicas jump out of even deeper valleys. Hotter simulation conditions cause increased thermal fluctuations, meaning atoms vibrate more intensely, which can lead to larger-scale conformational changes and an increased likelihood of exposing a cryptic pocket. It would be like if we lowered gravity AND increased your buoyancy simultaneously to climb out of the murky depths of the Marianas.

How does SWISH-X perform?

It performs beautifully when measured up against other approaches. I'll walk through the violin plot below to explain what's going on.

Let's start with the vertical axis. It's divided into three regions depending on the level of cryptic pocket exposure. The purple means the cryptic pocket has been discovered. Light blue means it's been partially characterized. Dark teal means we couldn't find it. In other words, better discovery methods should skew data towards the top of the plot.

The horizontal axis tracks various methods, including the left-most method which is the benchmark. Briefly, holo- structures are when the protein is bound to a ligand (inhibitor) whereas apo- structures just include the protein by itself, as shown below. Remember earlier when I said that cryptic pockets tend to expose themselves when bound to a ligand? Well, that's what we're comparing to on the left: the perfectly open (essentially solved) structure that is 'holo-like'. The closer our data looks to that, the better.

Doing unbiased MD on the apo- structure is awful, like climbing out of the Grand Canyon under Jupiter-like gravity and without a rope. Using a mixed-solvent approach, a simulation with explicit neighboring water and organic molecules is better. These can bind the protein and induce partial cryptic pocket opening, like we discussed earlier.

SWISH is even better yet! Half of the distribution results in an open, characterized pocket. Finally, SWISH-X performs the best with virtually all of its distribution resulting in a solved cryptic pocket. SWISH-X marks the spot!

At Dimension, we are keenly focused on the interface of MD and machine learning (ML). If you're working at this interface or are scouring the Earth for buried cryptic pocket treasure, don't be a stranger!

If you'd like to hear more of what we have to say about the present-future of hybrid physics-ML methods in drug discovery, read our recent (open-access) journal article. Finally, if you want to read more about SWISH-X, check out the full publication!

Amino acids are the building blocks of proteins—one of the most common and important macromolecules on the planet. Nature has evolved a roughly 20-member library of amino acids that all organisms share. Peculiarly, scientists have catalogued over 500 rare amino acids in the wild. Think of these like a secret annex to the 20-member amino acid library or a collection of rare books. They exist. They’re important. They’re mysterious. Where do these additional amino acids come from? Why do they exist and what do they do?

I aim to tackle all of these questions and more with this essay, with a particular tilt towards the growing therapeutic utility of so-called unnatural amino acids. Folks already familiar with the basics of proteomics—amino acids through translation—can skip guilt-free through the first section. Enjoy!

Table of Contents

1. Basic Primer

   1. Protein Overview

   2. Translation Overview

2. uAAs and Their Applications

   1. Unnatural Amino Acid Overview

   2. uAA Applications

   3. Drug Conjugates

   4. Protein Medicinal Chemistry

3. Frontiers of Work

   1. Synthesis of uAAs

   2. Incorporation of uAAs Into Nascent Peptides

   3. Machine Learning Meets uAAs

4. Outro

Subscribe now

Basic Primer

Protein Overview

Proteins are Nature’s foot soldiers. These biomolecules can do almost anything. They catalyze biochemical reactions, facilitate DNA replication, help cells communicate with each other, transport cargo around the body, and even help us see. Antibodies are proteins. Many hormones are proteins. More and more medicines are proteins. But what makes a protein a protein?

Proteins are built up from a roughly 20-letter alphabet of building blocks called amino acids. The number, type, and order of amino acids give a protein its identify, structure, and physiochemical properties. Smaller segments of a protein are called peptides or polypeptides and they too are composed of amino acids.

Similar to linguistic alphabets, each amino acid is unique—it produces its own ‘sound’. As highlighted below, all amino acids have some parts in common. They share the same backbone structure inclusive of a carboxyl group (—COOH) and an amino group (—NH2) oriented around a central (alpha) carbon atom. Where they differ is in their R-groups, which is the lazy chemist’s way of writing ‘rest of the molecule’.

R-groups dictate the identity and physiochemical properties of an amino acid. Some R-groups are small and uncharged while others are large and charged. Some R-groups are linear while others contain cyclic structures, as shown in the table below. The important takeaway is that each R-group is unique and Nature has evolved roughly 20 that are used commonly throughout life on Earth. For anyone interested in the question of, “Why these particular 20 amino acids?”—I’d recommend reading Frozen, But No Accident.

Polypeptides are linear as they’re first formed inside cells. The carboxyl group of the first acid forms a peptide bond with the amino group of the second acid whose carboxyl group forms another peptide bond with the carboxyl group of the third acid, and so on. This means that adjacent R-groups interact with one another in the first moments of the polypeptide’s lifespan, causing the flexible backbone to contort.

Adjacent, similarly charged R-groups might kink the backbone apart while adjacent, oppositely charged R-groups might bend the backbone in on itself. Before long, amino acids that were once distant in linear, sequence space are now close to one another in 3D space. These new dynamics cause the protein to bend into a stable shape (conformation) where it lives a good portion of its life, as shown in the diagram below. If you want to know more about protein folding and specifically how machine learning (ML) models like AlphaFold3 (AF3) do this computationally—check out my recent Field Guide to AF3.

A protein’s shape dictates its overall biomolecular properties. Once shuttled to its proper location in the body and surrounded by other molecules, these properties dictate the protein’s function. In that way, the role of an entire macromolecule ascends from the simple ordering of a 20-letter amino acid alphabet. But how exactly are polypeptides formed? What is the machinery that transforms a flotilla of amino acids into peptides?

Translation Overview

Amino acids are assembled into nascent proteins through a process called translation that occurs in a macromolecular machine called a ribosome. As diagrammed below, a messenger RNA (mRNA) molecule carrying the genetic blueprint for a peptide leaves the nucleus and finds its way to the small ribosomal subunit. Recall that mRNA is built up from a four-letter nucleotide alphabet containing A’s, C’s, G’s, and U’s. A long time ago, my mnemonic for translation centered on the conversion between two alphabets, the four-letter nucleotide alphabet and the 20-letter amino acid alphabet.

mRNA nucleotides are grouped into triplets called codons. Codons are the minimum chunks of information that map to amino acids. Importantly, this mapping isn’t 1:1 because there are 64 unique, three-letter permutations of four nucleotides. The rules governing how codon triplets are translated into amino acids follows the below codon chart. Readers may notice that only 61 codons code for amino acids while three others serve as STOP codons—signals for the translation process to end. This also implies the question of what codon STARTS the translation process.

While heuristics and charts are helpful, they’re also abstract. What I find beautiful are the physical underpinnings of these short-hand rules we use to simply the chaotic process of protein translation. By examining some of the other actors in the story, like transfer RNAs (tRNAs) and amicoacyl-tRNA synthetases (aaRSs), we gain a deeper biochemical appreciation for where our mnemonic devices come from. This level of rigor is also necessary to properly explain the main topic of this piece—unnatural amino acids.

Translation is divided into three phases: initiation, elongation, and termination. Once an mRNA molecule binds the small ribosomal subunit, a specialized molecule called tRNA begins to scan the mRNA for the codon AUG. tRNA molecules are like little tugboats. They each carry a single amino acid from the cellular milieu to the physical site of translation.

In this context, scanning means that the tRNA is running its bottom portion—called an anticodon—along the mRNA until it finds a START codon (AUG). Once bound and anchored, the large ribosomal subunit attaches, forming a full ribosomal complex and ending the initiation phase. Ribosomes have three 'grooves’ that tRNAs fit into during the elongation phase: E (exit), P (peptidyl), and A (aminoacyl), as shown below.

Thus begins the cyclic elongation phase. Here, the next codon following AUG is recognized by its cognate tRNA. Enzymes facilitate a peptide bond between the carboxyl group of one amino acid with the amino group of its adjacent amino acid. The nascent peptide extends at the A site and the complex ratchets rightward along the mRNA’s length. The amino acid-less tRNA is jettisoned at the E site and the process begins anew.

Termination happens when one of three STOP codons summons a release factor, which breaks apart the entire complex and frees the polypeptide to begin folding into whatever shape its amino acids dictate, as shown below. To give a sense of scale, protein translation occurs at a rate of 10 amino acids per second, meaning it would take about a minute to synthesize a hemoglobin protein.

uAAs and Their Applications

Unnatural Amino Acid Overview

Across all kingdoms of life on Earth, there are very few exceptions to protein translation occurring outside the codon rules we’ve established. Genetic recoding, the idea that amino acids can be added through non-normal codons, does seldom occur context-specifically in some fungal and bacterial species. There is a vast and strange world, however, of unnatural amino acids. This extended alphabet doesn’t leverage genetic recoding—in fact, it doesn’t rely on genetic encoding at all.

Unnatural amino acids (uAAs) harbor similar backbones to their natural counterparts. uAAs have both carboxyl and amino terminal groups. Where they differ is their R-groups. uAAs feature a novel set of R-groups, some that diverge slightly from natural R-groups and others that look and act quite alien.

uAAs aren’t directly coded for using the codon table. Instead they exist as precursors, analogs, or metabolic intermediates of natural amino acids. Nature has also evolved a separate set of enzyme complexes capable of incorporating uAAs within nascent peptides. For example, polyketide synthase (PKS) and non-ribosomal peptide synthetase (NRPS) can create proteins harboring uAAs, though these reactions are very bespoke.

Despite the name, uAAs are found throughout plants, animals, and microorganisms. uAA-containing peptides are functionally diverse. Plant uAAs are often produced in response to threats. Some species of grass release L-m-tyrosine from their roots which acts as a toxin. Bacterial and fungal uAAs are commonly known for their antibiotic and anti-inflammatory properties. While rarer, animal uAAs are generally metabolic intermediates of canonical amino acids and are used clinically to reverse liver damage and as diagnostic biomarkers. Altogether, researchers have discovered >500 uAAs.

uAA Applications

Since the discovery of uAAs, scientists have sought to engineer them into peptides. After all, if Nature has found uses for them, why shouldn’t we? The first in vitro, site-specific incorporation of a uAA into a peptide was reported in 1983. In 1997, the first genomic encoding of a uAA was executed in vivo. Exactly how these feats were accomplished is something we’ll discuss later.

The applications of uAA-containing peptides are as diverse as uAAs themselves. From a basic research perspective, scientists can introduce fluorescent/photo-crosslinkable probes into peptides at specific sites using uAAs. This allows for tracking of protein dynamics and protein-protein or protein-ligand interactions. Scientists can also mimic post-translational modifications (PTMs) with uAAs, facilitating our understanding of endogenous PTMs and their functions. Photo-caged uAAs can allow for light-activated control of protein function, enabling spatiotemporal studies of protein activity in cellular contexts. Despite all these, therapeutic protein engineering is the fastest growing application area for uAAs.

Drug Conjugates

Conjugate therapies are entering their golden era. Conjugates are a class of drug where a therapeutic payload is engrafted onto a targeting agent. Ostensibly, the targeting agent enables greater site-specific delivery of the payload compared to if that payload was delivered by itself into the bloodstream. Antibody drug conjugates (ADCs) are perhaps the most well known example of a conjugate therapy. Radioligand therapies (RLTs) are another. I’ve written about both in previous blogs, which you’re free to read here and here, respectively.

Most targeting agents are peptide-based. Therefore, drug developers must use a technique called bioconjugation to attach the therapeutic payload to an amino acid. Chemists generally exploit solvent-exposed lysine or cysteine amino acid residues for bioconjugation.

Lysine contains an amine group while cysteine contains a thiol group—both of which are highly nucleophilic, meaning they’re reactive under mild conditions. This makes lysine and cysteine ideal residues for bioconjugation chemistries. This technique has given rise to the first wave of FDA-approved conjugate drugs.

Unfortunately, drug developers can’t always choose exactly where or how many lysine or cysteine residues are present on the targeting agent. If too many or too few payloads glom onto the protein or if they attach to different residues, it results in a heterogenous drug product pool. This requires expensive downstream purification, lowering product yield and resulting in higher prices.

Drug designers can encode uAAs into targeting agents to overcome the issue with heterogenous product manufacture. By leveraging the diversity of R-group chemistries and the fact that uAAs don’t naturally appear in antibodies or small peptide binders, it allows chemists to selectively attach payloads precisely where they wish and drive those reactions to completion. Furthermore, this enables fine-tuning over these drugs’ pharmacokinetic (PK) profiles and increases the likelihood of homogenous product pools.

Site-specific incorporation of payload or other prosthetic groups is just one therapeutic application of uAAs. Perhaps an even more impactful area, and one that still resembles the Wild West, is protein medicinal chemistry.

Protein Medicinal Chemistry

Lead optimization is a staple of both small molecule and antibody drug development. During this development phase, scientists make minute refinements to small molecule or antibody scaffolds to improve their physiochemical properties. This process can be likened to multiparameter optimization as drug developers juggle changes that could improve solubility, but worsen potency, for example. Lead optimization for peptides is much less mature by comparison.

Protein biochemists may begin lead optimization by replacing each amino acid in a peptide with alanine—a simple amino acid. This is known as alanine scanning and it helps researchers understand the functional impact of each amino acid in the sequence, giving insight as to where and how further modification might affect bioactivity.

Drug developers first modify peptide backbones. For example, biochemists can add structural rigidity and protect against enzymatic degradation all by bending the peptide’s backbone back onto itself—a process called cyclization. Developers can make large changes to binding surfaces by installing defined secondary structures like alpha-helices or beta-pleated-sheets. Unsurprisingly, uAAs have a major role in protein medicinal chemistry just as well.

The world of peptide uAA modifications is extraordinarily vast. Scientists have looked towards natural uAA-containing proteins like cyclosporin A, tryglysin A, and darobactin A, to understand uAA-specific structure-activity relationships. Indeed, several classes of uAA seem to confer increased resistance to proteolysis, increased potency, cell permeability, and receptor selectivity, amongst others.

These lessons have been leveraged prospectively by drug developers. Two examples include Chugai’s RAS inhibitor (LUNA18) and Merck’s PCSK9 inhibitor (MK-0616). These macrocyclic compounds contain four unique uAA modification classes to enable greater cell permeability and oral bioavailability, respectively.

Peptide therapeutics, a historically maligned category of drug, is having a renaissance. Accounting for 5% of global pharmaceutical sales, peptides are a $70B market. Just 34 peptide drugs were approved from 1960-1999 while 52 have been approved this century, a rate tripling from 0.9 approvals per year to 2.7 approvals per year. Biologics, including antibodies, are a monster category—accounting for 20% of the global pharmaceutical industry. The combination of quickly growing protein and peptide-based modalities and the increasing utilization of uAAs in biologics suggests that uAAs will play a critical role in the present-future of the pharmaceutical industry.

Frontiers of Work

This final section deals with areas of active research furthering what’s possible with uAAs and uAA-containing peptides. These topics range from chemistry to biology to machine learning and everything in between. One main takeaway is the non-virtuous cycle between data generation and machine learning in the uAA-adjacent sciences.

uAA-containing peptides are hard to make, which means training data is scarce, which means models are less performative, which means industrial applications are less tractable, which means there’s less demand pull to make uAA-containing peptides easier to make, and so on and so forth.

Synthesis of uAAs

uAAs need to be synthesized before they can be installed into proteins. Scientists have a variety of methods to create uAAs—each with their own advantages and disadvantages. Oftentimes, the challenges with uAA synthesis drives researchers to purchases uAA from reagent suppliers rather than embarking on the synthesis journey themselves. Nevertheless, both suppliers and academics alike are pushing the frontier of how we make uAAs.

Broadly speaking, one can choose between purely chemical synthesis, chemoenzymatic synthesis, and cell-based, metabolic synthesis.

Purely chemical synthesis involves traditional organic chemistry mechanisms spread across multiple reaction steps and aided by chemical catalysts. This method is generally scalable and more deterministic than biosynthesis. Diverse uAA structures are accessible with chemical synthesis. However, this method involves harsh reaction conditions and has the potential to be inefficient and costly. Given the precise 3D geometries of uAAs, chemists use temporary protective groups and chiral auxiliaries to control against incorrect stereochemical intermediates. These chemical groups add to cost and decrease product yield as the number of reaction steps grows exponentially.

Chemoenzymatic synthesis combines chemical and biological reagents to achieve uAA biosynthesis in a cell-free context. Compared to purely chemical synthesis, this method isn’t as wasteful, involves milder reaction conditions, and can be more stereoselective given the specificity of enzymes. This eliminates the need for transient protective groups. However, chemoenzymatic synthesis can be limited in what uAAs can be created—a downside of the specificities of most enzymes. Since enzymes are involved, process scalability becomes less straightforward.

Metabolic synthesis involves genetically engineering cells to produce uAAs. We know very little of natural uAA biosynthetic pathways, precluding our ability to easily mimic them. Metabolic synthesis can access many uAA varieties and is cost-effective at scale. Similar to other biosynthetic methods, metabolic approaches require process development and scape-up, a challenging and capital intensive undertaking. As we’ll discuss soon, metabolic synthesis offers the tantalizing possibility to synthesize and incorporate uAAs into peptides all within the context of a single cell.

Incorporation of uAAs Into Nascent Peptides

Once synthesized, protein chemists must install uAAs into peptides. Similar to the upstream synthetic process, scientists have a variety of methods at their disposal to incorporate uAAs—once again, each with their own pros and cons. These include solid-phase protein synthesis (SPPS) and recombinantly using genetic code expansion (GCE).

SPPS isn’t dissimilar from oligonucleotide synthesis. Here, standard or unnatural amino acids are flowed onto a reaction substrate one at a time. Once the reaction has terminated and the peptide has grown by one unit, the process is repeated with the next amino acid until the peptide is complete. SPPS is ubiquitous, but severely limited with peptides beyond 50-70 amino acids in length. Each step has an efficiency penalty, which grows exponentially with peptide length—making long peptides exorbitantly expensive or impossible to make. Moreover, high concentrations of amino acids are required to drive synthesis, which adds to total cost particularly when many uAAs are involved.

GCE is an elegant and involved process. It requires engineering a cell’s translation machinery to install a uAA at a specific site in a peptide, generally by suppressing STOP codons and reassigning them to a custom uAA-specific tRNA molecule. This ensemble of cellular mechanisms is called an orthogonal translation system (OTS) and it’s one of the most fascinating areas of uAA scientific advancement.

The main complexity arises from the fact that a cell’s endogenous translation machinery and the OTS must work simultaneously without interfering with each other. Another issue is that engineered translation systems are seldom as efficient as wild-type (WT) systems.

There’s an appreciable charge to make OTSs more catalytically efficient and versatile towards a wider range of uAA types. One body of work I find interesting is the utilization of ‘cold-evolved’ translation machinery. The hypothesis is that enzymes evolved in cold climates (psychrophilic) have high efficiencies in moderate or ambient temperatures. One research group recently tested this theory, finding that psychrophilic enzymes can be uAA-specific while also exhibiting abnormally high incorporation efficiency, as shown below.

That this system can install uAAs at such low concentrations is a boon for single-cell uAA synthesis and incorporation regimes. Metabolic engineering of cells to produce uAAs is possible, but the cytosolic concentrations tend to be quite low, making incorporation with standard OTSs challenging. This could help remedy the issue.

Machine Learning Meets uAAs

Machine learning (ML) methods have supercharged protein engineering. We’ve written a review showcasing the cutting edge of computational protein engineering in the context of therapies and industrial enzymes. Proteins containing uAAs, however, pose problems for ML.

ML algorithms are only as good as the data they’re trained on. Peptides containing uAAs are underrepresented in Nature as well as experimental datasets for ML training. Therefore, models trained on standard peptides may perform poorly on uAA-containing peptides.

One unique challenge for ML on uAA-containing peptides is representation, which refers to how a model sees its training data. Peptides are mesoscale. They’re in no-man’s land. They’re not as big as large proteins (e.g., antibodies) nor as small as small molecule drugs. We typically represent large proteins using amino acids and a data format called FASTA. We represent small molecules with atoms in a format called SMILES. What should we use for uAA-containing peptides?

FASTA isn’t built to handle the expanded uAA alphabet. SMILES don’t scale well for the number of atoms in peptides. A flexible format for biomolecules called HELM has gained some traction for uAA-containing peptides, but it’s grossly unpopular compared to SMILES and FASTA. There’s no clean answer to this problem yet. Representations for uAA-containing peptides are still a mysterious frontier.

One research group concocted a multi-scale, fragment approach to uAA-containing peptide representation. This methodology uses large fragment blocks for the peptide backbone that fluidly branch off into atomic detail for side chains. Both the fragments and atoms get embedded in a heterogenous graph architecture, like the one above, for downstream prediction tasks. Amongst other uAA-specific ML accommodations, this group’s method showed improved predictive performance relative to other models across a series of tasks like binding affinity prediction and cell permeability prediction, as highlighted in the plots below.

Peptide viability is a key prerequisite for higher-level tasks like property prediction or de novo design. Which sites in an arbitrary protein can tolerate uAA substitution? How will uAAs affect protein yield? Without answers to these, property prediction and design just aren’t practical. Several research groups have taken aim at this root level task, though the field is still nascent.

In 2021, one research group combined molecular dynamics (MD) simulations with ML to generate proof-of-concept learnability for how one specific uAA (acridonylalanine, Acd) might affect a protein’s viability.

The group simulated uAA substitution into 51 unique sites across two proteins using MD. They used these MD simulations to train an ML model in lieu of physical, experimental data. The model’s goal was to predict how uAA substitution might be tolerated by these proteins—specifically, how uAAs might affect soluble protein yield. The researchers found that Acd imparted local physiochemical and steric effects that were highly predictive of site tolerability. While the model was 80-90% accurate, these results were obtained on just 10 held-out datapoints. This effort certainly isn’t generalizable to the grand problem of uAA x ML, but stands as a valiant proof-of-concept study.

Building on this effort, a second research grouped aspired to develop an ML algorithm capable of predicting site-tolerability of any uAA at any protein site. They curated a literature database of ~1,000 successful and ~200 failed uAA incorporations, each with a matching structure in the Protein Data Bank (PDB). The model considered features like changes in evolutionary entropy, steric effects, and physiochemical alterations. By examining feature importance, the second group replicated the previous hypothesis that local, uAA-induced physiochemical changes are more predictive of site-tolerability.

The researchers observed a decent predictive correlation to the literature (Pearson = 0.64) using the strictest training data cutoff. This is impressive given the diversity of uAAs and proteins. However, similar to the first proof-of-concept study, I suspect this model may not generalize well prospectively given (a) the lack of backbone structural diversity in the training set, (b) the fact that only one class of uAAs was used, and (c) that the training set is small on an absolute basis. Nevertheless, this work represents another critical step towards viable ML models applied to protein engineering with uAAs.

Outro

Unnatural amino acids are ancient. They’ve persisted for eons despite scant natural machinery to support their more frequent installation inside proteins. They’re functionally diverse and useful to the organisms that express them. Despite all of this, we know very little about them.

Biologics and peptide-based therapeutics are exploding in importance therapeutically and I see no reason this trend should abate. The biologics that feature uAAs seem to exhibit startlingly useful properties such as proteolytic stability. For this reason alone, my sense is more drugs will incorporate them in the future.

Whether it’s a physics-based breakthrough that enables rational uAA lead optimization despite data sparsity or a novel synthesis method that allows researchers to throttle up uAA design throughput, the design-build-test-learn (DMTA) loop of uAAs isn’t poised to stay anemic forever.

Once this flywheel starts to kick off and we begin to unravel the rules of uAAs, we will begin to wield them with more authority. This is tremendously exciting. We are tracking developments on all fronts. If you or a colleague are working on uAAs in any capacity, feel free to drop me a line on Twitter / X.

Radioligand therapy (RLT) is one of the most exciting spaces in modern pharma. Anchored in a century of knowledge about radiation and cancer, RLT is a practice-changing therapeutic paradigm. The excitement for RLT flows from its potential to expand into common cancer indications and to earlier lines of treatment. Though the hype seems merited, RLT faces myriad challenges to becoming a standard of care (SOC).

Subscribe now

An Abbreviated History

German physicist W. C. Roentgen discovered X-rays in 1895. Three days later, a vacuum tube manufacturer used X-rays to treat a breast cancer patient. By the early 1900’s, physicians understood that radiotherapy could reduce pain for patients with advanced tumors.

During the 20th century, external beam radiation therapy (EBRT) became an oncological mainstay. Researchers gained a deeper knowledge of radiation’s mechanism of action (MOA). High-energy particles would bombard a cell’s nucleus, lethally shredding its DNA. Quickly dividing cells (e.g., cancer cells) were especially susceptible.

EBRT machines evolved to be more precise and use higher energy radiation. Oncologists developed radiation dosimetry—formulae that helped physicians dial in the exact right amount of radiation. As with other therapies, EBRT had the potential to be toxic. The radiation beam had to travel through normal tissue to reach the tumor after all.

Personalized dosimetry got a boost in the 1980’s when scientists developed peptide receptor scintigraphy (PRS). PRS leveraged the field’s burgeoning knowledge on cell surface proteins. The method involved labeling an antibody with a low-energy, radioactive isotope. Aimed at proteins over-expressed on tumors, the labeled-antibody would accumulate on the surfaces of malignant cells. The label would lit up tumor masses and any metastases, informing dynamic dose calculations.

By the early 1990’s, oncologists extended this concept to treatment. Peptide receptor radionuclide therapy (PRRT) exchanged the low-energy radionuclide for a high-energy one—one powerful enough to kill surrounding cells. Conceptually, PRRT was thought to avoid the toxicities of EBRT by ensuring precision in radiation delivery. PRRT would serve as the genesis of contemporary RLT.

Radioligands 101

“If you can see it, you can treat it.” —Anonymous

Contemporary radioligand therapy is built on the concept of theranostics—the combination of therapeutics and diagnostics to treat disease. RLTs are modular, multipartite medicines. They’re composed of four parts—a targeting agent (e.g., a peptide analogue), a chemical linker, a chelator, and the radioactive payload. By exchanging the high-energy payload for a low-energy one, an RLT can be converted from a therapeutic compound to a diagnostic one to aid in imaging.

Each component of an RLT influences the efficacy and safety of the drug. Pharmaceutical companies must carefully mix, match, and re-engineer each element to fit a certain target product profile (TPP) for whatever indication the group is aiming at. There are excellent reviews on this topic that interested readers should check out. For the sake of this piece, I’ve summarized the essentials:

1. Targeting Agent (Warhead)

   1. Warheads bind to cell surface target proteins. They are the bulk of the mass of any RLT and contribute greatly to its physiochemical properties. Unlike antibody drug conjugates (ADCs), most RLT warheads are smaller vectors, such as small molecules, peptides, or antibody fragments. This is because smaller warheads enable deeper tumor tissue penetration, allowing radiation to affect the interior regions of tumors.

   2. Ideal warheads have high affinity for their targets, increasing the RLT’s residence time and the dose of radiation exerted on a tumor. Warheads should be selective for their targets, minimizing radiation to healthy tissues. Warheads should not instigate a strong immune response and they should be readily manufacturable at scale.

2. Linkers

   1. Linkers are chemical agents that bind the warhead to the payload group. They should be stable enough to not degrade in plasma, which could cause systemic toxicities. The size, flexibility, and charge distribution of linkers are all relevant design characteristics.

3. Chelator

   1. Chelators are molecules that form multiple bonds with a central metal atom or ion, acting like a sequestering agent or cage. For RLTs, chelators trap the radionuclide and hold it in place.

   2. Ideal chelators are kinetically inert and thermodynamically stable. Macrocyclic chelators are common as they form a stable ring around a radionuclide. however, the high number of coordinate bonds makes for harder chelator synthesis. Acyclic chelators are easier to produce, but less stable.

4. Radionuclide

   1. The choice of radionuclide is critical. As shown in the table below, drug designers must choose between an alpha or beta emitter. The former releases large amounts of energy over a short range while the latter releases smaller radiation salvos over long distances. Despite the field’s increasing interest in alpha emitters, my sense is that there’s plenty of room for both emitter types.

   2. The second decision involves the specific isotope, which depends on factors to do with half-life, emission dynamics, decay intensity, decay chain, handling considerations, and other logistics.

      

Why is Everyone Excited About RLT?

RLTs are caught in the zeitgeist for two reasons—their intrinsic attributes and the strength of incremental clinical data.

Radioligands’ theranostic capability opens up unique doors. Specifically, physicians can verify in vivo biodistribution in a manner not possible with other forms of targeted therapy. This aids personalized dosimetry. RLTs have an essentially guaranteed MOA. As mentioned earlier, these drugs lean on a century of medical knowledge and know-how. Radiation simply works. This fact makes RLTs more attractive than ADCs in some ways since there’s no compounded risk that a novel payload isn’t efficacious. Beyond these intrinsic advantages, the clinical data speaks for itself.

The beachhead indication for RLT is mid-gut neuroendocrine tumors (NETs). Established and emerging RLTs for NETs often target the somatostatin receptor SSTR2. In January 2017, one of the largest Phase III RLT studies (NETTER-1) showed that the drug 177-Lu-DOTA extended both quality and quantity of life for patients with advanced NETs. This trial served as the basis for the 2018 FDA approval of Novartis’s Lutathera for patients with SSTR2 (+) NETs.

Six months ago, the follow-up NETTER-2 study delivered its first tranche of results. The trial enrolled 226 patients to receive Lutathera or the current standard-of-care (SOC) (octreotide) in the first line (1L) setting for NETs. Compared to the SOC, Lutathera improved median progression free survival (PFS) from 8.5 months to 22.5 months. Importantly, the overall response rate (ORR) expanded nearly five-fold from 9% to 43%. Fewer than five serious adverse events (SAEs) were observed. The expansion from a later line to a 1L treatment speaks to the value and market growing potential of RLTs.

Metastatic, castration-resistant prostate cancer (mCRPC) is the second largest RLT application today. Also approved by the FDA, Novartis’s Pluvicto is a similar molecule to Lutathera, but aimed at PSMA—a target highly expressed in prostate cancer. Based on results form the VISION and TheraP trials, Pluvicto is indicated for use in later-line subgroups of mCRPC patients. In VISION, Pluvicto increased the median PFS from 4.3 to 8.7 months and increased overall survival materially from 11 to 14 months. TheraP confirmed these results.

Subsequent studies have also suggested the Pluvicto is useful in tumor re-challenge scenarios where more than the standard dose regimen is administered after a dosing holiday, suggesting that these can be more durable therapies. One patient has even undergone >20 cycles over 5 years with no DLTs and evidence of response after each re-challenge.

Okay, What’s the Catch?

One of the questions we kick around a lot at Dimension is, “Is the juice worth the squeeze?” Are the challenges with making and scaling RLTs worth the value they bring to patients and investors? We believe so.

The oft-referenced obstacle with RLTs is logistics. Some go so far as to say that RLT logistics are as complex as personalized cell therapies. While this piece isn’t about RLT manufacturing and logistics, these are critical points to bring up.

Manufacturing timelines are ultra-compressed because therapeutic radionuclides decay on the order of days. These isotopes are generally produced in highly-specialized, central facilities with particle accelerators. Raw elements are bombarded with high-energy particles, allowed to cool, and then swiftly packaged in lead containers and sent to vendors for assembly. Nuclides are added to the chelator and then labeled to the vector using a variety of bio-conjugation techniques.

Once assembled and QC’d, the drugs must get to patients usually within 14 days. Anyone who handles or administers RLTs must be an authorized user (AU) as stated by the National Regulatory Commission (NRC). After dosing , radioactive waste must be kept at the site in a specialized storage container for a minimum of 120 days or until the material has no detectable radiation.

Despite all these logistical hurdles, I believe the biggest obstacle for RLTs generally is kidney (renal) toxicity. If this isn’t solved, ameliorating all the logistical challenges becomes moot. Expanding the effectiveness of RLTs requires expanding their therapeutic indexes (TIs).

Nephrotoxicity

Drugs are one of the most common sources of acute kidney damage, causing roughly 20% of renal failure cases.

Drug-induced nephrotoxicity is especially common in older adults. Unfortunately, this demographic also has the highest cancer incidence rate. This poses significant challenges for cancer therapy, which may need to be stopped if renal damage accumulates.

Drugs exert their toxic effects on renal tissue through one or more common pathogenic mechanisms. The first is altering intraglomerular hemodynamics. In healthy young adults, the kidney’s filtration machinery (the glomerulus) processes roughly 120 mL of plasma per minute. Some drugs interrupt the kidney’s ability to regulate glomerular pressure, disrupting the flow of fluid through the kidney.

The second is tubule cell toxicity. Renal tubular cells are especially susceptible to damage since they concentrate and reabsorb filtrate, exposing them to high levels of toxin. A third mechanism is inflammation, which can lead to fibrosis and scaring throughout multiple tissues in the kidney. The list goes on, but these three modes of renal injury are most common, especially for RLTs.

How do RLTs Cause Renal Insult?

RLTs have a renal-predominant mode of excretion. These compounds’ pharmacokinetic (PK) properties cause kidney retention. Taken together, these facts explain why renal dose-limiting toxicities (DLTs) may narrow the therapeutic indexes (TIs) of RLTs. Oncologists may not be able to administer clinically meaningful RLT doses before signs of renal injury manifest.

Using the aforementioned nephrotoxicity framework, RLTs primarily cause tubule cell toxicity. Recall that most RLTs use small binders to increase tumor tissue penetration. Compounds below the glomular filtration barrier (60 kDa) are processed in the main, functional units of they kidney called nephrons. These are dense networks of tubes that allow biomolecular interchange between blood and urine, allowing the body to reabsorb things it wants while excreting things it doesn’t.

As shown above, renal tubule cells have evolved a network of surface receptors to detect and facilitate the reabsorption of critical-to-life biomolecules. Megalin and cubilin are two such receptors that are relevant to RLT toxicity. Because most RLT targeting vectors are based on natural human biomolecules, they’re highly susceptible to uptake by megalin and cubilin.

Once internalized, RLTs are metabolized inside lysosomes. While peptide fragments may find their way back into the bloodstream, nuclides generally dwell inside tubule cells—allowing toxic radiation to accumulate, causing collateral DSBs and oxidative stress.

Other RLT Parameters Affecting Nephrotoxicity

Beyond molecular weight (MW), the primary determinants of renal toxicity include RLT charge and charge distribution, nuclide residualizing activity, chelator stability, and target selection.

The surfaces of proximal tubule cells are generally negatively charged. By adding more negatively charged groups to the linker or warhead, it creates a less favorable surface interaction for renal uptake. However, the charge distribution matters too. Positively charged local regions, like chelator-nuclide groups, can interact with megalin/cubilin and undergo active transport into tubule cells.

Certain radionuclides cause greater kidney damage than others. For example, radiohalogens cause less nephrotoxicity than radiometals. Radiohalogens get metabolized into smaller metabolites and are more easily transported back into blood whereas radiometals can form larger byproducts, causing them to accumulate in tubule cells. Unfortunately, almost all therapeutic nuclides are metals. This is known as residualizing activity.

Beyond the nuclide, the chelation shell’s stability affects toxicity since free radioisotopes can be more damaging than their sequestered counterparts. Should enzymes within the lysosome break the chelation cage, the nuclide would be free to cause more toxicity. This can be doubly challenging for alpha emitters. Alpha emitters frequently recoil like a gun out of their chelation cages, freeing daughter nuclei to continue their decay chain haphazardly.

Target selection plays a pivotal role in kidney toxicity. For example, both SSTR2 and PSMA are expressed in low levels in renal cells. When RLTs bind their cognate targets expressed in the kidney, it constitutes on-target, off-site toxicity.

Renoprotection

Battle-hardened in the clinic, there are several first-generation workarounds to protecting renal cells from RLT-induced injury.

Perhaps the most pressing challenge to solve in the RLT space is that of nephrotoxicity. For decades, clinicians and drug developers have leveraged first-generation renoprotective strategies to mitigate renal injury and expand the TI of RLT drugs. This section summarizes these methods and hints at a more elegant future for renoprotection.

A common technique since the early days of RLT has been to infuse patients with a cocktail of positively charged amino acids (e.g., lysine and arginine). These acids compete with RLTs to bind megalin and cubilin. By occupying receptors with an inert substance, fewer copies of megalin and cubilin are free to bind and internalize potentially damaging radiation. This technique is called competitive inhibition.

As shown below, pre-targeting is another mechanism for sparing renal tissue. This method involves infusing an unlabeled RLT vector which binds to both the tumor and renal cells. A second injection containing the therapeutic nuclide is given. The nuclide binds to the unlabeled vector in vivo, enabling delivery of radiation to target tissue.

Drug designers have begun installing an albumin-binding moiety onto therapeutic RLTs. Albumin is a long-lived, heavy serum protein. By hitching a ride on the exterior of albumin, RLT residence time in plasma increases. Importantly, the complex is heavy enough to avoid becoming part of the glomular filtrate, lessening the likelihood of renal cell accumulation.

Additionally, designers have leveraged conditionally-cleavable linkers. These species are meant to break apart in the presence of enzymes located on the exteriors of proximal renal tubule cells. Conceptually, by cleaving here, radioactive nuclides are disallowed from accumulating inside kidney cells.

Perhaps one of the most valid, but challenging, techniques for avoiding nephrotoxicity is tuning the physiochemical properties of the RLT itself. As mentioned previously, weight, charge, charge distribution, and more all leave their mark on toxicity.

De-Targeting and Other Emerging RLT Trends

It’s the end of the beginning for RLTs. Radiation works. RLTs work. There are still some kinks in the system to work out, but the future looks bright.

Rationally engineering renoprotective properties into RLTs is one of the most critical research thrusts in this space, but it’s not the only one. As the number of RLTs in the clinic grows, it’s clear the field is increasingly adopting alpha radiation, broadening its vector and target choices, and rethinking how to approach logistics challenges. This section summarizes some of these advances.

Engineering De-Targeting

The entire RLT field would benefit from the discovery of general rules for sparing renal tubule cells from injury. First-generation renoprotective strategies have proven somewhat successful, but are crude. One path forward might be to treat RLT renal insult as a predictive toxicology problem.

Machine learning (ML) excels at domains currently dominated by human heuristics. This is only the cause with sufficiently large and diverse datasets, however. That’s a place where RLT struggles. Though the field has been around for decades, I’m skeptical that existing data repositories are sufficient for ML model training. Here are the main issues in my mind:

1. The majority of clinical data is concentrated on similar target-compound pairs, including SSTR2 and PSMA. This could affect a model’s ability to generalize to other modalities (e.g., antibody fragments) or anywhere far outside this distribution.

2. Though imaging data shows in vivo biodistribution, a unique feature of theranostics, the resolution may not be fine enough to assess whether injury to renal tubule cells is occurring.

3. There are simply not that many unique agents that’ve been tested in the clinical domain, certainly not enough for ML.

This suggests an enormous amount of highly curated data needs to be constructed first. I’m still on the fence about this, but I’m intrigued by the idea of high-throughput in vitro imaging techniques coupled to in vivo biodistribution data in an active learning format. There are holes here too, but it’s something I’m thinking about.

Evolving Nuclides

Highlighted below is a breakdown of the split between alpha and beta emitters featured in RLTs across various phases of development. Newer drugs are clearly leaning towards including alpha emitters. Alpha emitters are much more likely to deliver cell-killing levels of radiation owing to their relatively higher emission strengths. In theory, this should enable efficacy in scenarios where only a few copies of the drug are delivered to a cell. I do think beta emitters have a significant role still in sufficiently advanced, heterogeneous tumors. With their longer ranges, beta emitters can kill cells that don’t express the target antigen—a common scenario in tumors with multiple, dominant clones.

Manufacturing and Logistics

It’s no surprise that virtually all of the large acquisitions in the RLT space were of companies with verticalized manufacturing. Even so, there’s still some innovation in the space worth mentioning. Companies like ARTBIO and AdvanCell have switched to lead (Pb) isotopes. Unlike other emitters, lead can be manufactured in a bench top format, ostensibly allowing these companies to decouple from centralized production facilities. Meanwhile, groups like Nusano are leveraging novel technology to shore up our isotope supply. Nusano operates a linear particle beam technology they claim can simultaneously generate multiple therapeutically relevant isotopes.