VI PhDay-EIO
CSIC Misión Biológica de Galicia (MBG-CSIC)

Optimal Control, Reinforcement Learning
and Inverse Inference for
Biological Dynamical Systems

What cells do, why they do it, and how to infer it from data

Adrián Blanco Aguiar·Supervisor: Julio R. Banga

Computational Biology Lab, Misión Biolóxica de Galicia (MBG-CSIC), Pontevedra

VI PhDay-EIO · Vigo · 23 October 2026

Ministerio de Ciencia, Innovación y Universidades · Cofinanciado por la Unión Europea · Agencia Estatal de Investigación

Project DYNAMO-bio (PID2023-146275NB-C22) funded by MICIU/AEI/10.13039/501100011033 and ERDF/EU

Intro · Optimality

Why do bees build hexagons?

most honey · least wax?
Most honey, least wax
wall length per cell of unit area · shorter = less wax
Varro · 1st c. BC conjecture Darwin · 1859 adaptation Hales · 1999 proof
Hypothesis: evolution shapes living systems to optimise a criterion
This thesis: biological regulation as an optimal-control problem
Banga & Sager 2025 · Hales 2001
Intro · Optimality in biology

Optimality principles at every scale

  1. Molecular
    • Protein folding: native state has the lowest free energy
    • Optimality of the genetic code
  2. Pathway
    • Optimal control explains temporal gene expression in simple metabolic pathways
    • Flux balance analysis predicts optimal growth in bacteria
    nexta metabolic-pathway example
  3. Cell
    • Optimal control explains dynamic allocation of cellular resources
    • Optimal control for proteome adaptation in bacteria
  4. Tissues & organs
    • Optimality in the vascular system
    • Optimal design in compound eyes
  5. Organism
    • Optimality in the foraging behaviour of animals
    • Optimality of human gait
  6. Population
    • Optimal sex ratio
    • Optimal allocation of resources in a wasp colony
  7. Ecosystem
    • Optimality in microbial consortia
    • Optimal foraging in marine ecosystem models
Banga & Sager 2025, Table 1 (non-exhaustive; ~40 examples)
Intro · Motivation

Cells switch enzymes on just in time

e₁ e₂ e₃ S₁ constant pool S₂ S₃ S₄ Enzyme budget Σi ei ≤ 1 0 1 time t = 0.00
LPN3B optimum: switches at $t\approx$ –, $t_f^*\approx$ –
2002 predicted
Minimum time + limited enzyme budget ⇒ sequential, just-in-time activation (Klipp et al. 2002)
2004 observed
E. coli amino-acid biosynthesis (Zaslaver et al. 2004); yeast (Chechik et al. 2008)
objective? behaviour
Thesis plan · Setting

Regulation as optimal control

e₁ e₂ e₃ S₁ S₂ S₃ S₄
$$\dot{\cx{x}}=f(\cx{x},\cu{u},\cth{\theta}),\qquad \cx{x}(t_0)=\cx{x}_0(\cth{\theta}),\qquad \cy{y}=g(\cx{x},\cu{u},\cth{\theta})$$
x metabolites / gene expression  ·  u enzyme expression  ·  θ kinetic parameters  ·  y observables
$$\min_{\cu{u}(\cdot),\,t_f}\; J[\cu{u}]=\int_{t_0}^{t_f}\htmlClass{s03-L}{L}(\cx{x},\cu{u},\cth{\theta})\,dt+\Phi\big(\cx{x}(t_f)\big)$$ $$\text{s.t. dynamics},\quad \cx{x}(t)\in\mathcal X,\quad \cu{u}(t)\in\mathcal U,\quad \cx{x}(t_f)\in\mathcal X_f$$
LPN3B: $L\equiv 1,\ \Phi\equiv 0,\ S_4(t_f)=0.9,\ \sum_i e_i\le 1$, free $t_f$ after discretisation: NLP, in general non-convex
But $L$ is rarely known  →  ill-posed inverse problem
Tsiantis & Banga 2020 · Banga & Sager 2025
Thesis plan · Overview

Thesis plan: four levels

cf. Banga & Sager 2025
Thesis plan · L1 · O1 schematic

L1 · Learn: what if part of the cell is unknown?

value function
$$V(t,\cx{x})=\min_{\cu{u}(\cdot)}\Big[\int_t^{t_f}L\,ds+\Phi\big(\cx{x}(t_f)\big)\Big],\quad \cx{x}(t)=\cx{x}$$
Hamilton–Jacobi–Bellman
$$-\frac{\partial V}{\partial t}=\min_{\cu{u}\in\mathcal U}\Big[L(\cx{x},\cu{u},\cth{\theta})+\htmlClass{s05-hl}{\nabla_{\cx{x}}V\cdot f(\cx{x},\cu{u},\cth{\theta})}\Big],\qquad V(t_f,\cdot)=\Phi$$
OCoptimal control
$\dot{\cx{x}}=f(\cx{x},\cu{u},\cth{\theta}),\ \ L$
$f,\,L$ known → solve
RLreinforcement learning
samples $(\cx{x},\cu{u},L,\cx{x}')$
$f,\,L$ only through samples → learn
$V(t,\cx{x})$
one shared $V$
illustrative
Grey box
$\dot{\cx{x}}=f\big(\cx{x},\htmlClass{s05-pi}{\pi_\eta(\cx{x})},\cth{\theta}\big)$
$\dot{\cx{x}}=f(\cx{x},\cu{u},\cth{\theta})$
$\cu{u}=\htmlClass{s05-pi}{\pi_\eta(\cx{x})}$
known mechanism +
learned regulatory feedback
Casescentral carbon metabolismstomatal dynamics
Gap · no unified OC–RL framework exploiting the HJB equivalence in biological modelling
Bertsekas 2019
Thesis plan · L2 · O2 schematic

L2 · Infer: what is the cell optimising?

statistics: weighted least squares
$$\min_{\cinv{w}\in\Delta,\ \cth{\theta}\in[\cth{\theta}^{LB},\cth{\theta}^{UB}]}\ \sum_{k,j,i}\omega_{jki}\Big\|\tilde{\cy{y}}_{jki}-\cy{y}_{jki}\big(t_i;\;\htmlClass{s06-uo}{\cu{u}^*(\cinv{w},\cth{\theta})}\,,\;\cth{\theta}\big)\Big\|^2+\rho\,\mathcal R(\cinv{w})$$
$\omega$ inverse-variance weights $\mathcal R$ sparsity-promoting
forward OC solve
OR: optimal control
$$\htmlClass{s06-ui}{\cu{u}^*(\cinv{w},\cth{\theta})}\,=\arg\min_{\cu{u}(\cdot)}\int_{t_0}^{t_f}\sum_{m=1}^{M}\cinv{w_m}\,L_m(\cx{x},\cu{u},\cth{\theta})\,dt+\Phi\big(\cx{x}(t_f)\big)\quad\text{s.t. dynamics}$$
$L_m$ candidate cost kernels · illustration
$\cinv{w}\in\Delta=\big\{\cinv{w}\ge0,\ \textstyle\sum_m \cinv{w_m}=1\big\}$
every outer evaluation = one inner OC solve
Gap · no IOC/IRL that is mechanistic, data-consistent and interpretable at once
Tsiantis, Balsa-Canto & Banga 2018 · Banga & Sager 2025
Thesis plan · L2 · O2 schematic

L2 · Why the objective is hard to identify

Scaling: $\arg\min J=\arg\min cJ,\ c>0$
→ normalise on $\Delta$
Equivalent cost functionals: $\cinv{w}\neq \cinv{w'}$, same $\cu{u}^*$
Constraint-driven arcs: active constraints fix $\cu{u}$
→ no information on $\cinv{w}$
Ill-posed inverse LQR (Priess et al. 2015)
Tools sparsity & biological priors inverse KKT costate reconstruction $\mathcal H=L+\lambda^{\top}f,\qquad \dot\lambda=-\nabla_{\cx{x}}\mathcal H$
Test: do inferred objectives generalise across conditions?
Thesis plan · L3 · O3 schematic

L3 · Explain: which mechanism implements it?

Samples $(\cx{x}_k,\cu{u}^*_k)$ from OC / RL policies
$$\Theta(\cx{x})=\Big[\,1,\ \cx{x},\ \cx{x}^2,\ \frac{\cx{x}^{n}}{K^{n}+\cx{x}^{n}}\Big|_{(K,n)\in\text{grid}},\ \dots\Big]$$ $$\hat\xi=\arg\min_{\xi}\ \big\|\cu{\mathbf u}^*-\Theta(\cx{\mathbf X})\,\xi\big\|_2^2+\tau\|\xi\|_0$$
least squares → dense $\xi$
sequential thresholding → sparse $\hat\xi$: one Hill term survives
Black-box policy → few-term feedback law (e.g. Hill-type)
$$\cu{\hat u}(\cx{x})=\hat\xi_{K,n}\,\frac{\cx{x}^{n}}{K^{n}+\cx{x}^{n}}$$
Closed loop $\dot{\cx{x}}=f\big(\cx{x},\Theta(\cx{x})\hat\xi,\cth{\theta}\big)$: stability & invariance
Gap · sparse model discovery grounded in mechanistic constraints is underdeveloped
SINDy (Brunton et al. 2016) · SINDy-RL (Zolman et al. 2025)
Thesis plan · L4 · O4 schematic

L4 · Interact: what if several agents have their own goals?

$$\min_{u_i}\ J_i(u_i;u_{-i})=\int_{t_0}^{t_f}L_i(\cx{x},u_i,u_{-i})\,dt+\Phi_i\big(\cx{x}(t_f)\big),$$ $$\dot{\cx{x}}=f(\cx{x},u_1,\dots,u_N)$$
$$\text{Nash:}\quad J_i(u_i^*;u_{-i}^*)\le J_i(u_i;u_{-i}^*)\quad\forall u_i,\ \forall i$$
  • MARL behaviour vs analytical Nash: what do deviations mean biologically?
  • Inverse games / inverse MARL: infer each player's objective
Microbial communitiesPlant resource allocation
Gap · inverse games / inverse MARL almost absent for mechanistic biological models
Inga et al. 2019 · Molloy et al. 2020 · Lian et al. 2022
Current work

One brick under every level: solving OC reliably

Current work · Methods

Discretise, then optimise

continuous control $\cu{u}(t)$ $N$ piecewise-constant steps (PWC)
CVP (single shooting)
controls (+ $t_f$) only; states by integration
Multiple shooting
+ node states,
$\cx{x}_{k+1}-F(\cx{x}_k,\cu{u}_k)=0$
→ later
Full discretisation
collocation / RK defects: all states & controls
Three pipelines · same tolerances for all derivativesintegratorNLP solver
CasADi CVP & MS (PWC/PWL)
exact AD gradientsexact Hessian except BSUB: L-BFGS CVODES IPOPT
FiniteDiff CVP (PWC/PWL)
previous practice
forward FD ode15s fmincon
ACADOS (PWC only)
tuned per problem
exact AD (CasADi codegen)Hessian: exact or Gauss–Newton Gauss–Legendre IRK SQP + HPIPM
Andersson et al. 2019 · Verschueren et al. 2022
Current work · Protocol

Five pathways, 1000 starts, one strict referee

(states, controls) = 1 control
LPN3B (4, 3)
min $t_f$
LPDN4 (8, 4)
time + enzyme cost
GBD (9, 4)
min $t_f$
SC 6 enzymes
yeast diauxic shift
max survival
BSUB (26, 13)
B. subtilis nutrient shift
max ATP
1000 random starts
per (method, problem)
uniform controls · random $t_f$
MS nodes by forward simulation
solver
$\varepsilon=\delta=10^{-6}$
$\varepsilon_{acc}=10^{-4}$
classifier
converged
acceptable
failed
CVODES referee
re-simulation $10^{-10}/10^{-12}$
violation $<\delta$
verified
rejected
0solves
0summed solver time
(Drago, CSIC)
rejected:0
Current work · Results

No universal winner

success %
Ranking is problem-dependent
ACADOS: fastest on 3 small problems; SC 0 %, BSUB 4.1 %
AD-based vs FD-based CVP pipeline:
39–159× faster*
*Pipelines also differ in NLP solver and integrator
Current work · Results

Where do the starts end?

All successes agree
LPN3B, LPDN4, GBD, BSUB: successful starts agree on J (10−3)
→ risk = non-convergence
SC · yeast diauxic shift
No start converged; all successes acceptable
Wide spread of end points, not proven local optima
What comes next

What comes next: fast, verified inner solvers

Separate protocol workstation · cold start · 600 s · 90 held-out starts · not comparable with HPC numbers
Method search by an agentic multi-agent LLM workflow · pre-registered seeds · read-only verifier
Direct transcription (full discretisation: Radau collocation or RK4) · exact sparse Hessian · single IPOPT solve (two stages on SC): on all 5 held-out Pareto fronts
fastest baseline at tied $J^*$ · benchmark mesh
← faster expected time to first verified hit $\mathbb E[T]$ (s, log scale) · whiskers: bootstrap 95% CI
≈ 5–20×
faster $\mathbb E[T]$ at equal $J^*$
on the benchmark mesh (GBD, LPN3B, LPDN4, BSUB)
Next → tentative: the most promising method from this search as the inner solver of bilevel IOC (Level 2)
Closing

Take-home & thanks

S₁ e₁ e₂ e₃ S₂ S₃ S₄
1Optimality as a hypothesis
regulation = optimal control
L1 L2 L3 L4
2Thesis
OC ⇄ RL → inverse objectives → mechanisms → games
3Foundation
non-convex even in simple cases · independently verified · no universal solver

adrian.b@csic.es

Thank you!
Questions?
Ministerio de Ciencia, Innovación y Universidades · Cofinanciado por la Unión Europea · Agencia Estatal de Investigación

Project DYNAMO-bio (PID2023-146275NB-C22) funded by MICIU/AEI/10.13039/501100011033 and ERDF/EU

Computing: Drago (CSIC)

CSIC Misión Biológica de Galicia (MBG-CSIC)
Backup B1

Proposition 1: derivation and scope

$$\dot S_1=0,\quad \dot S_2=\htmlClass{b1-e1}{e_1}S_1-\htmlClass{b1-e2}{e_2}S_2,\quad \dot S_3=\htmlClass{b1-e2}{e_2}S_2-\htmlClass{b1-e3}{e_3}S_3,\quad \dot S_4=\htmlClass{b1-e3}{e_3}S_3,\qquad S(0)=(1,0,0,0)$$
e₁e₂e₃ S₁ S₂ S₃ S₄ ≡ 1
1Factorise
Constant $e$ ⇒ a linear cascade with $S_1\equiv1$ ⇒ $S_4(t_f)=\htmlClass{b1-e1}{e_1}\,\varphi(\htmlClass{b1-e2}{e_2},\htmlClass{b1-e3}{e_3},t_f)$
$$\varphi(a,a,t)=t-\frac{2\,(1-e^{-at})}{a}+t\,e^{-at}\qquad\big(\varphi(0.3,0.3,10)\approx4.163\big)$$
2Propagate
  • $A:(e,t_f)\mapsto(e,\dots,e,t_f)$ is linear and $\mathcal F_1=A^{-1}(\mathcal F_N)$ (PWC; PWL with equal node values)
  • MS, exact integration: the projection of $\mathcal F_{MS}$ onto $(u,t_f)$ is $\mathcal F_{CVP}$
3Scope
The counterexample uses the equality $S_4(t_f)=0.9$ (benchmark formulation)
Toy example: illustrates that non-convexity can appear
not claimed for collocation · non-convex $\not\Rightarrow$ multiple local optima
Constant controls in the $N$-interval spaceschematic
interval 1 intervals 2 … N
constant controls
$(e,\dots,e,t_f)$
$\mathcal F_N$
$S_4(t_f)=0.9$
midpoint
$\notin\mathcal F_N$
If $\mathcal F_N$ were convex, so would be $\mathcal F_1$: contradiction
Backup B2

Full multistart results (success %, mean time per start)

Success % · converged + acceptable, CVODES-verified  ·  mean time per start · all 1000 starts
  • $\varepsilon=\delta=10^{-6}$, $\varepsilon_{acc}=10^{-4}$ · max iterations 1000 (IPOPT, fmincon), 10,000 (ACADOS) · N
  • CasADi: exact AD gradients; IPOPT exact Hessian, except BSUB (L-BFGS)
  • 95% half-width $\le$ pp
  • FD vs CasADi: different derivatives, integrator and NLP solver
Backup B3

Non-identifiability in IOC: the mechanisms

1
Scaling
$\arg\min_{\cu{u}} J=\arg\min_{\cu{u}}\,cJ\ \ (c>0)$ ⇒ normalise $\cinv{w}\in\Delta$
J cJ u* ⇒ w Δ
2
Equivalent functionals
$$\int_{t_0}^{t_f}\big(L+\nabla\psi\cdot f\big)\,dt=\int_{t_0}^{t_f}L\,dt+\psi\big(\cx{x}(t_f)\big)-\psi(\cx{x}_0)$$
boundary term only: indistinguishable if $\cx{x}(t_f)$ is fixed or compensated in $\Phi$
cf. reward shaping in RL
ψ(x0) ψ(x(tf)) same boundary term on every path
3
Inverse KKT
$\mathcal H=\sum_m \cinv{w_m}L_m+\lambda^\top f$: $\nabla_{\cu{u}}\mathcal H=0,\ \ \dot\lambda=-\nabla_{\cx{x}}\mathcal H$
→ linear in $(\cinv{w},\lambda)$ on interior arcs
Active constraints add multipliers → residual absorbed → no information on $\cinv{w}$
t u umax constraint-driven arc interior
Context, cf. Banga & Sager 2025: parameters→structure→objectives→competing objectives
Standard arguments, not thesis results
Backup B4

Method search: protocol, mesh choices, SC result, caveats

Proposers →synthesiser →adversarial critics →7 method families
  • isolated git worktrees · independent audits · read-only harness & verifier
  • Seeds: dev 1–10, held-out 11–100 (pre-registered) · 3,600 held-out jobs ≈ 116 h
  • Success = classifier + 600 s cold-start limit + CVODES check
  • $\mathbb E[T]=\mathbb E[\tau]/p$: expected time to first hit under iid restarts (Wald) · 2,000-sample bootstrap 95% CIs
  • Main fronts allow coarser tied meshes (LPN3B $N=50$: 0.29 s; BSUB $N=4$: 0.40 s) → 8–20×; benchmark mesh only → ≈ 5–20× (S17)
SC · held-out test, 90 starts per method baselines $(N=10)$ new methods
$\mathbb E[T]$ (s, log scale) · whiskers: bootstrap 95% CI · $J^*$ = best verified survival time (higher is better)
$J^*$ vs mesh size $N$ (held-out test)
SC, mesh refinement ($N=10\to80/200/500$)
$J^*=$ 95.742 ($N=80$) on 90/90 · $\mathbb E[T]$ 13.6 s
Best baseline (MS PWL, $N=10$): $J^*=$ 94.738 · 13.3% hit · ≈ 4,150 s
Caveats: finer mesh; designed after seeing test results