Inference-time adaptation
Developing and serving inference-time adaptation techniques for deployed machine learning models.
Model behaviour should be generated for the input in front of it — not frozen at training time, and not slowly rewritten by everything the model has ever seen.
Thesis
Current model deployment is static. Models are trained, and then deployed. Models can be fine-tuned to adapt to different deployment environments, but within those deployments they are still static. Model harnesses are becoming prominent as a way to scaffold behaviour based on context, but the underlying model remains a static entity.
Continual learning is an endeavour to let a model be updated as it encounters novel information. This is a reactive mechanism for adapting a model to a deployment environment, and it is notoriously susceptible to catastrophic forgetting: behaviour degrades on existing capabilities. To counteract this, methods require accumulating a memory, which leads to bloat in the form of context rot, or increases latency as memories are retrieved. The cumulative changes induced by continual learning also hinder auditability — understanding or reproducing prior behaviour means storing model checkpoints, or a log of every change applied to the base model.
Turtle Research instead explores inference-time adaptation: proactive strategies for adapting model behaviour based on context. Updates are not accumulated. They are generated on-the-fly from the input context and, optionally, a small store of memory. This facilitates auditable, reliable, and effective deployment of dynamic machine learning models.
Reactive
Continual learning
- Updates accumulate into the weights
- Catastrophic forgetting
- Growing memory, context rot, retrieval latency
- Auditing needs checkpoints or a full change log
Proactive
Inference-time adaptation
- Adaptation generated per input, then discarded
- Base model is never overwritten
- Small, bounded memory — or none at all
- Behaviour reproducible from context alone
The bottleneck impeding the proliferation of inference-time adaptation is a lack of focus. Research into these strategies is preliminary, scattered, and often not conducted rigorously. Moreover, system design and optimisation is stuck in the static serving paradigm, and infrastructure facilitating end-to-end deployment of inference-time adaptation — from training to serving — is non-existent.
Questions we are answering
-
01
Can inference-time adaptation improve static models in- and out-of-distribution?
If adaptation only pays off under shift, it is a robustness trick; if it pays off on the standard case, it belongs in the serving path. This most appropriate setting to explore this is on robotics foundation models, against static fine-tuning.
-
02
For the strategy of generating adapters on-the-fly using a hypernetwork, which adapter shapes are best?
Adapters like LoRA and DoRA are not proposed in the inference-time adaptation context. So exploring their utility in this application is important. Performance, stability, latency and memory pull in different directions. Comparisons need to be under a fixed substrate, each against its own same-shape static control, so it is about the shape and nothing else.
-
03
How does inference-time adaptation behave in open-loop planning?
In open-loop planning, the interaction between planning updates and model updates could lead to unstable feedback loops. Thus, beyond improving planning success rates it is important to understand whether adapting inside a control loop produce unstable or erratic behaviour? S Quantities such as compounding error and regret against, in addition to success rate, should be measured when benchmarking inference-time adaptation strategies in open-loop planning.
-
04
What are the best practices for training inference-time adaptation strategies?
For adaptation strategies involving a phase of pre-training, what is the correct training pipeline? For example, for hypernetworks, are reconstruction, behavioural, or hybrid objectives more effective? How do these training pipelines scale with model and data budgets?
-
05
What data can be used to bootstrap or improve inference-time adaptation?
Adaptation has to be learned from something. Which corpora, traces, and forms of supervision actually transfer into better on-the-fly behaviour remains open.
-
06
How can a combination of inference-time adaptation strategies be served to many users efficiently?
Serving engines implement one point in a much larger space of adaptation operators. For efficient and effective deployment of inference-time adaptation, understanding the space of adapters that can be served, and how to do this concurrently across many users needs to be understood.
Contact
Collaboration, questions about the work, or a result you think bears on any of the six questions — get in touch.