One assistant, four optimizers: it trains, experiments, routes you, and acts

Every message you send to a modern AI assistant sets off several optimizers at once. One of them tuned the model's weights. Another is deciding right now which prompt or example will make the next answer better. A third, running in real time, picks which model should answer you at all. A fourth decides what the assistant should do next, with tools that change the outcome of everything it already did. Most of us learned to think of optimization in AI as one thing: taking gradient steps on a training loss. This tour's first claim is that a deployed assistant is not one optimization problem. It is four, and they feed on different signals, run on different clocks, and answer to different method families.

The map we follow comes from one lecture: AIAA 1010 Week 2 at HKUST(GZ), delivered in 2026 by Yao Shu, the principal investigator of the XPLOR Lab. The lecture's spine is a question-first habit, and this article inherits it: ask what can change, what feedback can be observed, when the choice must be made, and how today's choice changes tomorrow. Ask those four questions honestly, and the right method family mostly falls out on its own.

1. Why does one assistant need four kinds of optimization?

1.1 One system, four optimization problems

The lecture opens by drawing the pipeline that the rest of the course unpacks: four stages, left to right, with an arrow from each to the next.

Caption: The lecture's opening pipeline. One assistant runs four optimization jobs: improve weights (train and adapt), choose experiments (prompts and examples), serve users (model routing), and act over time (tools and feedback). The blue arrows mark a one-way flow: each stage feeds the next. There are no axes or numbers here, because this is a map of jobs, not a chart of results.

Each stage is an optimization problem with a different object, a different feedback signal, and a different rhythm. Walk the four in order.

Improve weights. The object of optimization is the model's parameters. The feedback is a gradient when you can afford backpropagation, and only function values when you cannot. The rhythm is an offline loop over batches of data, the slow daily grind of training and fine-tuning. Optimizers here are the classic first-order family and its relatives: gradient descent, curvature-aware methods, and, when no gradient is available, zeroth-order schemes that work from function values. No method internals are needed yet; just note where this stage lives.

Choose experiments. The object is not parameters but configurations: which instruction to put in the prompt, which examples to include, which system prompt to ship. The feedback is the measured quality of the validated configuration, and measuring it is often expensive, because each trial costs API calls or human review. The rhythm is a slow trial loop: propose a configuration, validate it, use the result to propose the next one. The natural family here is Bayesian optimization and its neural variants.

Serve users. The object is the live decision of which model should answer a request, and how serving resources are allocated. The feedback is partial in a precise sense: you observe the outcome only of the model you actually chose, while what the other models would have produced stays hidden. The rhythm is real time, per request, and the decision must be made before the next request arrives. The families here are contextual bandits and online optimization.

Act over time. The object is a policy over multi-step behavior: which tool to call, whether to revise an answer, what to try next. The feedback is the outcome of the whole trajectory, and today's action changes which actions are possible tomorrow. The rhythm is sequential. This is reinforcement learning's territory.

You might notice a pattern: all four are "find better settings" loops, and the differences are not in the loop but in what each one can observe during the loop and how far each decision reaches. Before any method gets named, that pair of differences is worth holding onto.

1.2 Where optimization sits, and whose map this is

The lecture places optimization between two other concerns. Approximation asks whether the model can express the target at all. Generalization asks whether what the model learned works on new data. Optimization is the activity in between: find the good settings that turn expressive capacity into something that actually performs. It is the middle choice between "can it, in principle" and "does it, in practice."

The deck also tells you where this particular map comes from, because it matters for how the rest of the lecture argues.

Caption: The deck's viewpoint slide: a Theory-First program in which theory guides learning and optimization, which in turn guides artificial intelligence. The three columns name the research pillars (optimization and decisions under limited feedback; learning theory; LLMs and AI agents), and the red box states the program's rule: "Theory guides algorithms. Applications reveal the next theoretical question."

That red rule is not decorative. It is the stated research position of the XPLOR Lab: theory first, with applications expected to surface the next open question. The lab's homepage describes the goal as developing foundational principles and practically useful algorithms that reshape how AI systems are understood and optimized, and this lecture is its teaching surface. HKUST(GZ)

So the stance of this whole tour is inherited from the lecture: understand the assumptions and the bottlenecks before choosing an algorithm. One assistant, four optimization jobs. The next question, which drives the whole article, is how you pick the right lens for each of them.

2. Ask before you choose: what can change, what can you observe?

The natural response to "an assistant runs four optimizers" is to grab a method and apply it. The lecture's counter-move: do not. Ask first. The deck proposes a diagnostic made of questions that force the problem to describe itself before any algorithm gets a seat at the table.

Caption: The lecture's question-first diagnostic, three columns. "What is assumed?" lists the hidden premises a method leans on, for example that gradients are available, or that every update needs fresh information. "What is the bottleneck?" asks which resource actually binds: time, memory, too few trials, too much noise. "What can we explain?" demands that a method come with an account of why it helps, when it should fail, and where else it could work. The red line states the author's lens verbatim: "I look for an explanation that can guide the next algorithm."

The three columns here: what is assumed, what is the bottleneck, what can we explain. Each method family is built on an assumption (gradients exist, updates need fresh data), addresses a bottleneck (memory, trials, noise), and ought to come with an explanation of when it works and when it breaks. If a method's assumption is false in your setting, no amount of tuning saves it. Notice also that this is a research habit, not a community-standard taxonomy; the deck offers it as a lens, and this article treats it the same way.

2.1 Collapsing to the two questions that do the work

Underneath the three columns sit four plain questions that the lecture returns to throughout: what can change, what feedback can be observed, when must the choice be made, and how does today's choice change tomorrow. Most of the time you do not need all four to pick a lens, because two of them are fixed by your situation before you start. "What can change" is dictated by your problem: weights, prompts, routes, or actions. "When must you decide" is dictated by your deployment: offline batches, per-request real time, or continuous loops.

The two questions that actually vary from problem to problem, and that determine the method family, are the other two. The deck condenses the whole diagnostic into them on page 39, and the banner reads:

"what feedback is available, and how do decisions affect the future?"

Read that banner as the operational form of the four-question checklist. Everything you need to pick a start method comes down to: after you make a choice, what signal can you observe, and does that choice change the options you will face tomorrow. Chapter 4 draws the entire method map on exactly these two axes.

So the habit to take from this chapter is: methods are conclusions, not starting points. Name the feedback and the horizon first; families follow. What remains surprising is that five families that look utterly different all turn out to run the same loop.

3. Five very different methods, one shared loop: choose, observe, learn

Zeroth-order optimization, Bayesian optimization, contextual bandits, online optimization, and reinforcement learning. On the surface these are five worlds with five vocabularies, five sets of guarantees, five communities. The lecture's central claim is that all five are the same loop wearing different clothes: choose a trial, observe an outcome, learn from experience, repeat.

Caption: The shared loop at the center of the lecture. Choose a trial (the slot-machine lever), observe an outcome (the camera), learn from experience (the magnifier over past records), then loop back to the next choice. In the middle of the circle the deck writes its unifying message: "Different feedback. Related learning questions." The five paradigm families fill the same three slots; what differs is the content of each slot.

Now fill the three slots for each family. One or two sentences per family is all this tour needs; the mechanics live elsewhere.

The table is the whole argument. The loop is identical; the "observe" slot is what differs: a loss value, an expensive evaluation, a chosen-arm outcome, today's losses, a trajectory return. Different feedback, related learning questions. That is also why the loop closes: what you learn changes what you choose next, which is where exploration enters, a theme we will meet again in chapter 5.

Engineers already run this loop daily, under another name: pick a feature variant, watch the instrumentation metrics, decide whether to roll it out, then start again on the next variant. The mapping is exact, because choose, observe, learn is not a metaphor; it is the actual structure underneath the five families.

3.1 The loop is a live research area, not a metaphor

The lecture is not alone in seeing the loop. A 74-page survey by Kimiaei and Kungurtsev, submitted to arXiv in September 2025 (coverage cutoff 2025-09), formalizes the same workflow for black-box optimization solvers: proposal, evaluation, refinement, repeat. Its framing of machine learning's role is striking: ML and RL do not introduce entirely new classes of solvers, but instead enhance existing ones by providing richer models, adaptive updates, and data-driven strategies. In other words, the research community is busy building a second layer on top of the same choose-observe-learn foundation. BBO Survey

The survey's own taxonomy makes the point concrete.

Caption: Independent confirmation that this territory is active: a re-render of Table 2 of the September 2025 survey, mapping eight ML-enhanced black-box optimization methods to the survey's four ML roles: surrogate modeling, optimizer-inspired updates, meta-learning portfolios, and generative neural models. The survey formalizes the shared loop as its "generic workflow." Coverage ends September 2025, and this table is the survey's own organization of the field, not a complete taxonomy and not a figure from the original paper. BBO Survey

So two independent sources, a lecture and a survey, converge on one loop. What separates the cells is what each step consumes and produces, and that is precisely where the two questions from chapter 2 cut the map in two.

4. Two questions split the map: what feedback, what horizon?

The lecture's own methods map draws the four lifecycle stages and, next to each, the method family that lives there. Read it once and the two axes appear by themselves.

Caption: The four stages paired with their method families. Improve model weights: gradients, Hessian/curvature, or zeroth-order evaluations. Choose an experiment: Bayesian optimization, prediction plus uncertainty. Learn while serving: bandits and online optimization, with partial feedback and unknown future losses. Learn how to act: reinforcement learning, where actions change future possibilities. Top to bottom, two things change together: the feedback you can observe thins out, and your decisions reach further into the future.

That is the two-question banner from chapter 2 made spatial. Every method on this map is positioned by where it falls on two axes: feedback availability, and decision horizon.

4.1 The feedback axis: from full gradient to black box

Feedback availability runs a spectrum. At one end you observe the full gradient of your objective. Then curvature information (the Hessian), if you can afford it. Then function values only, no derivatives at all: zeroth-order territory. Then partial feedback, where only the chosen action's outcome is visible. At the far end, the objective itself is a black box that answers only with a score. Each step toward the black-box end loses information about what you are optimizing, and each step forces the method to lean harder on structure, history, or more trials.

The two non-gradient ends deserve concrete anchors, because they are where the axis stops being a textbook diagram. Both exist at LLM scale today.

The loss-only end. Fine-tuning a large language model is bounded by memory before anything else, and gradient-based fine-tuning is memory-hungry. MeZO showed that a forward-only optimizer, one that works from function values alone, can fine-tune OPT-30B on a single 80 GB A100, at the same memory footprint as inference, where Adam fine-tuning on the same hardware fits only OPT-2.7B. The cell "no gradient, only loss values" is a real production regime, not a corner case, and that is why zeroth-order methods stay relevant at all. MeZO

Caption: MeZO anchors the loss-only end of the feedback axis. Left panel: GPU memory consumption across OPT model sizes for Adam fine-tuning, prefix fine-tuning, and inference. Right panel: the largest OPT model each method fits on given hardware; on a single 80 GB A100, Adam fine-tuning fits OPT-2.7B while MeZO, running at inference-grade memory, fits OPT-30B. The panel is about memory footprint, not about accuracy. MeZO

The expensive black-box end. At the other extreme, the object you optimize is closed to you entirely: an instruction for a black-box LLM, scored only through API calls. INSTINCT treats the target model as a black box: white-box decoding (Vicuna) proposes candidate instructions, black-box evaluation (ChatGPT) scores them on a validation set, and a neural surrogate trained on history picks the next candidate. One loop iteration equals one validated configuration, and each verification spends API calls on the whole validation set. Everything is expensive, everything is opaque, so the method must spend its trials wisely. INSTINCT

Caption: INSTINCT anchors the expensive black-box end: observation history feeds a neural surrogate, an acquisition step picks the next soft prompt, white-box decoding turns it into an instruction, black-box evaluation scores it on the validation set, and the scored pair joins the history. The arrows close the loop: one iteration, one validated configuration. INSTINCT

MeZO on one side, INSTINCT on the other: the two images bracket the entire feedback axis. On the left you observe only loss values at LLM scale; on the right you observe only scored configurations through an API. Everything in between is a compromise between these two extremes.

4.2 The horizon axis: does today's choice change tomorrow?

The second question is about time. A decision is single-step when today's choice does not change the options you will face tomorrow: pick a point, evaluate it, done. A decision is sequential when today's action changes the state you start from next, and credit for eventual success must be spread back across the steps that led to it. The map's top stages live at the single-step end; the bottom stage, reinforcement learning for agents, lives at the sequential end.

The lecture makes a point of showing that this axis is a research result, not an analogy. The slide below is the deck's own evidence.

Caption: The deck's "Established connections" panel. Left: zeroth-order estimation connected to single-step policy optimization, under matched objectives and assumptions, with the two dated milestones: 2025 gradient estimation, and 2026 curvature estimation (ZoVH). Right: the ongoing research direction, carrying useful principles, explaining the limits, and designing better algorithms. The red banner states the value standard: a useful framework helps us ask better questions and build on existing knowledge.

Two papers back the timeline. On the gradient side, Qiu et al. (2025, ICML workshop) proved that the Gaussian-smoothed zeroth-order objective is identical to a single-step policy optimization objective, and that the standard two-point zeroth-order gradient estimator is mathematically equivalent to a single-step REINFORCE estimator under a specific baseline. ZoAR On the curvature side, Qiu, Hong, Xu, and Shu (2026, ICML) extended the same lens from gradients to Hessians: general zeroth-order Hessian estimators are theoretically equivalent to the Hessian of a smoothed single-step policy optimization objective. ZoVH

Both results are strictly single-step, and the slide's footnote says so in terms that must be read literally: "Single-step connections do not imply equivalence to general multi-step RL." The bridge connects estimation to one-step decision-making; it does not license any claim about sequential agents. Keeping that boundary visible is what makes the horizon axis honest.

The map is now complete: feedback availability tells you what you can observe, decision horizon tells you how far your choices reach. What remains is the question that produces actual research value: which ideas cross the boundaries between cells.

5. Noise, history, exploration — and counting the real cost

The lecture's transfer framework is worth reproducing in full, because it is the deck's answer to why a unified map pays off. An idea that works in one setting becomes a better starting point in another, but only through the middle step: understanding why and when it works.

Caption: The lecture's transfer framework. Top: an idea that works in one setting, understood for why and when, becomes a better starting point in a new setting. Two sidebars qualify the claim: what might transfer (noise, history, exploration), and what makes transfer useful (clear assumptions, an explanation, a better design). The red line gives the value standard: unification is valuable when it helps us reason and create.

Three themes recur across every cell of the map. Two of them can be shown concretely, one with a pair of figures.

Noise. Every cell's feedback is noisy. Mini-batch gradients are estimates of a true gradient, bandit rewards are random draws, reinforcement learning returns are stochastic, and function evaluations carry corruption. A method family is partly an answer to how much noise it tolerates and what it does with that noise: average it out, model it explicitly, or explore through it. This is why the diagnostic in chapter 2 asks "too much noise?" as one of its bottleneck questions.

History. Past observations are a resource, and different cells spend it differently. The same theme appears at zeroth order and at first order.

At zeroth order, ZoRD treats the optimization trajectory itself as reusable information: under a Gaussian-process assumption on the objective, the derivative is itself a GP whose posterior mean and uncertainty can be computed from past queries alone, so estimating a derivative costs zero additional function evaluations. More history, tighter estimate. ZoRD

Caption: ZoRD makes "history" literal at zeroth order. Four panels show the derived GP for derivative estimation as the number of trajectory queries grows from n=3 to n=12: black dashed is the true derivative, green solid is the GP posterior mean of the derivative, the light band is its posterior standard deviation, and the dots are function queries. With more past queries the green estimate tightens onto the true curve, and no extra queries were paid to get it. ZoRD

At first order, the same theme reappears. OptEx learns from the gradient history, uses it to predict where future iterations will visit, and then runs real gradient updates at those predicted points in parallel.

Caption: OptEx makes "history" literal at first order. At iteration t, a kernelized estimator built from past gradient evaluations predicts the next N candidate iterates, and N real gradient updates then run in parallel on those candidates. The unit of efficiency here is the sequential iteration, not the arithmetic operation count. OptEx

Two cells, one theme: history substitutes for fresh information at both ends of the feedback axis. What the theme saves differs, which leads directly to the accounting discipline below.

Exploration. Finally, every choose step faces the same trade: take what looks best now, or try something uncertain that might teach you more. Bayesian optimization prices that trade through acquisition functions, bandits through explicit exploration terms, zeroth-order methods through which perturbations to draw, and reinforcement learning through the exploration of actions. The form changes; the tension does not.

The lecture is equally clear about when transfer is worth attempting. Three themes recur, but recurrence alone is not a license: transfer is useful only when the assumptions are clear, an explanation ties the original setting to the new one, and a better design actually comes out the other side. Without those three conditions, borrowing an idea across cells is cargo cult, not science.

5.1 Count the real cost

The last transferable discipline is about honesty in accounting. A worked example from the lecture separates two resources that are constantly confused, query count and wall-clock time:

The numbers are illustrative, a schematic from the lecture, not a measurement of any real run. Read them anyway: optimizer A uses more evaluations yet finishes faster; optimizer B uses fewer evaluations yet takes longer. "A is faster" is only true on the wall-clock axis, and "B is cheaper" is only true on the evaluation-count axis. An efficiency claim that does not name its resource is meaningless, and this is the lecture's recurring demand.

The deck's own methods make the point verifiable. OptEx's acceleration is stated in exactly one resource: sequential iterations. With parallelism N, the SGD-based variant achieves an effective acceleration of Θ(√N) over standard SGD in sequential iteration complexity (its Corollary 2), meaning it saves sequential waiting time, not evaluations and not memory. That is the shape of an honest claim, and the standard for every other claim in this map. OptEx

The lecture condenses the whole chapter into one line, and it is worth keeping verbatim: "Learn from limited feedback. Connect useful ideas. Count the real cost." Learn is the diagnostic from chapter 2, connect is the shared loop from chapter 3, and count is the accounting above. Three imperatives, one article.

6. Where to go next: the reading route through the map

The map is complete. The deck closes by reconnecting every stage to the concrete methods that populate it, and the one-map reconstructs cleanly as a table:

Two observations about this table. First, the four perspectives are compatible, not competitors. The deck's own footnote says it plainly: "These are compatible perspectives. A single AI system can use several of them, and online or RL algorithms may use gradients." A production assistant trains its weights with gradients, selects prompts with Bayesian optimization, routes requests with bandits, and improves its agent behavior with reinforcement learning, all at once. The map is not asking you to pick one square; it is asking you to recognize which square a given decision lives in.

Second, the route through the map is ordered by the two axes. Start where feedback is richest, and move toward decisions that reach further into the future:

Caption: The reading route this article lays out. Start at direct gradients, where the signal is richest and the classic first-order families live. Pass through limited-feedback decisions, where experiments and serving are chosen under partial information. End at future-shaping actions, where reinforcement learning assigns credit across steps. The route is ordered by both the feedback axis and the horizon axis at once.

Start at the top of the feedback axis, where the terrace is well mapped: weights, gradients, curvature, and the first-order optimizers that run on them. If you want the mechanics behind this cell, the companion piece Rolling Downhill at Scale: Gradients, SGD, Adam, and OptEx goes through it method by method. The experimental cell is where trials are expensive and every choice must buy information; a full treatment lives in Choosing the Next Expensive Trial: Bayesian Optimization for RAG and Prompts. The deep end is the action cell, where credit must be assigned across time and black-box rewards still support updates; that is When Actions Change What Happens Next: Reinforcement Learning for AI Agents.

What this tour leaves you with is not a catalog of algorithms. It is a way of reading any optimization problem you meet: ask what feedback is available and how far the decision reaches; recognize the choose-observe-learn loop underneath whatever method is on offer; check which of noise, history, and exploration the problem is really about; and demand that every efficiency claim name the resource it saves. The deck closes with three imperatives — learn from limited feedback, connect useful ideas, count the real cost — and by now they read as procedure rather than slogan.

参考文献

本专题引用的核心资料如下，建议深入学习时阅读原文：

1. AIAA 1010 · Week 2: Optimization and Decision Making in AI — lecture slides by Yao Shu (HKUST(GZ), 2026; uploaded course material, no public URL)
2. Yao SHU faculty profile and XPLOR Lab page (HKUST(GZ)) — https://facultyprofiles.hkust-gz.edu.cn/faculty-personal-page?id=544
3. Revisiting Zeroth-Order Hessian Approximation: A Single-Step Policy Optimization Lens (ZoVH) — arXiv:2605.30960
4. Zeroth-Order Optimization is Secretly Single-Step Policy Optimization (ZoAR) — arXiv:2506.14460
5. Zeroth-Order Optimization with Trajectory-Informed Derivative Estimation (ZoRD) — OpenReview: n1bLgxHW6jW
6. OptEx: Expediting First-Order Optimization with Approximately Parallelized Iterations — arXiv:2402.11427
7. Fine-Tuning Language Models with Just Forward Passes (MeZO) — arXiv:2305.17333
8. Use Your INSTINCT: Instruction optimization for LLMs using Neural bandits Coupled with Transformers — arXiv:2310.02905
9. Machine Learning Algorithms for Improving Black Box Optimization Solvers — arXiv:2509.25592