Shared Noeis wiki

How to Think About AI

A sourced primer on the few ideas that make modern AI systems legible: how they scale, how they act, what they can see, how they are measured, and whether they do what we meant.

Scaling Laws

A scaling law is an empirical regularity: as a language model gets more parameters, more training data, or more training compute, its prediction error falls along a smooth and predictable curve.^[1] Kaplan and colleagues found that this loss follows a power law in each of those three quantities, while architectural details such as width and depth matter far less within a wide range.^[1] The same equations let researchers decide how to divide a fixed compute budget before a large training run begins.^[1]

How to divide that budget has itself been revised. Kaplan et al. favored very large models trained on a relatively modest amount of data.^[1] Hoffmann et al. concluded instead that model size and training tokens should grow in equal proportion, and that many large models of the time were "significantly undertrained."^[2] Their 70 billion parameter Chinchilla, trained on more data with the same compute as the 280 billion parameter Gopher, outperformed it.^[2]

Smooth gains in loss do not settle how specific abilities improve. Wei et al. described emergent abilities that are absent in smaller models and present in larger ones, arguing they cannot be predicted by extrapolating from small models.^[3] Schaeffer et al. countered that many of these jumps are produced by the choice of evaluation metric, and that continuous metrics often show steady improvement instead.^[4] The disagreement is a reminder that what scaling appears to deliver depends on how it is measured.

Read next: Evals, Capability vs Alignment.

Agents

Anthropic distinguishes workflows, in which language models and tools are orchestrated through predefined code paths, from agents, in which the model directs its own process and tool use.^[1] The useful question is who decides the next step: the programmer or the model.^[1]

Research on agents showed the value of letting a model alternate between thinking and doing. ReAct has a model produce reasoning traces and task actions in an interleaved way, so its plans can be updated with what its actions return from the outside world.^[2]

Autonomy has a price. Because an agent chooses its own path, it tends to cost more and its mistakes can compound over a long run, so Anthropic advises "finding the simplest solution possible, and only increasing complexity when needed."^[1] One way to gauge what agents can handle is METR's time horizon: the length of task, measured by how long it takes humans, that a model completes with 50% success.^[3]

Context Windows

A model's context is the set of tokens it sees when producing a response: the instructions, documents, tool results and conversation for the task at hand.^[1] It is a working set for one task, not durable memory, so anything an agent needs later has to be written somewhere outside the window and brought back in.^[1]

A bigger window is not the same as better use of it. Liu et al. found that performance is often highest when relevant information sits at the beginning or end of the input, and degrades when the model must find it in the middle of a long context.^[2] Anthropic describes context as "a finite resource with diminishing marginal returns."^[1]

This moves the craft from writing a single prompt to curating what enters the window at each step, which Anthropic calls context engineering.^[1] For long tasks, one technique is structured note-taking, in which an agent regularly writes notes that persist outside the context window and retrieves them when relevant.^[1]

Evals

An eval is a structured test of what a model does, and comparisons are only as good as the tests behind them. The HELM project found that models had been evaluated so inconsistently that some prominent ones did not share a single scenario in common.^[1] Its response was to test many scenarios on several axes at once: accuracy, calibration, robustness, fairness, bias, toxicity and efficiency.^[1]

The scoring rule shapes the conclusion. Schaeffer et al. showed that abilities which seem to appear suddenly can look gradual under a continuous metric, arguing that such abilities "appear due to the researcher's choice of metric."^[2] Reading any eval result therefore means asking what was scored, and how.

Some evals try to report capability in units that stay meaningful as models change. METR measures the length of tasks, timed by how long they take humans, that a model can complete with 50% success.^[3] A score framed this way answers a practical question about how much work a system can be trusted to finish.

Read next: Capability vs Alignment, Scaling Laws.

Capability vs Alignment

Capability is what a system can do; alignment is whether it does what its designers and users actually intend. The two do not rise together automatically: Ouyang et al. found that making language models bigger does not by itself make them better at following a user's intent.^[1] In their study, people preferred outputs from a 1.3 billion parameter model fine-tuned with human feedback over those of the 175 billion parameter GPT-3.^[1]

Misalignment often begins with the objective. DeepMind defines specification gaming as "behaviour that satisfies the literal specification of an objective without achieving the intended outcome."^[3] In its examples, the fault lies in a misspecified task rather than in any flaw of the learning algorithm.^[3]

Amodei et al. framed such failures as accidents: unintended and harmful behavior that can emerge from poor design of real-world AI systems.^[2] They grouped the open problems as avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration, and robustness to distributional shift.^[2] Because these problems are separate from raw capability, alignment needs its own measurement rather than being inferred from how capable a system is.