Source: arXiv

Paper

Transformers as In-Context Samplers: From Closed-Form Diffusion to Estimation-Free Sampling — arXiv 2609.08981 (Sep 2026).

What it claims

LLMs are often sold as memorizers that need fine-tuning to generate new data. This paper extends in-context learning theory — a frozen transformer inferring from prompt examples with no parameter updates — from supervised tasks like linear regression to data generation.

The authors prove transformers can simulate iterative generative samplers from in-context samples alone. They construct closed-form and smoothed closed-form diffusion samplers: softmax attention computes responsibility weights and weighted empirical averages; feedforward layers implement Euler updates.

On semantic-topic sampling — prompts of words from one category (animals, foods, cities) — normalized hidden states show two-stage geometry: intermediate layers drift to a uniform spherical reference, then return to topic-dependent structure at the output. An interacting-particle energy on these clouds traces the same U-shaped curve across depth. The authors then prove transformers can approximate an energy-based sampler reproducing that U-shape — linking diffusion-style generation to layer-wise dynamics in pretrained models.

The breakdown

This is theory plus geometry, not a shipping sampler API. Proofs show what frozen attention plus MLP blocks can represent when prompt examples stand in for a training set. Semantic-topic tests use simple category lists — not long Korean documents — but the hook is layer trajectory: hidden states expand then contract in a measurable U-curve, not a flat forward pass.

Treat the energy-based proof as existence and alignment, not proof your production LLM already runs this on your tasks. Estimation-free sampling — generating without learning an explicit density estimator first — matters when you price prompt-only workflows against fine-tune or RAG.

Why readers outside the lab should care

Korean teams shipping HyperCLOVA, Solar, EXAONE, and other bilingual stacks often pitch in-context behavior to skip retraining on niche generation — category lists, template fills, structured outputs. This paper sets a theoretical ceiling: frozen transformers can run diffusion-like and energy-based samplers from examples alone. Ask vendors whether short semantic prompts show the two-stage spherical geometry measured here, or whether longer, noisier, Korean-mixed prompts need fine-tuning.

Overseas readers on Korea-hosted LLMs should note the gap between in-context regression (well studied) and in-context generation (formalized here). “Drop five Korean examples, get novel samples” bets on sampler-like behavior that is architecturally possible — not proven on every checkpoint. Safety teams: generation from user-supplied examples without weight updates makes prompt injection and example poisoning generative attack surfaces.

What builders and Korea-touching teams should watch

• Do map your prompt-only generation bets to this paper’s scope — short semantic categories first, not long-form Korean document synthesis.
• Don’t treat in-context learning marketing as proof your model runs diffusion-style samplers; ask for layer-wise evals or ablations on your task shape.
• Expect U-shaped layer geometry only where topic structure is clean; mixed-language or noisy prompts may break the spherical reference story.
• Re-check safety filters when users supply in-context examples — frozen-weight generation from examples is a different failure mode than fine-tune drift.
• Demand clarity on when Euler-update-style depth is needed versus shallow pattern completion on your latency budget.

Context

Read this as a bridge from in-context learning to in-context generation — diffusion and energy-based samplers inside transformer blocks, with a U-curve in semantic-topic prompts. Korelay’s frame: theory says frozen transformers can sample; your deployment must prove it on your categories, languages, and prompt lengths.

Source

Primary: arXiv:2609.08981 (abstract and framing cited; open the OA PDF for proofs and layer geometry figures). Do not republish the PDF.