Transformer

The current generative AI landscape is often reduced to a battle of parameter counts and compute budgets, but for those building the actual engines of creation, the reality is far more nuanced. Sander Dieleman, a research scientist at Google DeepMind, offers a look under the hood at the specific mechanics that enable VEO and other generative media tools to turn text into high-fidelity imagery. While Large Language Models (LLMs) rely heavily on auto-regression, the world of pixels and frames has converged on diffusion—a paradigm shift that treats generation as a process of systematic denoising rather than sequential prediction. The curation paradox and latent bottlenecks In academia, research incentives are historically aligned with beating benchmarks on static, standardized datasets. However, Dieleman argues that in the era of large-scale generative models, this approach is fundamentally flawed. Data curation has become the "secret sauce" of high-performance models, yet it remains one of the least discussed topics in peer-reviewed literature. The shift from using whatever is available to meticulously filtering and labeling data is often more impactful than tweaking a model’s hyperparameters. This unlearning process—moving away from fixed datasets toward active curation—is what separates experimental toys from production-grade tools. Beyond just the quality of the data, there is the massive technical hurdle of representation. Shoving raw pixels directly into a transformer is a recipe for memory exhaustion. A 30-second 1080p video at 30 frames per second consumes gigabytes of memory as a single training example. To solve this, developers use Autoencoders to create a latent representation. Unlike standard video codecs like JPEG or H.265, which prioritize file size at the expense of data structure, these learned compressors maintain the topological grid of the image while stripping away fine-grained textures. This allows the diffusion model to operate in a compressed space, where it can manage the immense complexity of video without overwhelming the hardware. Diffusion as spectral autoregression To understand why diffusion works where other methods fail, Dieleman points toward Fourier Analysis. Natural images and videos follow a power law in their frequency spectra—most of the energy lives in the low frequencies (global structure), while high-frequency components (fine detail) have less energy. When a model adds noise to an image, it effectively acts as a low-pass filter, drowning out the fine details first. By reversing this process, a diffusion model essentially performs "spectral autoregression." It doesn't predict the next pixel; it predicts the next layer of frequency. The sampler starts by sketching out the coarse semantic boundaries of the subject—the silhouette of a bunny, for instance—and gradually fills in the high-frequency whiskers and fur textures. This coarse-to-fine progression is naturally aligned with how visual information is structured, providing a more intuitive generation path than the arbitrary sequence orders required by traditional autoregressive models. The guidance trick and architectural compromises Perhaps the most significant differentiator between modern diffusion models and their predecessors is "classifier-free guidance." This technique allows a model to punch well above its weight class, delivering high-fidelity results even with relatively modest parameter counts. During sampling, the model makes two predictions: one based on a text prompt and one without. By amplifying the difference between these two predictions, the system pushes the output toward the user’s intent with startling force. This increases sample quality at the cost of diversity, but for most users, a single high-quality image is preferable to a wide variety of mediocre ones. On the architectural side, the industry is seeing a convergence toward Transformer backends. While early models like Stable Diffusion relied on U-Net architectures, the massive body of knowledge regarding how to scale Transformers has made them the default choice. This allows developers to borrow techniques directly from the LLM world, such as advanced model sharding and parallelism using tools like JAX. For video, this often manifests as a hybrid approach: models may jointly denoise entire 3D volumes of video or use a temporal autoregressive setup to generate frame-by-frame while diffusing the content within those frames. The future of steering and control As these models mature, the industry is moving beyond the simple text box. Users increasingly demand high-level semantic control, such as manipulating camera motion, event timing, or maintaining character consistency through reference-based generation. These "control signals" present a new challenge: how to introduce specific constraints without breaking the pre-trained distribution. The prevailing strategy involves post-training phases where the model is fine-tuned on preference data or specific conditioning signals. This is where tools like Genie excel, acting as world models that can be steered by interactive inputs. Whether through Distillation to reduce the number of sampling steps—traditionally 50 or more—down to a handful via Consistency Models, or through Direct Preference Optimization, the goal is to make these digital artists as responsive as they are imaginative.