Character.AI

Why video evaluation breaks the standard text playbook Generating video has become incredibly cheap, but evaluating its quality remains a massive bottleneck. While modern video generation models produce stunning single frames, they often fail at the narrative level. Traditional evaluation metrics like clip score work well for static images, but they miss temporal failures entirely. A video can look visually polished in a single frame but fall apart over time, featuring floating characters, doors opening and closing simultaneously, or characters changing appearance between shots. Video is fundamentally a storytelling medium, meaning evaluation must focus on narrative flow, physical consistency, and precise audio-visual synchronization. Using frontier LLMs as judges can help, but they are incredibly slow, highly sensitive to prompt variations, and expensive to run. If you prompt a model to tell you if a video is "consistent," the answers you get will vary wildly based on the phrasing of the system prompt. To solve this, developers need a repeatable benchmark that combines frame-level metrics with an automated judge that is constantly calibrated against human annotators. Why you should compare rather than score Assigning an absolute score from one to ten to a video fails because human taste is highly subjective. What one annotator rates as a seven, another might easily rate as a four. However, when presented with two clips, almost everyone can agree on which one tells a better story. Character.ai developer Maor Bril shifted the company's approach from absolute metrics to pairwise preference. By asking the model to compare Video A against Video B, the evaluation process achieves a much higher alignment with human taste. Training on pairs rather than arbitrary scales makes the model far more robust and easier to generalize. Training Qwen3-VL to catch video hallucinations Using massive frontier models is too slow and expensive for high-volume production. Instead, Bril's team trained a smaller Vision-Language Model, Qwen3-VL, to serve as the primary evaluator. They manufactured dataset variety by taking high-quality footage and deliberately corrupting it, teaching the model to identify specific bugs like floating limbs and broken physics. In early iterations, the model failed confidently by scoring the overall "vibe" and visual gloss of a video rather than the actual content. For instance, it rated a completely static shot as a 9.2 for camera work simply because the image quality looked professional. To fix this, they paired real footage with AI-generated footage, ensuring both had identical encoding to prevent the model from turning into a simple AI detector. This forced the model to score the actual visual narrative rather than digital artifacts. Placing the evaluation engine inside the generation loop To build reliable video systems, developers must catch drift as early as possible. Evaluating a completed five-minute video is incredibly expensive if the error occurred in the first ten seconds. By running the JudgeJudy evaluation harness directly inside the generation loop, agents can verify their own work at each step. If a six-second generation drifts, the agent immediately catches the error and regenerates that specific clip before stitching the final assets together. This agentic workflow ensures speed, maintains unit economics, and stops visual slop before it ever reaches the user.