A model can write flawless legal prose, cite a convincing paper, and still accept a question that means absolutely nothing. That failure is easy to miss because the answer looks useful. BullshitBench v2 is built to catch it.

The open benchmark's July 25 update adds a new 100-question set and a public leaderboard covering 190 model and reasoning configurations. The questions are grammatically clean but semantically broken. They sound like work. They are not.

BullshitBench v2 benchmark logo and project mark

The test is about knowing when to stop

Most language-model evaluations reward an answer. BullshitBench rewards the opposite behavior when the prompt has a false premise. A response gets the green label when it clearly rejects the setup. Amber means it notices a problem but continues to play along. Red means it accepts the nonsense and writes an answer as if the question were legitimate.

That distinction matters more than it sounds. A chatbot that refuses a nonsense question can feel unhelpful in a demo. Put the same behavior inside a compliance workflow, a medical intake tool, or a research agent, and the refusal may be the only thing preventing fabricated work from entering the system.

The v2 set spreads 100 prompts across software, finance, legal, medical, and physics. Software gets 40 questions. The other four areas get 15 each. The authors also tag 13 failure techniques, including nonexistent frameworks, misapplied mechanisms, nested nonsense, and what they call the specificity trap. That last one is especially nasty: a prompt piles on precise-looking details until the model assumes the underlying concept must be real.

The scoring avoids relying on one judge. Three judge models, Claude Sonnet 4.6, GPT-5.2, and Gemini 3.1 Pro, grade every response. The benchmark takes their mean. The maintainers report roughly 80 percent agreement between judges, which is decent for a task involving language and context rather than a clean right-or-wrong answer.

The leaderboard has an uncomfortable shape

The July 24 snapshot mirrored by BenchLM contains 190 rows. Claude Opus 4.8 at its lowest reasoning setting leads the clear-pushback view at 95 percent. The same model at xhigh reaches 94 percent, while Claude Sonnet 4.6 at high reasoning reaches 91 percent. In other words, spending more reasoning tokens does not automatically make a model better at recognizing nonsense. The no-thinking configuration actually wins the public snapshot.

The wider results are less flattering. An earlier BullshitBench analysis found detection rates as low as 2 percent and as high as 91 percent across 88 configurations. The gap is not explained by model size alone. Qwen 3.5, with 397 billion total parameters and 17 billion active, reached 78 percent, well above the size trend. DeepSeek V3.2, listed at 685 billion parameters, scored 13 percent. ERNIE 4.5, listed at 300 billion, scored 4 percent.

Those numbers are a useful antidote to the usual parameter-count race. Bigger models have more room to store facts, but that does not mean they know when a sentence has glued together unrelated facts. A model can be extremely capable at retrieval and still be too agreeable when the prompt is dressed up as an expert request.

The project also tracks whether newer models improve, whether reasoning effort helps, and whether active parameters matter in sparse models. Those charts are more useful than a single rank because they let you ask a practical question: what kind of model behavior are you buying when you turn on a more expensive mode?

My guess is that many teams will get the answer they do not want. Reasoning is often treated as a universal repair layer. BullshitBench suggests it can be a style change rather than a reliability fix. A longer chain of thought may produce a more elaborate answer to a broken premise. That is worse than a short wrong answer because it gives reviewers more text to trust.

What the prompts reveal about model training

The benchmark's prompts are not random gibberish. They are plausible combinations of real vocabulary. One example asks for the recommended cadence of a "bilateral indemnity regression" when a contract portfolio spans conflicting legal systems. Each phrase sounds professional. The whole question has no stable meaning.

That is close to the kind of prompt people actually send to enterprise systems. Internal documents contain invented project names, half-remembered regulations, stale acronyms, and confident mistakes from earlier drafts. A useful model needs to separate a legitimate hard question from a polished pile of words.

This is where standard factuality scores fall short. A model can answer a well-formed question correctly while failing to challenge an invalid assumption. Retrieval can make the problem worse if the system finds nearby documents and uses them as decoration. The output then looks researched, even though the original question was never valid.

BullshitBench is not a complete reliability test. The prompt set is public, so models can eventually memorize it. The judge panel is also made of models, which creates a second layer of shared blind spots. And a clear refusal is not always the right response: users sometimes use deliberately strange language to describe a real internal process.

Still, the benchmark asks a question that most leaderboards ignore. Can the model decline to perform expertise when the premise does not cash out? That is a better test of an assistant than another collection of trivia questions.

For builders, the practical move is simple. Add a handful of nonsense prompts to your own evaluation set, drawn from the documents and terminology your system sees every day. Score clear pushback separately from generic refusal. Then inspect the red answers by domain. A model that does fine on software nonsense but accepts fabricated legal standards needs a routing rule, not another glossy capability score.

BullshitBench v2 will not tell you which model is smartest. It tells you which ones are willing to say, "I do not think this premise is real," before spending your budget on a confident explanation. That is a much smaller claim. It is also one most production systems should measure.

Sources