Every new AI model arrives with a table of scores. Higher numbers, bolder claims, a chart in which the new model beats its rivals by a comfortable margin. Anyone who has bought software knows to be a little suspicious of vendor-run tests, which is why a startup like Vals, backed by Andreessen Horowitz, is pitching itself as an independent referee. Its stated ambition, to become the gold standard for AI benchmarking, points at a real gap in the market.
What a benchmark is supposed to do
A benchmark is a standardized test. You give different models the same questions or tasks, grade the answers by the same rules and compare results. Done well, it answers the practical question every buyer has: which model should I use for this job? Done badly, it produces a leaderboard that rewards the wrong thing.
Why public leaderboards break down
- Contamination. If test questions appear on the web, they may end up in a model’s training data. The model then looks brilliant because it has effectively seen the answers.
- Saturation. Once top models score close to the maximum, the test no longer distinguishes between them.
- Overfitting. Developers tune their systems on popular tests, much as schools can drill students for an exam.
- Narrow tasks. Trivia and puzzles are easy to grade, but businesses care about messy work such as reading a contract or reconciling an invoice.
- Self-reporting. The company that built the model chooses which tests to show and how to prompt it.
What buyers actually need
A bank evaluating a model for document review does not care about its score on grade-school math. It wants to know how often the model misses a clause, how consistent it is across repeated runs, how much it costs per thousand pages and how it behaves when the input is unusual. That calls for tests built around domain tasks, graded by people who understand the work, and refreshed often enough that answers cannot be memorized.
This is the space where a specialized evaluation company can add value. Industries such as law, finance and healthcare have professional standards for what a correct answer looks like. Encoding those standards into repeatable tests is slow, expensive work, and most AI labs would rather not do it for their competitors.
Independence is the product
An evaluator has a credibility problem if it depends on the companies it grades. Think of credit rating agencies and the lesson from 2008: the party paying for the stamp influences the stamp. A trustworthy AI evaluator needs transparent methodology, clear disclosure of funding and the ability to publish unflattering results. Buyers should ask who pays for the evaluations, whether vendors can see test questions in advance and whether results are published even when they are bad.
What good evaluation looks like
- Hidden test sets. Part of the data stays private so models cannot train on it.
- Realistic prompts. Tasks resemble real workflows, including messy inputs and ambiguity.
- Multiple runs. Models are non-deterministic, so one attempt tells you little.
- Human review. Experts audit the grading, especially where automatic judges are used.
- Cost and speed. Accuracy alone is not enough; latency and price decide what is deployable.
- Versioning. Results are tied to a specific model version and date, since models change quietly.
Build your own tests anyway
No outside score can replace a small internal test. Collect fifty to a hundred real examples of the work you want done, with correct answers written by someone in your team. Run candidate models against them, record the results and keep the set private. Rerun whenever a vendor updates a model. It takes a few days to assemble and will tell you more about fit than any public ranking. Outside benchmarks are most useful as a starting filter to decide which models deserve that deeper test.
The business logic behind the startup
As companies deploy AI in regulated settings, they need evidence for auditors, insurers and customers. A credible third-party report becomes part of the compliance paperwork. That creates a market for evaluation services, similar to how security firms test software for vulnerabilities. Investors see a role for an independent name that everyone recognizes, and they are betting that trust, once earned, is hard to displace.
What to watch
If Vals or any rival wants the label of gold standard, the proof will be in its transparency and consistency over time. Watch whether results are reproducible, whether methodology is documented in enough detail for outsiders to critique, and whether labs cooperate rather than dispute every score. The best outcome for the industry is not one winner but a healthy ecosystem of evaluators, each honest about what its tests can and cannot tell you.
Until then, treat every leaderboard as a clue, not a verdict, and let your own data have the final word.
