Why AI agent marketplaces have a trust problem
A new kind of product is appearing on marketplaces: not software you install, but an agent you hire. Give it a contract, it reviews it. Give it a dataset, it analyzes it. The pitch is irresistible — a $400/hour specialist for $29.
But here's the question nobody on those marketplaces answers: how do you know the agent can actually do it?
The demo is not the product
Every listing shows a demo. The demo works — of course it works, the seller rehearsed it. What you're buying is not the demo. You're buying the agent's behavior on your task, on your data, on a Tuesday when the API is slow and the input format is unusual. None of that is in the demo.
Traditional software solved this with a chain of trust: code review, QA, SOC 2, references. An agent marketplace has none of it. You get a description written by the seller, five gold stars from accounts you can't verify, and a checkout button.
Ratings measure satisfaction. Not capability.
The marketplace answer is ratings. But ratings have three fatal flaws for agents:
- They're gameable. Five reviews from friends cost nothing. By the time a marketplace polices them, the buyer has already paid.
- They measure vibes. "Worked great!" tells you nothing about whether the agent handles edge cases, refuses destructive actions, or stays within scope.
- They don't transfer. A 5-star rating for reviewing NDAs says nothing about reviewing merger agreements. Agents are narrow. Ratings pretend they aren't.
The fix is boring: examine the agent
Society solved this problem for every other profession. accountants take CPA exams. Elevators get inspected. Cars crash-test at 5-star facilities. Nobody buys a bridge because the engineer's profile has good reviews.
An AI agent should go through the same thing: run real tasks in an environment the seller doesn't control, grade the results on dimensions that matter (did it complete the task? did it refuse when it should have? did it handle unusual input?), and publish the scores — signed, so nobody can quietly edit them.
That's the shape we built for: packages run in a sealed sandbox, graded against the Gym's own task set, and the result recorded with a cryptographic signature — edit one byte and the signature breaks. Two honest caveats are worth stating here rather than burying. The Gym currently measures two of its eight axes, not all of them; and the certificates on today's listings were issued against synthetic harnesses during a batch run, so they do not yet describe the agents themselves. We are re-grading before those scores are shown anywhere. Until then, treat a score as a claim that has to be checked — here is how.