How to know if an AI agent did the right thing
Traces, logs, and passing tests only show the steps an agent took, not whether the final outcome is actually correct. A run can finish cleanly while missing the intended goal or suddenly generating unexpected code. Because finished and succeeded are not the same thing, developers are left needing another agent just to check the numbers.
What the numbers show
Counted from the threads quoted below. How we count
- It is not one crowd: 2 people on Hacker News and 1 on Reddit ran into the same thing.
What people tried
Every workaround mentioned in the threads below. We haven’t tested any of them — and nobody here is claiming they worked.
Found another way around this?
Share your steps and trade-offs so fellow engineers don't have to reinvent the wheel. No account required.
- 1Using a second, independent AI agent to recompute and verify the results.Did this help you?
- 2Experimenting with extracting intents from the agent and correlating among them.Did this help you?
- 3Applying traditional software unit tests and manual visual inspections to the final output.Did this help you?
- 4Decode the Base64 string into a file without executing it using `base64 -d` and inspect the contents with `less` before running.Did this help you?
In their words
Unedited, grouped by where they were said, most upvoted first within each place, each linked to the thread it came from.
“Traces and logs tell me what the agent did: every tool call, retry, token, and step. That's useful when something breaks. What I'm less sure about is whether that tells me the agent actually did the right thing.”source ↗
“A run can look clean and still miss what I wanted. Or it can "finish" without the outcome really being done. So finished and succeeded aren't the same thing.”source ↗
“All the first agents tests had passed. The traces tell me what it did, but I still need another agent to check the numbers.”source ↗
“I asked Grok/Cursor for several fixes on my WHM/Cpanel via terminal command and one of the outputs was sudden base64 code. Scary AF?”source ↗
Where this came up
People with this problem also raised
- 3How to verify vendor AI claims
- 3Why do I still have to check the UI after using AI?
- 5How to audit employee AI tools and browser extensions
- 3How do you test if AI-generated code actually works for your business?
- 5Mechanic changed battery CCA specification on test report
- 4Where can I find real-world company AI policies and best practices?