Snorkel AI evaluation workflows review expert-generated labels for specialized domains. Their Multi-Agent Question-Answer Validator alone processes thousands of results every week. Ultimately, it verifies annotations used to create datasets, Reinforcement Learning from Human Feedback (RLHF) pipelines, and reasoning benchmarks.
However, when something went wrong, finding where and why felt like archaeology. The team fumbled through a fragmented debugging workflow and raw logs that were scattered across databases and Amazon S3. They manually reconstructed agent execution paths, and their decision trees had no visualization. Also, aggregate-only metrics obscured issues with individual trajectories.
“Before AI Gateway, Snorkel debugged LLM applications with a fragmented patchwork of logs in S3, DB queries and manual visualization. The clean AI Gateway UI allows us to identify, debug, and rebuild agents with confidence.”
Shae Selix
Staff Engineer – TLM, Snorkel AI
“With AI Gateway, every agent, subagent and tool call is visible in one hierarchy. You can see how decisions branch and converge. Seeing the full execution tree in one view completely changed how we debug.”
Shae Selix
Staff Engineer – TLM, Snorkel AI
Prisma AIRS AI Gateway stood out for its:
- Visual hierarchy showing every agent call, subagent and tool invocation as a nested tree.
- Individual prompt-response inspection at any node in the execution tree.
- Custom tagging and filtering to identify patterns across traces.
- Side-by-side trace comparison to understand why good and bad cases diverge.
Making agent execution visible
By using AI Gateway, the team could finally see what was happening inside their agents. Every call, every subagent, every tool invocation appeared as a nested tree. They no longer had to reconstruct execution paths from scattered logs.
The difference was immediate. In the failing trace, they could see the moment things went wrong. The Planner would receive a chart with no question, search the web for context, and spiral into 38 calls chasing information that didn’t exist. Each step was visible in the hierarchy. Successful traces with AI Gateway told a different story. AI Gateway demonstrated a clean path from Planner to Code Agent to Verdict, with the right tools firing in the right order. Comparing the two methods side by side revealed where executions diverged.
The impact
After implementing Prisma AIRS AI Gateway for agent debugging, Snorkel AI saw tangible improvements across their evaluation workflows.
- 20% accuracy boost in agent evaluations. The team catches and fixes edge cases that were previously invisible in aggregate metrics.
- 2x faster issue detection in development. Problems surface immediately in traces rather than requiring hours of log archaeology.
- Faster iteration cycles. What used to take hours of manual reconstruction now takes minutes of visual inspection.
share this story
