Agents in the Wild

Monte Carlo had every team build an AI agent in one week and logged 150+

Monte Carlo gave GTM, engineering, product, recruiting and legal one week to build agents with Claude. It catalogued 150+ agents and skills, including one that lifted an eval accuracy from 78% to 97%.

Organization
Monte Carlo
Industry
Data observability software
When
Mar 2026
Evidence
Company reported
Agents and tools
ClaudeNotionSlackGongLinearSalesforceLooker

Reported outcome

150+ agents, skills and projects catalogued in one week; one agent's accuracy on an internal eval rose from 78% to 97%; a 2-3 hour report now takes 15 minutes.

  • 150+ (company)
    Agents, skills and projects
  • 78% to 97% (company)
    Prompt eval accuracy
  • 2-3 hours to 15 minutes (company)
    Health report time

Caveats

Stated by the company that uses the agent (blog, earnings call, interview). Not independently audited.

  • The accuracy figure is measured on Monte Carlo's own eval set of 10 scenarios, not an independent benchmark.
  • Most projects have no hour figures. The post says only that some saved hours per week.
  • It is a company blog post and also promotes Monte Carlo's platform.
  • The author calls week one 'the foundation'. Some goals, like triggering onboarding agents from customer behavior, are aims, not results.

What the agents do

Teams across the company built their own agents. Examples from the post: a customer-signals agent that scans 72 hours of Slack, Gong, Linear, Gmail and G2 and writes a digest to Notion; a pipeline agent that runs Mondays at 6am on Salesforce data and labels segments Strong, Watch or Dark; and eight onboarding agents that produce a Day-30 health report.

Why this works

A shared registry in Notion and a shared skills repository let teams reuse each other's work. One data scientist packaged an "auto-research" method: Claude ran about 20 experiments against an eval set to improve another agent's prompt.

What to copy

  • Keep one registry of every agent so work is not repeated.
  • Build a small eval set first. Let the model tune against it.
  • Package a method that worked as a reusable skill.

Starter prompt

Written by us, not by Monte Carlo.

Here is an agent prompt: {current_prompt}
Here are test cases with correct answers: {eval_cases}

Run each case, report accuracy and recall, and list the cases it got wrong.
Propose one small change to the prompt. Re-run and compare.
Stop when accuracy stops improving for 3 tries. Show every change you tried.

Sources

  1. All in on AI — Monte Carlo, Mar 31, 2026

Added Oct 19, 2026. Spotted an error? Send a correction.

Get new use cases weekly

One short email. Sources included.

Related use cases