How to keep track of an AI agent team
A one-page checklist for founders running several AI agents on real work.
Getting one agent to do a task is the easy part. Running five of them across a real business is different: work overlaps, things get "done" that aren't, and you become the bottleneck that approves everything. Here's a checklist for keeping that under control, whichever tools you use.
1. Who's on what
- Keep one list of every agent and what it owns. Name, tool, job, and the one person accountable for its output.
- Give every task an owner and a boundary. Anthropic's engineering team found that each subagent needs "an objective, an output format, guidance on the tools and sources to use, and clear task boundaries. Without detailed task descriptions, agents duplicate work, leave gaps, or fail to find necessary information."
- No two agents on the same task unless you meant it (for example, a second agent reviewing the first).
- Write down what an agent may not do, like spend money, email customers or delete files, next to what it may do.
2. Definition of done
- Decide what "done" means before the work starts. A file at a path, a merged change, a sent draft, a filled sheet.
- Done means something you can open. "I finished" in a chat isn't done. The output has to be a real artifact you can check.
- Name the check. Tests pass, numbers tie out, links work, the brief's three questions are answered.
- Set a stopping point. Anthropic notes it's common to include stopping conditions "such as a maximum number of iterations" to maintain control, so an agent doesn't loop forever on a task it can't finish.
3. Human review gates
- List the actions that always need a person: anything external (sending, posting, publishing), anything with money, anything that can't be undone.
- Review the output, not the transcript. Open the artifact and accept it or send it back with one clear reason.
- Have agents stop and ask when they're blocked. Agents "can then pause for human feedback at checkpoints or when encountering blockers" (Anthropic). Make that the expected behavior, not a failure.
- Keep review time honest. If you spend more time approving than the work saves, cut the number of gates or the number of agents.
4. Logs and records
- Every run leaves a record: what was asked, what the agent did, what it produced, and where it is.
- Keep records outside the chat. Save outputs to files or a shared folder so the next agent (or you, next week) reads the real thing. Anthropic suggests sending subagent output to a filesystem to minimize the "game of telephone."
- Make the agent's steps visible. "Prioritize transparency by explicitly showing the agent's planning steps." When something goes wrong, you'll want to know why, not just that it did.
- Date and version anything that matters. Hash or timestamp final files so you can tell which version you approved.
5. Handoffs
- Hand over files, not summaries. The receiving agent gets the actual output, the brief and the definition of done.
- Write a three-line handoff note: what's finished, what's open, what to watch out for.
- One handoff at a time. Small errors grow across steps. Anthropic notes that agents "are stateful and errors compound," so check output at each handoff instead of only at the end.
- Confirm receipt. The next agent restates the task before starting. If it can't, the handoff failed.
6. What needs your attention
- One queue for everything waiting on you: approvals, questions, blocked tasks. Not five inboxes.
- Sort it by consequence, not by arrival: external or irreversible first, then blocked work, then nice-to-know.
- Everything in the queue is a decision: approve, send back, or decide later with a date. No item sits there without one of those.
- Cap it. If the queue grows every day, you have too many agents or too few rules about what they can do alone.
7. Weekly review (30 minutes)
- What finished and was accepted? Count real outputs, not activity.
- What got sent back, and why? Fix the brief or the boundary that caused it.
- What's stuck? Unblock it, reassign it or kill it.
- What did it cost? Time you spent reviewing, plus tool and model spend.
- Retire or simplify one thing. Anthropic's advice is to find "the simplest solution possible, and only increasing complexity when needed." One fewer agent is often the best change you can make.
Start small
You don't need special software for most of this. A shared folder, a single task list and a written rule for what needs your sign-off cover a lot of it. Start with sections 1, 2 and 6 this week, and add the rest as your team grows.
We're building Plenarly to be the operating system for AI-run companies. If you already run AI agents on real work, request early access.