---
title: "Orchestras, not factories: six rules for building with AI agents, and what the evidence says about each"
description: "A startup founder's six principles for building with AI coding agents, checked one by one against the research, with his product pitch kept separate."
publishedAt: 2026-09-27
updatedAt: 2026-09-27
author: Ena Pragma
url: https://enapragma.co/field-notes/orchestras-not-factories-six-rules-for-building-with-ai-agents-and-what-the-evidence-says
tags: ["human-in-the-loop-ai", "ai-operations", "ai-verification"]
---

Charlie Holtz co-founded Conductor, a Mac app for running several AI coding agents at once. At the AI Engineer World's Fair in San Francisco, which ran from June 29 to July 2, 2026, he gave a talk called ["Orchestras, Not Factories: How the Fastest Builders Work"](https://www.youtube.com/watch?v=TRfzFJCJ7ZE), posted to YouTube on September 27. The day's talk track was the "software factory," and his final principle rejects it: "The whole like talk track today is about software factories. And I honestly kind of hate the term."

His six principles are practical, and most of them are good advice. But each is a claim about what works, and some of them can be checked. Below is what he recommends, what the evidence says about each one, and what Conductor itself has actually shipped, kept separate because a founder's principles and his product pitch deserve different kinds of scrutiny.

## The six principles, in his words

1. **Stay near the frontier.** Try new tools and workflows "basically the day they come out."
2. **Don't try to beat the market.** Before investing heavily in a new workflow, ask "why isn't this workflow the default?" If it works for everyone, the big AI labs will probably build it in.
3. **Create slop-free zones.** "a part of the code base or a part of the app that requires really strict human review." At Conductor, "any change to migrations file requires" a human to review it. He says they have had to rewrite the whole app "a couple of times" because they were not careful about this.
4. **Feed the beast.** An internal agent Conductor calls the CIA saves every Slack message, every bug report users post in Discord, and recordings of meetings into one central database, so agents can query how the company works.
5. **Free-range agents.** Give agents a sandbox off your laptop, where "they're not going to get shut down when you close your laptop lid."
6. **Orchestras, not factories.** The human should be "in front of an orchestra like waving my baton," directing agents, not managing an assembly line.

## What the evidence says, principle by principle

**Stay near the frontier: sensible, but your sense of speed is not evidence.** In a randomized trial of experienced open-source developers in early 2025, [METR found](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) that "when developers use AI tools, they take 19% longer than without." They had expected AI "to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%." METR's [2026 follow-up](https://metr.org/blog/2026-02-24-uplift-update/) says that, based on conversations with participants, developers are likely sped up more in early 2026 than a year earlier. But it calls its data "only very weak evidence" for the size of that increase, partly because developers who did not want to work without AI stayed out of the study. It calls its own estimate likely "a lower-bound." The honest reading: newer tools may help, and your own feeling about whether they help is not a reliable measure. Holtz's claim that relying on your network leaves you "three to six months behind" is his opinion; the talk gives no data for it.

**Don't try to beat the market: a reasonable personal rule, not a finding.** This is about how much to tune your own setup, and it is a judgment call. His exception is "real alpha," which he defines as information "about either your users or your code base that the models might not know about." For a business that is a useful test: spend your effort where you know something the tools do not.

**Slop-free zones: the risk is real, and a required review is not the same as a review that happens.** The case for protecting some code is strong. On [Veracode's](https://www.veracode.com/blog/spring-2026-genai-code-security/) benchmark (Veracode sells security tooling), AI models now "achieve syntax correctness rates exceeding 95%," but "security pass rates remain stubbornly stuck at approximately 55%" when no security guidance is given in the prompt. Its best-scoring reasoning models reached 70% to 72%. The case for what goes inside the zone is more mixed. A [study of context files](https://arxiv.org/abs/2602.11988), the instruction files Holtz says his team works hard on, found that "providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average." The authors conclude they "are useful for specifying non-standard coding practices." In our reading, that is close to what a protected zone is for.

The sharper point comes from medicine, as an analogy. When the WHO surgical checklist was trialed, [deaths after major operations fell](https://www.sciencedaily.com/releases/2009/01/090114172304.htm) from 1.5% to 0.8%. After Ontario required its hospitals to publicly report checklist use, hospitals reported using the checklist in "never less than 98 per cent" of operations. Yet a study of 101 hospitals found [adjusted mortality](https://www.ices.on.ca/news-releases/study-shows-introduction-of-surgical-safety-checklists-in-ontario-have-not-reduced-post-operative-mortality-or-complications/) of 0.71% before and 0.65% after, and ICES reported that the checklists "did not improve operative mortality." (Ontario's starting death rate was already lower, though the two studies counted deaths differently.) One of the lead authors suggests a bigger effect "might occur with more intensive team training or better monitoring of compliance." Our reading: a rule that says a human must review the migrations file tells you a review was required, and even a reported review may not be a careful one.

**Feed the beast: right, if the beast knows who may read what.** Keeping company knowledge somewhere agents can search is sound. Dumping all of it into an agent's context is not: Anthropic's own guidance on [context engineering](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) is to find "the smallest possible set of high-signal tokens." Holtz describes a database the agent queries, which fits that. The part the talk skips is access. Simon Willison's ["lethal trifecta"](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/) is an agent that combines private data, untrusted content and a way to send data out. Internal Slack messages are private data, and bug reports posted by outside users are untrusted content. In 2024, researchers showed Slack AI could be tricked into [leaking data from private channels](https://www.promptarmor.com/resources/data-exfiltration-from-slack-ai-via-indirect-prompt-injection) through a message planted in a public one.

**Free-range agents: useful, and your data gets a new home.** Cloud agents keep working when your laptop is closed. They also change who holds your data. Conductor's own [pricing page](https://www.conductor.build/pricing) says that with local Conductor, "session data is stored on your device," while "chat messages sent in cloud workspaces are stored on Conductor's servers." That may be fine for you. It is a decision to make on purpose, not a default to accept.

**Orchestras, not factories: the better line is who keeps the judgment.** Holtz's objection is to factories that remove the human. Real factories do not have to. Toyota's production system is built on jidoka, which [Toyota's own site](https://global.toyota/en/company/vision-and-philosophy/production-system/index.html) says "can be loosely translated as 'automation with a human touch'": machines stop when something is wrong, and operators can stop the line. The same page is blunt about who improves the system: machines "can't evolve any further on their own. Only humans can implement kaizen." The useful distinction is not orchestra versus factory. It is whether people keep the judgment calls, which is what Holtz is really arguing for.

## What Conductor has actually shipped

Kept separate from the principles, because these are product claims:

- **Cloud collaboration.** At the conference, Holtz said "we're rolling this out to all conductor users this week." Conductor's changelog shows a "Conductor collaboration alpha" on [July 9](https://www.conductor.build/changelog/0.74.0-lots-of-little-things), early-access multiplayer on [July 23](https://www.conductor.build/changelog/0.77.0-early-access-multiplayer-api-background-tasks-performance), and "Conductor Cloud is here!" on [July 30](https://www.conductor.build/changelog/0.78.0-introducing-conductor-cloud), which also marked "the previous Cloud settings section as Legacy." Multiplayer and Conductor Cloud are Pro features; the free plan is "Local workspaces on your Mac."
- **Controlling agents from your phone or Slack.** In the demo, Holtz texted his own agent, "Lord Crandon," which "has access to the conductor API and so can kick off work itself." That is a do-it-yourself route through Conductor's API, which the company promotes for this: its July 23 release invites users to "build Slackbots that kick off agent work." The [API](https://www.conductor.build/docs/api) is in beta and needs a Conductor Pro account. Pro also includes a "Conductor mobile app." Texting or Slack control, as in the demo, is something you build yourself on the API.

## The throughline

**What.** Six working rules from someone who builds with AI agents all day: stay current without chasing every trend, protect the parts that must not break, give agents the company's knowledge, let them run in the cloud, and keep a human conducting.

**Who.** The founder of a company that sells a tool for exactly this. That does not make the advice wrong. It is a reason to separate the principles from the product.

**How.** Our reading of the evidence: the advice is mostly sound, and two of the six principles are really about keeping a human check. The evidence suggests a required check and a performed one are different things, and a developer's feeling of speed is not a measurement.

**When.** Now, for any team letting agents write code or handle company information.

**Why.** Because agents make it cheap to produce work, the scarce thing becomes someone who checks it. The orchestra needs a second listener.

## What this means for a business using AI agents

This is our advice, built from the sources above, not a tested method.

- **Name your slop-free zones, and check that the review in them happens.** A required approval that nobody reads is a false sense of safety. Look at a sample of approvals now and then.
- **Measure speed; do not trust how fast it feels.** Time a handful of real tasks with and without the new tool before committing to it.
- **Before you feed the beast, decide who may read what.** Keep private material and untrusted input apart, and do not give the same agent both plus a way to send messages out.
- **Know where your data goes when agents run in the cloud.** Read the vendor's data page before you switch it on.
- **Keep a person on the calls that matter,** and give them what they need to judge, not just a button to approve.

## What Ena Pragma takes from this

Two of the core values in our operating plan make the same argument as the strongest part of this talk: "Humans where judgment matters. Agents may prepare, execute, monitor, and escalate. People retain consequential judgment." And "Receipts over optimism": work is complete "when the evidence can be inspected." In practice that means a second listener. Before this note was published, a separate reviewing agent checked its quotes and claims against the sources. It found real problems, including hedges we had dropped, and we fixed them.

*How this was researched:* our AI research agent searched for sources on the talk, Conductor and the surrounding research and wrote a graded evidence sheet. We then re-read the sources quoted here directly on September 27, 2026: the talk's captions, the conference's event page, Conductor's changelog, pricing and API pages, METR's two write-ups, the context-file study's abstract, Veracode's report, the checklist summaries, and the Anthropic, Willison, PromptArmor and Toyota pages. We did not use Conductor ourselves.

## Sources

- [Charlie Holtz, "Orchestras, Not Factories: How the Fastest Builders Work," AI Engineer (YouTube)](https://www.youtube.com/watch?v=TRfzFJCJ7ZE)
- [METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity"](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/)
- [AI Engineer World's Fair 2026, event details](https://www.ai.engineer/worldsfair/llms.md)
- [METR, 2026 update](https://metr.org/blog/2026-02-24-uplift-update/)
- [Veracode, Spring 2026 GenAI code security update](https://www.veracode.com/blog/spring-2026-genai-code-security/)
- [Context-file study (arXiv 2602.11988)](https://arxiv.org/abs/2602.11988)
- [ScienceDaily, on the WHO surgical safety checklist trial (NEJM 2009)](https://www.sciencedaily.com/releases/2009/01/090114172304.htm)
- [ICES, on surgical safety checklists in Ontario (NEJM 2014)](https://www.ices.on.ca/news-releases/study-shows-introduction-of-surgical-safety-checklists-in-ontario-have-not-reduced-post-operative-mortality-or-complications/)
- [Anthropic, "Effective context engineering for AI agents"](https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents)
- [Simon Willison, "The lethal trifecta for AI agents"](https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/)
- [PromptArmor, "Data Exfiltration from Slack AI via indirect prompt injection"](https://www.promptarmor.com/resources/data-exfiltration-from-slack-ai-via-indirect-prompt-injection)
- [Toyota Production System](https://global.toyota/en/company/vision-and-philosophy/production-system/index.html)
- Conductor: [pricing](https://www.conductor.build/pricing), [API docs](https://www.conductor.build/docs/api), [changelog 0.74.0](https://www.conductor.build/changelog/0.74.0-lots-of-little-things), [changelog 0.77.0](https://www.conductor.build/changelog/0.77.0-early-access-multiplayer-api-background-tasks-performance), [changelog 0.78.0](https://www.conductor.build/changelog/0.78.0-introducing-conductor-cloud)
- Earlier Field Notes: [Shopify's AI coding gates are built so the agent can't grade its own work](https://enapragma.co/field-notes/shopifys-ai-coding-gates-are-built-so-the-agent-cant-grade-its-own-work)