ena pragma
·16 min read·agent-runtime · agent-cost · ai-verification · research-method

AWS says 75,000 AI agents fit in a Raspberry Pi. The number is a division, and the cost to watch is the cold restart, not the waiting

AWS's Infinity runtime parks waiting agents so they cost almost nothing. We read what the benchmark measured, what it left out, and give a test you can run on your own agents.

Contents

The short version. On October 23, 2026, AWS released Infinity, a runtime that lets an AI agent go to sleep while it waits and wake up when the answer arrives. The headline is "75,000 concurrent agents in the 8 GB of a Raspberry Pi 5." That figure was not measured on a Pi; the run was on a server processor. It is a division: 8 GB divided by the roughly 100 KB each of 80,000 idle agents used on a server, with a simulated model. What it measures is how little it costs to store a parked agent's chat history, and AWS deserves credit for showing its method so plainly. It does not tell you how many agents can think at once. It also does not cover the cost that matters most to anyone paying a model bill: whenever an agent resumes, the model is given its whole history again, and if the provider's cache has expired because the wait was long, that history is billed at full price. That happens whether the agent was parked or held open. So the cost to watch is the cold restart after a long wait, not the waiting itself. We have not measured how large that is in practice. This note explains what Infinity is, the pattern behind it, what the sources support and do not, and a short test for whether any of this applies to you. AWS calls it "an early research release," and that is how it should be read.

#What it is, in plain words

Most AI agents work in a loop: ask the model what to do, run a tool (build the code, search the web, wait for a person to approve), get the result, ask the model again. While the tool runs, the agent usually sits there holding a process and memory, doing nothing. The AWS post puts it this way: "While the agent waits, it holds a process and its memory, and it cannot respond to events that arrive in the meantime." Its examples are a build that runs twenty minutes, a triage agent that waits hours for the next pull request, and an assistant that waits until the next morning for an approval (AWS Open Source Blog, Shadaj Laddad, October 23, 2026).

Infinity's answer is to stop holding anything. When the agent calls a tool, the runtime saves the conversation, lets the process end, and brings the agent back when the result arrives. The post borrows an idea from programming languages, where an await lets a function pause and resume. It is open source (Apache 2.0), written in Rust, and part of Hydro, a project between AWS and UC Berkeley. Its repository was created April 13, 2026, and the first tagged release (v0.1.0) is dated September 3, 2026 (GitHub API, read November 4, 2026).

#How it works

  1. The agent runs in short slices. The post: "In a slice, the runtime loads the agent's history from a pluggable store, runs one model completion, persists the output, and dispatches at most one tool call. The slice then ends." A tool call is the pause point.
  2. Between slices an agent is only its saved history. There is no process, no open connection. Anything that can read the store can continue the agent.
  3. Everything that wakes an agent is a message on a queue. Each agent has its own ordered line, and different agents run side by side. "The runtime guarantees that messages for the same thread are processed in order, one slice at a time, while slices for different threads run in parallel."
  4. On AWS it runs with no server at all. The reference setup uses a queue message to start a short function that runs one slice and exits, with history in a database. A tool that takes three days costs no running compute while it takes.
  5. News that arrives while the agent waits is dressed up as a tool call. Model APIs insist that each tool call gets exactly one result, so the runtime inserts a call the model did not make, followed by a result holding the event. By default, each event is handled by a temporary child agent that inherits the parent's context and reports back only if something matters.

#What the 75,000 measured

QuestionAnswer from the sources
What was run?80,000 agents, 3.2 million completions in total, on "a single thread of an AMD EPYC 9R14 with in-memory stores" (a server processor). Each agent ran twenty turns against "a simulated model."
What was measured?Memory after the agents finished and went idle: "about 100 KB per idle agent." The benchmark's own code comment says it samples memory when "every agent is idle," so the figure is the cost of a parked history and "nothing else."
Where does 75,000 come from?A division. The post: "At this rate, about 75,000 agents fit in 8 GB, which is the memory of a Raspberry Pi 5." The project's site shows the raw run: "80,000 agents in 8.4 GB," about 103 KB each.
Was it run on a Pi?Not that we can find. An independent write-up calls it "a capacity extrapolation from a simulated workload, not 75,000 real model inferences running on a Pi."
What was not measured?Throughput, delay, how many model calls can happen at once, or any real model.

Two more things shape how to read it. First, most of the 100 KB is the chat history, so it grows with history. The test histories were short: 3.2 million completions over 80,000 agents is 40 each, at about 1.2 KB, so roughly 48 KB of completions, before tool results (our arithmetic). A working agent with a 200,000 token context holds on the order of 1 MB of text at about four characters per token (our arithmetic and rule of thumb). At that size, 8 GB holds roughly 8,000 parked agents, not 75,000 (our arithmetic, ignoring everything else on the machine). Second, memory for idle agents is only one cost. Model calls cost money on every resume, and the next section shows how a long wait can make that bill larger (our reading, not a measured comparison).

The independent reading we found is one person's. MushroomDAO ran four of the project's own tests on October 25, 2026 (mock model, in-memory store) and wrote: "We did not reproduce this benchmark." We found no one who has run the benchmark, no named production user, and no discussion of the post that adds a measurement. A search tool is a lead, not proof, so that is "we found none," not "none exists."

#Why the cost to watch is the cold restart

Here is the part the post does not cover. In the post, the only use of the word "costs" is about memory. It does not discuss what a resume costs in model fees, or rate limits. A tech-news write-up of the release says the same thing from outside: "suspension saves waiting-time compute, not model call fees."

Every time an agent resumes, parked or held open, the model has to be given the whole history again, because a model API does not remember. Providers give a large discount on text they have seen very recently. On Anthropic's documentation (the page shows no date; we read it on November 4, 2026), the cache lasts "5 minutes by default," with a 1-hour option "at additional cost," and a cache hit on Claude Sonnet 5.5 is "priced at 0.05x the base input price." The same page lists, per million tokens for Sonnet 5.5: $2 input, $2.50 to write a 5-minute cache entry, $0.10 for a hit.

Take a history of 194,000 tokens, which is about what one of our own requests re-reads on average (our usage report for a five-person week, recorded November 3, 2026; more below). We use it as an example size. At those list prices:

  • Warm request (cache hit): 194,000 tokens at $0.10 per million, about $0.02.
  • Cold request (cache expired, history written back to the cache): at $2.50 per million, about $0.49.
  • Cold request, no caching: at $2 per million, about $0.39.

So a cold request costs about $0.47 more than a warm one, about 25 times on that input (our arithmetic, list price, from an undated page; prices change, so read it as a ratio, not a quote). If the agent waits three days, the next request is cold, parked or held open. If it waits under five minutes, the cache stays warm either way. Infinity's Amazon Bedrock provider does put a cache marker on the last message (we read the source), but the post does not say what that buys, and we did not verify how long the cache lasts there.

What parking saves is a process that sat idle. What parking cannot give you is a reason for the cache to stay warm across a long wait. Which number is larger depends on your waits; we have not measured ours.

#What it signals, as far as the sources go

  • The waiting problem is real enough that several platforms ship an answer. Cloudflare's docs, last updated October 27, 2026, say "Agents are durable identities, not always-on processes," and describe the cost of the alternative: "An agent that is 99% dormant and 1% active still costs you 100% of a server" (Cloudflare Agents docs). AWS's own AgentCore added runtime instances for sessions of "up to 14 days" on August 6, 2026, while the default is designed for "up to 8 hours" (AWS What's New).
  • Three ways to wait are on the table. Hold the session open (AgentCore). Let the caller poll: the Model Context Protocol's July 28, 2026 release candidate lets a server answer a tool call with a task handle that "the client drives" with tasks/get (MCP blog). Or push and sleep: the tool calls back when finished, which is Infinity's Reactive Agent Protocol (RAP) and is close to Cloudflare's approach. No source we found says which wins.
  • AWS sells a held-open product and published a suspend-and-resume research project within three months of each other. That is a sign of an open design question, not a settled one.
  • The authors list what they have not studied: models "are mostly trained on traces in which tool calls are synchronous," so how interleaved events change an agent's decisions is unknown; scheduling is first come, first served; compaction (shortening history) across long-running agent trees is unstudied; and whether an event is authentic is unsolved.

Maturity signals from the repository on November 4, 2026: 662 stars, 76 forks, 60 open issues, version 0.2.0 released October 22. We read these as early and active, not proven.

#The shape, apart from the product

The pattern is worth knowing even if you never touch the code.

  • An agent is a log plus a mailbox, not a process. If everything needed to continue is in the saved history, any worker can continue it.
  • Every blocking point is a yield. Start the job, record it, let go. The answer comes back later as an ordinary message.
  • One ordered line per agent, many agents in parallel. Order matters inside one agent, since two writers would overwrite each other's turns. Across agents, nothing needs coordinating.
  • Save the turn before you dispatch the call. Then the call is on record before its answer can arrive. Redelivered messages are skipped by tracking which IDs were processed.
  • Show the model events in a shape it already understands. A fake tool call plus result keeps the conversation valid.
  • Handle noisy events in a disposable child. The parent hears only what matters. But the child inherits the parent's context, so each child re-reads the parent's history. It pays for itself only if it keeps the parent's history much shorter than the children re-read.

One limit worth stating: the runtime skips duplicate messages of its own, but that "does not guarantee exactly-once payments or mail across systems," as the independent write-up puts it. A tool that sends an email or charges a card can still run twice if it is retried.

#A test you can apply

Answer in order. Stop when an answer rules parking out.

  1. Is waiting most of the agent's life? Waiting share = time spent waiting on tools, people or events, divided by total time. Below about 50%, parking saves little. The 50% line is our judgment, not a figure from any source.
  2. Does resident memory actually bind? Multiply the number of agents you want live (N) by the memory each holds (R) and compare with what you have. If an agent is a bare saved history, R is roughly the history size: about 100 KB in the Infinity test, about 1 MB at 200,000 tokens of context. If your agent is a full program or container, R is that whole thing, and you can read it off your system's process list. If N times R is a small slice of your memory, the runtime buys you little; you can still want the pattern.
  3. How long are the waits against the cache lifetime? Under 5 minutes the cache stays warm either way. Over it, the next request is cold whether the agent is parked or held open, at up to about 25 times a warm one on input at the list prices above. The 1-hour cache costs more to write but can fit your gaps. Run the arithmetic on your own history size.
  4. Can you add idempotency keys to every tool that has a side effect? (A key a service uses to ignore a repeated request.) If not, do not park or retry anything that sends, pays or deletes. A "no" here is not an all-clear; it is a reason to fix the tools first.
  5. Can you live with the stack? Rust only. The suspend-to-zero setup needs five AWS services (a queue, functions, a database, a second store and a scheduler). In local mode, one independent check found the driver "stays alive while a tool result is pending."
  6. Can you live with the maturity? Version 0.2.0, a research release, with 60 open issues.

If you pass 1 and 2, and either your waits are short or you accept the cold-restart cost, 4, 5 and 6 decide whether to pilot the product. If memory does not bind, you want the pattern, not the product.

What to measure first, before building anything: (a) your waiting share; (b) how many waits run past five minutes; (c) your tokens on the first request after a gap versus a warm request; (d) your real per-agent memory.

#What we would do

We build and run AI agents every day, so we held our own setup up against the test above. This is our reading, and where we have not measured something we say so.

We would not run Infinity. It is Rust, its suspend-to-zero path is AWS only, it is version 0.2.0, and idle agent memory is not our limit. Our own usage report for one five-person week (recorded November 3, 2026) shows 95.7% of tokens as cache reads, 3.7% as cache writes and 0.6% fresh, across 29,875 requests, with about 194,000 tokens re-read per request. Our cost is in that re-reading, which a long wait, parked or not, would make worse.

We would borrow two patterns and one cost rule.

  1. Yield at a wait. When a job in our build loop waits on a check, a review or a person, it could give up its working slot and be re-queued when the result arrives. We have not measured how much of our slot time is spent only waiting, and we do not know whether a waiting job holds a slot today. First step: measure it over the last seven days. Our thresholds, both judgment: if more than half of slot time is pure waiting, design a parked state; below a fifth, drop the idea.
  2. Handle noisy events in a bounded child. We would try it on one noisy stream first. Compare the parent's tokens re-read per request before and after. The child re-reads the parent's history, so if more than half of events produce a report, or a child's fresh tokens exceed the parent re-read it saves, drop it. Cap the number of children, since each is its own bill.
  3. The cost rule: log the cold restart. For each agent, record the first request after an idle gap of more than five minutes and its fresh tokens. If most of our fresh tokens come from those requests, shortening the gaps or paying for the 1-hour cache matters more than parking. If few do, long waits are cheap on tokens for us. We do not know which yet.

We would leave RAP alone. The rule it points at is already ours: every job we dispatch must end in a result, with a stated check, stop and bail-out. If we build a tool server whose calls outlast a connection, we would look at the MCP task design first.

#What we did not verify

  • We ran no Infinity code, no benchmark and no Pi. Every performance figure is the authors' or an outside reader's.
  • One person ran four of the project's tests and reports what they showed. We did not reproduce those either.
  • The cost arithmetic is ours. It uses an Anthropic pricing page that shows no date, so treat the prices as a snapshot from November 4, 2026. We did not verify how long Bedrock's cache lasts.
  • Our token mix is one five-person week, and we have not measured our waiting share, our cold-restart share or whether a waiting job holds a slot.
  • We found no production user and no independent benchmark. We cannot rule out ones outside our search.
  • The 1 MB per agent and 8,000 agents figures are arithmetic from a rule of thumb, not measurements.

Related field notes: Cheaper AI agents: what the savings depend on, and what to keep a person on covers what cost cuts depend on, and Agents Did Not Clear the Backlog. They Moved the Bottleneck. covers where agent work piles up once the writing is fast.

#Sources

All read November 4, 2026, Central time, unless noted.

Keep reading