How We Stopped Babysitting Our Agents

When we first let agents write to our database, they made a mess of it. They spent a lot of time guessing what data structure the database expected, either making up fields, leaving them out or getting the typing wrong. None of this is unique to us.

In τ-bench, when agents were given the same task eight times in a row, they got it right all eight times in fewer than a quarter of cases, and TheAgentCompany found similar. Most of the industry has looked at those numbers and concluded that a human needs to approve what an agent writes: AWS says as much for anything that could modify production data, OWASP tells you to require human approval for high-impact actions, and Okta treats the whole thing as a least-privilege problem. Cognition, who build agents for a living, won't even let two agents write at the same time. Anthropic found that approval prompts wear people down until they "tune them out," and their fix was to move review up to the plan instead of every individual action.

At Orbital Industries we looked at the same numbers and went the other way. Since the start of April, agents have committed more than 800,000 writes to the production database that holds our entire organizational corpus of data, without a human approving a single one.

Agent output is a proposed transaction

We've been building CurieOS that turns users' day-to-day work into steps over data that agents and humans work on together. To enable this, we have strict typing on our data.

{
  "type": "PullRequest",
  "fields": [
    { "key": "number", "kind": "number", "required": true, "unique": true },
    { "key": "title", "kind": "string", "required": true },
    { "key": "author", "kind": "string", "required": true },
    { "key": "classification", "kind": "enum", "values": ["small fix", "feature", "refactor"] }
  ],
  "states": {
    "values": ["open", "ready", "needs_human_review", "closed", "merged"],
    "transitions": [
      { "from": "ready", "to": "closed" },
      { "from": "ready", "to": "merged" }
    ]
  }
}

That keeps the data consistent without us having to be opinionated about the data itself.

However, agents are stochastic by nature, and prompting isn't an exact science. At first, we explained in the prompt the rules of the data typing and expected the agent would produce JSON output that mapped to it:

You are the Orbital Industries BOM builder. A purchase request has been approved
or queued for purchase. Create BOM items from its line items IF they don't
already exist.

STEPS:
1. Read the Source Node's properties: line_items (JSON array), project,
   vendor_name, request_id.
...
4. Determine the project. Map to a BomItem status:
   - "Cirrus" -> "Cirrus"
   - "Nimbus" -> "Nimbus"
   - Anything else -> "Consumables"
...
6. For each line item, increment the counter and create a BomItem node ...

Output ONLY raw JSON:
{"created": number, "skipped": 0, "items": [{"bom_id": "BOM-NNNN",
"part_name": "...", "project": "..."}]}

This proved unreliable, with agents often missing required fields, misclassifying data types, and using poor data structures. This led us to the idea of a validation path for agents, where the agent can call an endpoint and say "is this data valid?". If no, we return an explicit error; if yes, we instantly store the submitted data, rather than relying on the agent to convert it back to JSON for output. That removes one more step where the data can get mangled.

When the agent got this wrong, it would return plain English where a JSON list was expected:

1 validation error for tagged-union[CreateNode,CreateNodesBatch,UpdateNode,...]
create_nodes_batch.items
  Input should be a valid list
  [type=list_type, input_value='The Salesforce query cou… Salesforce
  connection.', input_type=str]

The validation path now returns structured feedback instead, explicit enough that the agent can fix its own mistake on the retry:

downstream delta simulation failed for chat_id=019e6eca… — retrying (2 remaining):
- [type_mismatch] Field 'generation' expected number, got str
- [unknown_field] Candidate key 'action' is not consumed by any downstream task —
  either it's a misspelling or the workflow doesn't use it.
  Consumed keys: ['generation', 'hypothesis', 'name', 'smiles']
- [required_field] A downstream task references $ref:strategize.hypothesis but the
  candidate output didn't emit 'hypothesis'. Without it, the downstream write
  would silently substitute an empty string.

This led to a major improvement in reliability. The agent task failure rate dropped from 28% to 6.5% in the three weeks after we introduced the validation path, while the weekly task volume tripled. However, this also created a dilemma. Users could create data either through our chat agent or through workflow steps in an automated process, and while both ran on the same base agent they were now running on two different code paths, one for pure CRUD (user chat), one for validation loops (tasks). Looking closer, the chat path was hitting the same wall the task path had, because validation failed the same way no matter how an agent reached the database, and we had been treating agent output as an answer to parse when what we needed was a transaction to propose.

One pipeline for every writer

The solution was a single mutations pipeline that every writer uses, regardless of the source. This pipeline changes the input request into a proposed transaction that can be processed and validated. The agent call is no longer a special case because every call to change the data is the same, and the same set of steps applies to all of them.

Four kinds of writer feed the same front door: a human using the React UI, the chat agent inside CurieOS, the workflow engine's automated steps, and connector or LLM-funnel ingest. The UI and chat agent both go through a POST /mutations REST endpoint; the workflow engine and connector ingest call in-process. Either way, the request becomes a CompoundDelta — the proposed transaction — and every one of those, with no exceptions, passes through a single mutation pipeline: the one write gate that runs schema checks, business rules, permissions, locks and version checks. That gate reads its rules from the GraphSchema (schema plus rules, stored as pure data) and, once a change clears it, writes to the GraphStore backing our Neo4j database.

So, every request becomes a proposed transaction that will be saved to the database in a language the engine understands. This lets us conduct a full validation and simulate how it will affect other parts of the system. We can then confirm whether it's a valid change set; if it isn't, we notify the requesting service and allow retries. This means we maintain absolute control over the data that enters the database without having to run manual checks, and we get full observability on every change made. All we need to do is enforce strict type settings on the data fields and follow the input validation rules we established.

This has helped agents become the main contributors to our database, with 88% of all changes directly made by agents at the time of writing. This statistic exists because of the single write path, which allows us to keep a record of every change and identify its author, whether human or not.

One issue remained, specific to our agent-driven workflows: a change set could pass all validation checks and still be incomplete. Each change could be valid on its own but might lack a necessary node or edge that a later step needs. To catch that, we must know what the agent was meant to produce.

A contract for shape, not values

To understand the requirements here, we need to revisit the core concept of the application: users build workflows, and many of the steps ask an agent to reason about something and write the result to the graph, with early steps often producing data that later steps rely on.

What makes this possible is that we know what data we need to output at each stage of the workflow. So we can go further than validating that the output data is valid; we can prepare a contract that explicitly says what must be output.

{
  "produces": {
    "deltas": [
      {
        "kind": "update_node",
        "node_type": "PullRequest",
        "target": { "context_ref": "subject_node_id" },
        "required_fields": ["classification"]
      }
    ]
  }
}

The contract carries no property values, only a type, a target and the field that must be set. It's the contract for the "Persist PR Classification" step in a workflow like this: the step must update the PullRequest that triggered the run (context_ref), and that update must set the classification field. It says the field has to be present but it does not say anything about whether the answer is small fix, feature or refactor.

The contract is reference-based, letting us declare that a later step relies on output Y from the task with id X, and here the "Persist PR Classification" step consumes the value the "Classify PR" step produced. We can build up complex sets of tasks that rely on data that doesn't exist yet and will change from one workflow run to the next. The contract does not and cannot care about the specific data: whether classification is one of the allowed enum values is the schema validation's job, and whether it's the right call for this PR is the agent's; the contract only cares that the declared shape is delivered. As an author-time constraint, it lets us confirm the workflow is logically correct as written, with all the dependencies in place. The contract has two jobs: the same declaration that validates the workflow at write time is what we check the agent's actual output against at run time.

The two-layer gate

So we have a complete validation picture at write time; now we need to make sure the producer's output is also valid. As we go through our pipeline, the change set turns into a proposed transaction, and several checks are run on it.

Layer 1

This layer checks if the change set is properly formed on its own. Are the fields the right types? Are the edges directed correctly? Is the status transition valid? We can evaluate all of this by comparing the proposed change to the basic rules of the database. These are the same checks that every human write goes through. The dry run uses the same validation middleware, so there's no need to create, manage, or allow for drift in agent-specific validators.

Layer 2

Layer 2 asks a narrower question: the change set is valid, but does it meet the contract? We have already verified that the contract itself is sound at write time. So now, we just need to check the output against it, as shown below.

- [contract_incomplete] contract requires field 'classification' on the
  PullRequest update but the compound sets no such field
    expected: update_node PullRequest (required: classification)
    fix_hint: author an update_node on PullRequest that sets 'classification'

- [contract_mismatch] update_node target does not match the contract
    expected: context_ref:subject_node_id
    received: ref:other_pr

Every rejection carries expected, received and a fix_hint, because the error is written for a machine reader. Rather than a generic "invalid request," the error spells out which part of the contract went unmet and what the agent should author on the next attempt.

Any error caught in these two layers is instantly surfaced back to the producer, so in the agent case it can loop until the errors stop, or it runs out of retries and fails loudly into human review. The loop works because retries are inexpensive: the contract check involves a simple, synchronous comparison of two in-memory structures, with no database visit and no extensive simulation, and layer 1 makes use of the same validators that every human write already goes through.

The output of all this is a valid transaction, stored and ready to be applied to the database.

What happens after the gate

Having proved the change set is correct, the whole of it then has to land together, and if any part fails, we roll the rest back. In the last two weeks the pipeline rejected and atomically rolled back ~1,300 compound transactions, roughly one in twenty-five attempts.

You may think this shouldn't be possible, given the checks described so far, but we're operating a scaled app with multiple processes writing and state changing live. Say a node has a rule that it can only have one outgoing edge: at validation read time the count is 0, but two concurrent processes are acting on that node, so they both see the 0, both pass validation, and both commit their change, leaving the graph in an errored state.

Our answer is to serialize the writers, taking a write lock on the contended node before running the relational checks so that two concurrent change sets touching the same node are forced to queue. The second one re-runs its checks after the first commits, sees the count is now 1, and is correctly rejected. Alongside the locks, every node and edge carries a version, so a write based on stale data becomes an explicit conflict surfaced back to the writer, rather than a silent overwrite.

None of what we've described so far knows what a PullRequest or a BOM is. The design rule that dominates our codebase is generalization: the code operates on whatever types the schema declares.

There are no domain types in the code at all, since the node types, the fields, the rules, and the contracts are all data, declared in an uploaded schema definition. A PullRequest is simply what the user defined in their schema. Which means every new domain — and we run ~130 of them in production, on the same engine — inherits the entire pipeline the moment its schema is uploaded. Nothing in this post was built for chemistry, purchasing, or PR review specifically; it was all built once.

Agents as untrusted clients

We have borrowed this idea from something that everyone uses every day, web browsers. In web development it's standard practice not to trust the browser. Whatever arrives from the client gets validated on the server, every time, without exception. An agent is the same kind of thing, an untrusted client that's probabilistic rather than malicious, and the answer is the same: make the server strict.

The contract and the gate are that same principle applied to agents. We gate everything that's mechanically checkable (the types, the shape, the wiring, the transitions) so that the only thing riding on the model is the work an LLM is actually good at, which is filling in the data. Whether the data on the PullRequest is correct is the agent's problem, and whether the data is structurally sound is the schema's. Our key insight is that one write path can serve both: the gate enforces the schema's half and stays out of the agent's.

We have automated more than 800,000 writes to our production database since April. Agents now make 88% of all mutations to our graph, 30 for every 1 a human makes by hand, and the failure rate fell while volume tripled. We still allow humans to review, but only at the moments they deem right, not at every single data entry.

This project fits underneath our wider work on CurieOS, our operating system for agentic AI in science and engineering. To read more about CurieOS, and how we use it to accelerate a range of tasks, see Introducing CurieOS: From the Atoms Up.