Ecosystem Factory GitHub

Open source · built in public

Verified agent-system generation

Describe a problem. The factory interviews you, designs a small AI agent system for it, and has to prove it with evaluations before it goes live. A human approves every step that matters.

View the code on GitHub Read the build story Follow the updates
The factory's path: brief, interview, charter, gate G0, blueprint, gate G1, generate, evaluate, gate G2. Each gate is a human approval.

The problem

"It seems to work" is not an acceptance criterion

Most agent demos stop when the output looks reasonable. Nobody measures whether the reviewer agent catches real problems, invents problems that aren't there, or quietly softens what it finds. This project treats that as the main question, not an afterthought.

Small by design

As few agents as the problem needs, on fixed routes with hard limits on loops, calls and tokens. Sometimes the right answer is a script.

Measured, not assumed

Reviewer agents are calibrated against planted defects and near-miss clean cases. A weaker model must score lower on the same suite.

Human-approved

Every gate is a decision made in the owner's own terminal. An approval is bound to the exact files approved; change one character and it's void.

How it works

From a short brief to a proven system

Four stages, three human gates. Agents do the work; only a person can approve it.

1

Interview

You fill a short brief. An interviewer agent asks only about what's missing, vague or contradictory.

2

Charter G0

Your answers become a structured problem statement. Every line cites the answer or file it came from.

3

Blueprint G1

The charter becomes a design: which agents, which rules, which gates and limits, each one justified.

4

Generate and prove G2

The system is generated on a fixed, tested engine, then has to pass its evaluation suites before go-live.

What's proven so far

The first real job, end to end

A generated two-agent system turned meeting notes into an action list: a writer lists each action with its owner, due date and source sentence; a critic checks the list against the rules.

1
valid finding by the critic, fixed by the writer
0
issues missed, as marked at the human gate
~24k
tokens for the whole job
650+
automated tests (November 2026)
Cases passed by the critic's evaluation suite: first run 12 of 20, failed; after one fix 20 of 21, passed; a weaker model 7 of 21.

The failed evaluation was the most useful result

The first evaluation run failed: in some runs the critic softened major problems into minor ones, and the writer listed a conditional plan ("if the café says no…") as an action. One fix round later, both suites passed with their thresholds unchanged, and a weaker model still scored far lower, which shows the tests aren't too easy.

Synthetic test data throughout. Figures from the project's build log.

Lessons

Five lessons that carry to any agent project

Each one came from something that broke. The full list is in the repository's lessons file.

  1. An unreadable file is not an empty file.A settings file with a missing comma was read as "no settings" and passed the check. Now "can't read it" is always a failure.
  2. Evaluate the prompt the job will actually send.Evaluations passed while the real job's reviewer had no rulebook. Jobs and evaluations now share one prompt builder, compared word for word by a test.
  3. A tool's promise is only as good as the tool.An agent declared with no tools still received tools on one platform version. Agents now run sealed, and the setting is re-checked on every update.
  4. When a test fails, check the answer key before the agent.Several "failures" were wrong expected answers. Each failing case is now judged on its own merits before anything changes.
  5. A check that looked at nothing is not a pass.A publishing check reported "clean" after scanning zero files. Every pass must now say how much it examined, and a metric that can't be computed fails.

Status

Where the build stands

Built phase by phase; a phase only counts once its exit criteria are approved. Full plan in the roadmap.

Phase 0FoundationEngine, guardrails, approval gates, evaluation runner, setup checkerPassed
Phase 1Thin end-to-end versionA generated ecosystem ran a real job with evaluations, a gate and measured usagePassed
Phase 2IntakeInterview agents that turn a brief into a charter; now being evaluatedIn progress
Phase 3DesignThe factory chooses the design itself: a script, one agent, or severalPlanned
Phase 4Generation and lifecycleGenerated systems pass held-out functional tests, not just structural checksPlanned
Phase 5Evidence-based autonomyLow-risk approvals relaxed only where the measured record supports itPlanned

How it's built

A product manager, Claude Code, and a strict process

The author is a product manager who doesn't write code. Claude Code wrote and tested every line; the author specified, reviewed and decided, with a second Claude conversation reviewing every plan and result before approval.

The working rules

  • Every task: a plan, a check that must pass, one commit, one log row
  • Wrong agent output is never fixed by hand; the agent or its rules change
  • No threshold is ever lowered to make a test pass
  • Synthetic or public data only; nothing from anyone's work
  • Every push passes a publish check for secrets and personal data

Scope

What this is, and what it isn't

It is

  • A factory for small, purpose-built agent systems for one owner
  • Local, single user, on a Claude subscription
  • Evaluation-first: nothing goes live unproven

It isn't

  • A multi-repository code generator or dependency-graph tool
  • A general agent swarm or orchestration framework
  • A hosted, concurrent or enterprise-compliance service

Updates

Follow the build

An update at each milestone, with the numbers and with what broke.

Part 1: An agent factory that doesn't trust its own agents

The first real run, the evaluation that failed, and five lessons. Link coming soon.

Next: the first charter from a live interview

Coming at the end of Phase 2.

Discussions

Questions, ideas, and how you measure your own agents.