Automated AI performance optimization with a harness and goal-driven loop

I built a performance optimization skill for an AI agent. The interesting part was not patch generation. Every patch had to pass through a harness, a goal, and a ledger. In one Workspace run, strict profiles showed improvements such as Workstream 5089ms -> 2519ms and Report Center 10021ms -> 6762ms. The first important round was less glamorous: it repaired the FMP measurement contract so later numbers meant something.

The real problem

The target looked simple: improve Workspace FMP.

The actual target was stricter:

Normal AI coding is too weak for this job. If you ask an agent to “make FMP faster,” it will happily suggest lazy loading, chunk splitting, prefetching, caching, and SDK deferral. Some of those ideas may be correct, but without a harness the agent is optimizing a story, not a system.

So I designed the skill around a different contract:

No strict measurement, no performance claim.

The agent can write code, but code is only one step in the loop. The loop owns profiling, waterfall diagnosis, local verification, deployment, strict comparison, and documentation.

Harness first, optimization second

The first useful ledger entry was ugly. It said the production route did not emit valid final FMP reports for the tested subapps.

Several routes ended with invalid or missing events:

The root cause was not a slow chunk. It was a bad measurement data structure.

The host wrote one kind of key, subapps read another, and one idle reset could erase the map. Worse, some subapps used Object.keys(window.custom_performace).length === 1 as a route-origin detector. That is fragile: a timestamp map was being used both as data storage and as a state machine.

The first fix was therefore measurement repair:

1. Preserve and normalize the host FMP session.
2. Write canonical app keys instead of route-shaped keys.
3. Make direct-route source explicit.
4. Let each subapp validate its own expected key.
5. Add missing final FMP reporters, especially for Workstream list.

This produced no user-visible speedup by itself. It made the later speedups trustworthy. That is the kind of work a performance loop must do, and the kind of work an unguarded agent tends to skip.

The goal-driven loop

The skill runs one round at a time:

profile -> waterfall -> diagnose -> fix -> local verify -> commit/push
-> deploy to swimlane -> profile again -> compare -> document -> repeat

Each strict profile uses the same kind of capture:

• authenticated route;
• target deployment lane headers;
• same route and final marker;
• CPU 4x slower;
• fast 4G network profile;
• disabled cache;
• Service Worker enabled when that matches the production path;
• runtime version proof.

The verdict vocabulary is part of the system:

This looks bureaucratic until you have a failed optimization. Then it becomes the difference between engineering and self-deception.

Compressed into a minimal skill contract, I would keep only these rules:

Run measured rounds until the goal is reached.
One round = one bottleneck + one patch + one comparison.
No comparable profile, no performance claim.
Build, test, deploy, and smoke test are safety gates, not performance proof.
If measurement is broken, repair measurement before optimizing.
If visible behavior or post-visible jank regresses, mark not-a-win.
Record every round in the ledger before starting the next one.

What changed in the successful run

After the measurement contract was repaired, the loop could optimize real bottlenecks.

One strict-profile summary:

There was also a weekly P90 view across modules. Some routes reached the 2.5s target, some got close, and some still missed it. That distinction matters. A serious performance system should report both the wins and the remaining gap.

Pattern 1: post-FMP scheduling beats blind deferral

The easy version of deferral is setTimeout. That is garbage. It is disconnected from the product’s real readiness.

The safer version waits for a route-specific final first-screen mark:

function finalMarksForRoute(pathname: string): string[] {
  if (pathname.includes('/workspace/scheduling')) {
    return ['scheduleManagementPageFirstRender'];
  }

  if (pathname.includes('/workspace/workstream')) {
    return [
      'Workstream List Page FMP Complete',
      'Workstream Detail Page FMP Complete',
    ];
  }

  return ['Report SubApp FMP', 'Audit Workbench FMP'];
}

function scheduleAfterFirstScreen(task: () => void, pathname: string) {
  const marks = finalMarksForRoute(pathname);

  if (performanceHasAnyMark(marks)) {
    runWhenIdle(task);
    return;
  }

  observeMarks(marks, () => runWhenIdle(task));
  fallbackAfter(5000, () => runWhenIdle(task));
}

Notification work is a good example. The download bar itself was not always the parser blocker. The real cost was SDK/auth/callback/API/main-thread work competing before the final marker. Moving it after the route-specific FMP mark improved the measured path without removing user-facing UI.

Pattern 2: prefetch is a contract, not a request

Report Center exposed the worst prefetch failure mode: the host fired a prefetch, but the consumer used a different body or cache key. The network request existed, the page still waited, and the “optimization” became extra traffic.

The rule that came out of the ledger:

The cache key is the contract.

A valid prefetch must align:

The consumer still needs a fallback. If metadata does not match, reject the warmed response and perform the normal request. Lost FMP benefit is acceptable. Wrong data is not.

Pattern 3: cache hits are real readiness signals

The system used two prefetch layers:

The important detail: a Service Worker or memory cache hit for init data is still a valid readiness signal. If core data is available earlier, init-dependent prefetch should start earlier. Do not accidentally treat cached data as second-class.

This is also where the goal-driven loop matters. Prefetch has real cost: bandwidth, backend pressure, cache memory, and wrong-data risk. The skill only accepts prefetch when the first-screen request is deterministic, safe, and consumed under the exact same key.

Pattern 4: correctness guards are performance work

One intermediate Workstream optimization changed visible details: avatar, pinned section, loading style, table border, and unpin button styling. The ledger marked it as a correctness guard, not a performance win.

That is the right call. A faster page with broken visible behavior is not an optimization.

The final version preserved the UI while keeping non-first-screen work out of the FMP path. This sounds obvious, but it is exactly the sort of tradeoff an agent needs written into the skill. Otherwise it will happily optimize the chart and damage the product.

Similar failed rounds became hard evidence in the ledger:

The ledger is the control plane

The ledger is not a diary. It is the control plane for the loop.

The best ledger rows were compact and harsh:

This is why the system is goal-driven rather than prompt-driven. The goal picks the metric. The harness decides what evidence is valid. The ledger decides whether the round can be accepted.

What I would reuse elsewhere

You do not need the same internal infrastructure to copy the design.

The reusable structure is:

1. Pick one user-visible metric.
2. Build a repeatable harness for it.
3. Define strict comparison rules before changing code.
4. Make the previous valid profile the next baseline.
5. Let the agent attack exactly one bottleneck per round.
6. Treat measurement repair, regressions, and correctness fixes as first-class outcomes.
7. Keep a ledger that records evidence, not activity.

The technical details will change across products. The loop is the part worth keeping.

The result I cared about was not one clever AI-written optimization. It was an agent running an engineering loop with evidence: observe, choose, change, deploy, compare, record, and keep going.