Loosely Coupled Thoughts

agentic-workflows

Levels of AI Adoption

Every software engineer should be paying attention to Boris Cherny's Steps of AI Adoption.

The products and tools are obviously Anthropic Claude/Claude Code specific, but it should transfer generally to the likes of Codex, OpenCode and others as these harnesses continue to rapidly mature.

Level 0 and 1

Level 1 is really the entry point into agentic engineering. I'm surprised by the number of engineers I talk to that are still operating here. Things are moving too fast to be complacent at these levels.

The bottleneck things mentioned for level 0, in my opinion, apply at all the levels.

Level 2

Most software engineers are, or should at least be, operating at this level.

Level 3

This is where I spend most of my time. It's incredibly satisfying when Claude is fully autonomously cooking on something, or a few things. Automated maintenance tasks are something I can lean more into, but for features or larger improvements I just don't have enough work in a fully planned design/spec state to be running many things in parallel. The planning/design phase, knowing exactly what to build, consumes a very large portion of my time. The good news is this can be done while Claude is cooking on something else.

Below is an example of a recent feature build I ran using a dynamic workflow script that I have been refining. It's slow and meticulous, taking 13 hours to complete. It handled everything from writing its own implementation plan for a design I gave it, all the way through implementation and verification loops with unit, component and e2e tests, to creating a draft PR for me to review offering proof of completion and a ledger of work completed and decisions made. It was only 66 agents, but the nature of this particular piece of work meant not that many things could run in parallel without the agents stepping on each other's toes. Side note, Opus 5 on this workflow run smashed this piece of work out of the park. 73 files +3431 -1147 lines of code. After self-reviewing and self-testing, not a single thing needed changing; this was a straight 100% complete ready-to-merge change. Fable and Opus 5 are just incredible!

Level 4

To be honest, I was a bit surprised to see here that Claude Code isn't even used at this level, but instead building custom mini-harnesses with the Agent SDK. I can see how this works; it's basically the output of Claude Code creating a dynamic workflow, which is just a script built with the Agent SDK. I haven't tried running these without Claude Code yet, but it would appear these dynamic workflow scripts can be used in just this way.

Getting to this level must mean knowing exactly what the agents should be building. Who is planning this work and getting it into a ready state for the agents to just jump in and start building? Surely it can't be just one person — yet Boris Cherny says he is now operating at level 4. My guess here is that it's larger, more ambitious pieces of work that can run for days or even weeks at a time, provided the goal is clear to the agent and a harness exists to keep them on track on their mission.

[... 718 words]

Superpowers 6

Jesse Vincent just released version 6 of his Superpowers plugin for coding agents, and has made it a lot faster and a lot less token hungry.

One thing that can be frustrating is planning and building with the plugin skills can be slow and use large amounts of tokens. It's no surprise though considering what it does.

A bunch of the reasons that Superpowers builds have taken longer and cost more are the same reasons that it delivers good outcomes for so many users. It does a ton of up-front planning work to make sure your implementations can be hands-off, forces strict red-green TDD while implementing, and then the orchestrator inside Superpowers reviews every single change on two axes:

  1. did the agent implement exactly what was asked, no more and no less.
  2. is the quality of the work up to snuff.

Just by the nature of what it's doing, it's going to be slower than yoloing an untested implementation and calling it a day.

However, it seems he also managed to take advantage of the few short days that Fable 5 was available and used it deeply analyse thousands of Subagent Driven Development sessions and effectively have it create an autoresearch harness that ran 25 experiments to make it more efficient.

/goal once this is done, run an autoresearch loop to improve cost-efficiency of the superpowers build loop. test with opus as the coordinator. make an hypothesis log. run experiments. run at least 25 experiments.

Resulting in a whopping 50% speed up and 60% reduction in token usage.

The long and the short of it it is that across about 36 hours of work and what would have been $650 of unsubsidized token spend, our Anthropic eval benchmarks were looking like we'd reduced wall-clock runtime for Superpowers builds by 50% and token spend by 60%.

I use this plugin a lot and has served me well. Speed has never been a major problem as it runs in the background and can do other things. But the context switching while waiting can be tiring and this will probably take the edge off of that a bit. And probably means less watching that 5 hourly rate limit when kicking of large builds. And with stronger and more costly models becoming available the token usage reduction is welcome.

Implementing a Full-Stack Feature Using Claude Code's Dynamic Workflows

This is a continuation from the post I wrote recently titled Designing a Feature with Claude Design – Then Handing It to Claude Code.

In that post, I stepped through how I used Claude Design to create a UX design as well as a set of initial requirements for new tag selection and suggestion functionality in my blog's custom CMS.

This post steps through using those artefacts to drive the actual implementation in the frontend and backend apps. Specifically the goal here is to build this end-to-end using Claude Code's new Dynamic Workflows.

The outputs of the Claude Design exercise were a fully functional interactive prototype of the design and a markdown file with a set of requirements and acceptance criteria.

I already have a structured process for defining per-feature design specs, requirements and implementation plans in the projects repo. It consists of several things, but most relevant here is a folder structure and set of markdown files which I would consider the "spec" as part of the planning phase of this AI-driven engineering workflow. These are used as context for the implementation and verification, which often run across several separate sessions.

Usually when starting a feature I would be starting from scratch, or just a high-level set of requirements from the initial blog design and feature roadmap. For this piece of work, Claude Design gave me a working prototype that was already using my frontend style guide and a detailed set of requirements, which I put into the code base for the duration of the build. So I had a solid starting point. However, those didn't fit into my structured process that I already have in place, and I still needed a design spec for the full end-to-end solution.

For larger pieces of work I like to use Jesse Vincent's superpowers plugin. He has done a fantastic job of baking in real software development methodologies and workflows into a set of agent skills that drastically increase the quality and coherence of what's being built.

The first two skills I normally reach for are the brainstorming skill, which is for creating a design spec and the writing-plans skill, which is for writing an implementation plan for the spec. The design spec is the most important part. For any given feature this is where I spend most of my time. The clearer this is defined upfront, the more seamless the rest of the process will be and the higher the chance of it building what I actually want.

Once I'm happy with the design spec and implementation plan, after rounds of refinement, I would then hand it off to Claude Code to start building, usually with subagent development or agent teams.

I already found a general workflow that I follow when building software with AI. In its most simple form it's the Plan-Generate-Evaluate methodology. But that's a gross oversimplification of the entire underlying process. In fact, for each of the Plan-Generate-Evaluate phases I will run inner Evaluate-Regenerate flows, iterating over this at different phases until outputs are where they need to be. The first outputs are rarely good enough.

[... 2746 words]

Maintainability sensors for coding agents

This is a great post from Birgitta Böckeler on martinfowler.com talking about the sensors that can be used with generative AI engineering to shift quality assurance left of the delivery pipeline.

One big part of this is the general state of the codebase and repository, even before adding in sensors. I like the definition used to describe this: internal quality.

There are multiple dimensions we usually want to achieve and monitor in our codebases: Functional correctness (works as intended), architectural fitness (is fast/secure/usable enough), and maintainability. I define maintainability here as making it easy and low risk to change the codebase over time - also known as “internal quality”.

And why this is important for engineering with AI.

Internal quality problems affect AI agents in similar ways that they affect human developers. An agent working in a tangled codebase might look in the wrong place for an existing implementation, create inconsistencies because it has not noticed a duplicate, or be forced to load more context than a task should require.

I have heard developers say their agents go off track easily and implement things with mixed styles. There are likely a few reasons for this happening, but this internal quality is probably the top of the list.

She talks about a number of different approaches to implementing these sensors, both computational and inferential: linting (standard and custom), static code analysis to control cross-dependencies, coupling data and modularity patterns, and test suites as a regression sensor.

There are a lot of good things in there, but the section on test suites I found to be valuable. LLMs love to write tests and they can be highly verbose, testing things that are not important while totally skipping things that are critical. Or worse, fixing failing tests by making them pass against broken or buggy code. Like most other things with generative AI, these things can be steered to create genuinely meaningful tests that the LLM can largely manage with light oversight. I'm a huge fan of end-to-end tests, or a subset of those, component-testing that runs locally against running services, with local or mocked dependencies. These allow for greater control of different scenarios - edge cases, acceptance criteria and user scenarios - that would likely result in heavier, slow running and flaky tests if running in a deployed environment.

There currently seems to be a trend towards more end-to-end style acceptance tests. As mentioned in the beginning, AI has gotten really good at generating tests, so it has become quite normal for developers to just let AI generate lots of tests, without much review. Reviewing unit tests in particular can be very tedious. I'm not saying it's a good thing not to look at them at all — but I acknowledge the reality that it is unrealistic to think that human review of all tests is sustainable, and it's unrealistic to think that people will actually do it. So while we search for the appropriate testing pyramid/ice cream cone/muffin shape of the AI coding future, techniques like approved scenarios are becoming popular. As demonstrated above, acceptance tests increase coverage, but are often not very assertion-heavy, giving us a false sense of security in test effectiveness — mutation testing helps us monitor that gap.