I built an AI reviewer to check my AI's work. The first version made it worse.
I built Alfred Ultra, an AI reviewer skill, to check my AI's work. The first version scored below the plain model. Here is what two tests showed.
By Wasif Mumtaz4 min read
I use AI models to build websites, write code and produce design work. The output is usually good, and sometimes it is wrong in ways I only catch after a client has seen it. So I built a reviewer. I call it Alfred Ultra: a set of instructions that any AI model can load to check another model's finished work before it reaches me.
The idea was to capture how the most capable model I can use, Claude Fable 5.2, goes about judging a piece of work, and hand that to Claude Opus 5.5, which does most of my day-to-day building. Alfred Ultra covers the five kinds of work I do most: web development, security audits, SEO, graphic and motion design, and video. It only runs when I call it by name, the same way my agents wait for my approval before they act.
Then I tested whether it did anything.
Test one: a design job
I gave Opus 5.5 a wireframe for a manufacturer's website and asked for a polished design with motion, delivered as images and a short video. A second Opus 5.5 reviewed that work with Alfred Ultra loaded. Fable 5.2 wrote its own judgment separately, before reading the review, so I had something to measure against.
The review with Alfred Ultra scored 6.5 out of 10. It reached the right verdict, and it caught a broken image that the stronger model had missed. It also missed the two biggest design problems.
I assumed 6.5 was still better than no skill, and I guessed plain Opus 5.5 would land around 5. I ran the same review without the skill to check. Plain Opus 5.5 scored 8, which meant my skill had made the model worse.
What went wrong
Opus 5.5 is already a careful reviewer. Most of what I had written told it to do things it does anyway, and one part of my instructions held it back. I would not have found that by rereading the skill, because every line in it sounded sensible.
I rewrote it and ran the test again with a fresh copy of Opus 5.5 that had never seen the earlier runs or reviews. This time the Alfred Ultra review scored 9 against plain Opus's 8. It found the main design problem that both earlier reviews had missed, plus one fault that no earlier review had spotted.
Test two: a development job
The first test was almost all design, so I built a second one for code. I made a small booking app and planted 16 faults in it: security holes, missing features, logic errors, and accessibility and SEO problems. I wrote the answer key before either review started. The app came with a confident report claiming everything worked.
Both reviewers found all 16 faults. Neither raised a false alarm, and both called out the false claims in the report.
The difference was in what they handed back. Plain Opus proved 12 of the 16 by running the app and took the other four from reading the code. With Alfred Ultra it proved all 16 by running the app, found seven more real faults I had not planted (plain Opus found four), and wrote fix instructions another model could follow without me. I scored them 9 and 9.5.
The scores
- Design test, first version of the skill: plain Opus 5.5 scored 8, with Alfred Ultra 6.5.
- Design test, rewritten skill: plain Opus 5.5 scored 8, with Alfred Ultra 9.
- Development test, rewritten skill: plain Opus 5.5 scored 9, with Alfred Ultra 9.5.
What it costs
In both tests the Alfred Ultra review used about 29% more tokens than the plain review. A review of any kind roughly doubles the cost of a task, because the reviewer has to open everything the builder made.
What I took from it
A skill can lower a model's performance, and you will not see it unless you test against the same model with no skill at all. My guess at that baseline was off by three points.
The gain was largest on design, where the question is whether the work is good enough for the client. On code with clear faults, Opus 5.5 finds them without help, and the skill mainly improves the proof and the fix instructions.
These are two tests with one run each, and the design scores are a stronger model's judgment, so I read the numbers as a direction.
I am keeping how Alfred Ultra works to myself. I have not tested it on a smaller model yet, and that is where I expect the gain to be larger.
(Keep reading)
- Personal brandsWhat Cristiano Ronaldo taught me: 5 lessons from a die-hard fan
- WebsitesWordPress vs Shopify for small business: how I choose
- AI automationAI agent human approval: what my agents can't do without me
- ArchitectureYour business is a system. Draw it before anyone builds it.
- TeamsMost projects break at the handoff
- Personal brandsWhat building sites for an Olympian and a PGA pro taught me about personal brands