With-skill vs. without-skill on the same rubric: 50% → 100%. The interesting part wasn't the score — it was the feedback loop.
Evals are how I think. When I rolled out AI tooling to a whole enterprise, the question was never “does this demo well?” — it was “how do we know it’s working, and what do we measure when it isn’t?” So when I packaged one of my own IT workflows as a Claude skill, I wasn’t willing to call it done just because the output looked plausible. I wanted a number.
The workflow: connecting a SaaS app to OneLogin — SAML SSO, SCIM provisioning, attribute mapping. It’s the kind of task an experienced admin does on autopilot and everyone else gets subtly wrong. That made it a perfect skill candidate: the knowledge is real, procedural, and mostly invisible.
The benchmark was simple by design. Three realistic evals — Box with a native connector, Figma with a maximal attribute push and full user lifecycle, and translating a vendor’s SSO documentation into tenant configuration. Same model, same prompts, graded against the same assertions. The only variable: whether the skill was loaded.
Without the skill: 50% of assertions passed. With it: 100%.
What’s telling is how the baseline failed. It wasn’t bad at SAML — it was bad at reality. It guessed attribute field names instead of querying the tenant’s schema first. It built generic custom connectors when native, provisioning-capable ones existed in the catalog. It invented example credentials instead of asking for real ones. It reused the shared signing certificate instead of issuing one per app, which is how one rotation takes down every integration. The skill wins not by knowing more, but by refusing to skip the boring steps.
The part I care about most isn’t the score — it’s the loop. After the first run, a human reviewer (someone who actually does this work) reviewed the outputs and left feedback: a real cert-naming convention, a values-to-exchange quick-reference chart, more verbose steps for the service-provider side. I turned each piece of feedback into new assertions — the rubric grew from 5 checks per eval to 8 — edited the skill to absorb the fixes, and re-ran. The skill held at 100% against the harder bar; the baseline dropped. Human judgment became executable criteria. That’s the whole game: the eval harness isn’t how the skill gets scored, it’s how the skill gets better.
Honest caveats, because a benchmark without them is marketing: it’s one run per configuration, so no variance across repeats. The second-iteration baseline was regraded against the harder rubric, not re-run. And a separate experiment — automatically optimizing the skill’s trigger description — flatly failed; the harness never fired, and I picked the description by hand. That’s in the repo too. Negative results are results.
Everything is public: the skill, the harness, the raw transcripts, the human feedback, and the failed optimizer log. Nothing summarized away.