CLEAR is my qualitative coding pipeline: notes from a close read
I finally saw my manual error-analysis process in a paper. Then I saw how it scales past me. On my teams, people treat error analysis as too manual. ICs who rarely do qualitative coding hit cognitive overload when the volume grows.
Open coding, automated
I already ran open coding by hand. I read a failing trace and wrote a free-form critique of what broke. I wrote that diagnostic loop in Stop Operating Blind. CLEAR is the same loop with an LLM judge and Key Points Analysis.
The judge writes the per-instance critique, with a score beside it. Key Points Analysis clusters those critiques into recurring issues and counts how often each one shows up. That clustering is axial coding at dataset scale. Mapping each issue back to its instances is the prevalence pass I used to do in a spreadsheet.
My hand-built pipeline does not scale when several teams need the same capability and most of them have never done error analysis.
Node, trace, rubric
In Agentic CLEAR, a judge scores each run at three levels: node (one step), trace (the whole run), and rubric (a task checklist the judge writes, then marks fulfilled or not).
When I open-coded by hand, I packed node and trace into one line: “System provided a generic answer. Tool call expected but was not called.” That line holds both a step miss and a run-level miss. Whereas in Agentic CLEAR, separate judges score each level. Then it clusters node critiques per step with KPA and clusters trace-plus-rubric critiques across the system.
Figure: Agentic CLEAR flow. Traces go to Judge J for Trace Eval, Rubric Eval, and Step Eval. CLEAR aggregates system view (trace + rubric) and node view (steps split by node). Report UI shows sys, node, and trace views.
What a fixed taxonomy can’t do
A fixed failure taxonomy only sees categories I already named. My first pass missed a kind of failure I had not named. A later pass had to add it. Until I did that manual cycle, scored runs never counted that miss.
With open coding and Key Points Analysis, critiques come first. Clusters form from the data. A new failure type appears in the report before I rewrite the label set.