AIOpsAnalystHow it works
Five stages. Only one of them uses a language model, and it is never the one that decides what happens.
Everything becomes one shape
A GitHub issue, a failed CI run, a status-page incident and a line from a server log look nothing alike. Each source has a small adapter that turns what it reads into the same envelope: where it came from, when, a title, a body, and a link back to the original. Nothing after this point knows or cares which kind of source it was.
Adapter: a small piece of code per source. Adding a new source means adding one adapter; nothing else changes.
- GitHub issues330
- failed CI runs150
- status-page incidents175
- server log lines20,500
- supercomputer log lines10,000
- in one shape31,155
Same problem, one pile
Before any model is involved, events that are the same problem are put in one pile. Log lines are grouped by the sentence they share, with the numbers and names that change from line to line set aside. Written reports are grouped when their wording is close, first by shared words and then by meaning.
The grouping is deterministic: the same input always produces the same piles, run after run. That matters because every later decision is attached to a pile.
Pile: what the code calls a cluster. 31,155 events became 905 piles, 34.4× fewer things to read.
Nov 15 04:35:42 combo kernel: Out of Memory: Killed process 28475 (httpd). Out of Memory: Killed process 6695 (httpd). Out of Memory: Killed process 6699 (httpd). Out of Memory: Killed process 30289 (httpd). ... and the rest of a pile of 2,579 lines
One pile of 2,579 lines, read once.
The model reads each pile once
A language model is shown a few samples from each pile and asked four things: what kind of problem it is, how bad, how sure it is, and to say it in one sentence. It also points at the sample events that support its answer, and that is all it returns. It is asked 905 times for this corpus, not 31,155.
Confidence: the model's own estimate, from 0 to 1, that it named the kind of problem correctly. The rulebook reads it; it never acts on its own.
- Kind of problem
- error
- How bad
- medium
- How sure, 0.95
- In one sentence
- Playwright fixture tests are failing on the main branch across multiple unrelated commits, indicating a flaky or broken test suite in CI.
A rulebook, not the model, decides
The model's answer goes through a short table of rules. They are read top to bottom and the first one that matches decides. A critical problem always goes to a person, however sure the model is: paging someone is cheap, a wrong automatic action during an outage is not.
The whole rulebook
- severity is criticala person looks first
- confidence at or above 0.9acts on it alone
- confidence at or above 0.7drafts it for approval
- severity is higha person looks first
- anything elserecords it as unsure
When it acts alone, it sends a ticket only for crashes, errors, slowdowns and feature requests. Anything else it is sure about is routine, and dropped.
- 350Becomes a ticket: Sure enough, and something a team should fix. The rulebook sends it to the tracker.
- 18Wakes a person: Critical, or serious and uncertain. A human looks before anything happens.
- 216Drafted, waits for approval: Fairly sure. A ticket is drafted and a person approves it.
- 304Routine, dropped: Sure it is routine chatter or a question. Recorded, nothing sent.
- 17Not sure, recorded: Too unsure to act and nothing forces a human. Recorded as unsure, on purpose.
When it is not sure, it says so
17 piles ended as "not sure" and 18 went to a person. Those are outcomes, not failures: a system that always has an answer is guessing some of the time.
The clearest example is a vscode issue titled "NA", with a stack of system information and no description of what went wrong. The model rated it 0.60 sure, below the line where anything happens on its own.