AIOpsAnalystHow it was measured
Every result here is checked against labels that nobody working on this project chose. The method comes first, then the number, then what it does and does not show.
Does it agree with the people who maintain these repositories?
Method: 111 GitHub issues from transformers, next.js, pytorch, langchain, vscode and kubernetes, each already labelled by that repository's own maintainers (bug, feature and similar). Its labels were compared with theirs, without either side seeing the other.
0.631
Cohen's kappa, 95% interval 0.493 to 0.763
Raw agreement 82%. Kappa corrects that for the agreement two raters would reach by chance; 0.6 to 0.8 is usually called substantial.
| Label | Precision | Recall | Issues |
|---|---|---|---|
| defect | 0.94 | 0.82 | 78 |
| feature request | 0.84 | 0.81 | 32 |
Does it surface what the operators flagged?
Method: 10,000 lines of the Blue Gene/L supercomputer log, every line labelled by the operations staff who ran the machine as routine or as an alert. The labels never reach the model. A pile it calls routine is hidden; anything else is shown to a person.
14 of 40
piles shown to a person, instead of 10,000 lines
Every pile that held an operator alert was shown (2 of 2), and all 66 flagged lines with them: recall 1.00.
The line-level figures swing between runs because one pile, a kernel message about core dumps, is most of the log. Operators called it routine; as text it reads like a crash. This run hid it, so 91% of routine lines were suppressed; an earlier run showed it and suppressed 9%. The pile count is the number that holds.
A better score, rejected
Method: the labelled issues were split in half with a fixed seed. Three rules were written against the disagreements in one half, and the other half decided whether they helped.
| Prompt | Kappa on the held-out half, per run |
|---|---|
| 1.1, in use | 0.565, 0.584 |
| 1.2, revised | 0.649, 0.604, 0.764 |
The revision won every run. It was still rejected, because of what it cost elsewhere:
| 1.1 | 1.2 | |
|---|---|---|
| alert recall on the supercomputer log | 1.00, 3 of 3 incident types | 0.998, 2 of 3 |
| questions reaching the right cluster, hit@1 | 0.90 | 0.80 |
| clusters escalated to a person | 5, three of them critical | 0 |
| verdicts using the critical severity | 3 | 0 |
The last row. Without critical, the one rule that escalates whatever the confidence never fires.
Can you ask it questions?
Method: 20 questions written before the search was built and never edited, such as "when did the SSH password failures start". A hit means the right pile came back first.
0.80
right pile first, 0.90 within the top three
Counting and timing questions are answered from the database, not by a model. The one question it cannot answer uses words the corpus never does: what happened with the Tomcat connector
.
Where it falls short
- Stack traces split apart. Logs are read a line at a time, so a Linux kernel stack trace becomes one pile per line of the trace. 13 of the 18 piles sent to a person are single frames of kernel stack traces rather than whole problems.
- A lost alert type. The supercomputer log was cut to 10,000 lines by taking every eighth line, without looking at the labels. One alert type appeared on a single line and fell out of the sample. Keeping it would have meant choosing lines by their answers.
- The tracker is a sample. The rulebook would file 350 tickets; the demo Jira holds 26, so as not to flood it.
- Runs vary. Free model tiers route each run to a different mix of models, so the same prompt on the same issues has scored anywhere in the ranges above.