Portfolio · Case study · AI-assisted failure analysis
Load tests that diagnose themselves.
We built a k6 performance testing framework with an AI analysis stage. After every run it reads the raw results alongside client and gateway telemetry, groups failures by likely cause, and writes a plain-language report with root causes, severity and a prioritised fix list, so the team spends its time fixing, not triaging.
- Client
- Confidential enterprise
- Service
- Performance engineering & AI analysis
- Platform
- k6 · Node.js · Azure DevOps
- AI models
- Claude or GPT-4o
The challenge
Plenty of numbers. Very few answers.
After every load test, engineers had to work out why requests failed by hand:
- Reading thousands of lines of raw k6 output
- Telling real server faults apart from bad test data, throttling (HTTP 429) and client-side limits
- Matching client latency against gateway and backend metrics in separate dashboards
- Writing recommendations from scratch for every report
Worse, an API that returns 2xx for invalid input looks green in every standard metric: a silent failure no dashboard catches.
Our solution
Facts from code, judgement from AI.
- Deterministic classification: a parser counts and groups every failure by type and endpoint before the AI sees anything.
- Three layers of evidence: client metrics, API gateway telemetry and the failure profile, combined into one context.
- Structured AI diagnosis: root causes with cited evidence, severity, and P1–P3 fixes in a fixed format.
- Runs from the pipeline: anyone on the team can start a test and get a diagnosis without writing code.
Architecture
From test run to diagnosis report
Each run flows through five stages. If no AI key is available or the API call fails, a rule-based fallback produces the same report format, so the pipeline never ends without an answer.
- 01 · Load test
k6 run
Endpoints and test data from one JSON config; tokens generated automatically.
- 02 · Classify
Failure profile
Status codes, defect types, request rate per second and thresholds, counted in code.
- 03 · Context
Combine evidence
Client metrics, load-generator resources and API gateway telemetry are merged.
- 04 · Analyse
LLM diagnosis
Claude or GPT-4o returns root causes and fixes in a fixed JSON schema.
- 05 · Report
Shareable HTML
Published as a pipeline artifact, ready to share with the team.
Fallback No API key or AI call fails → rule-based analyser produces the same schema → report still generated.
How failures are grouped
The “2xx-but-invalid” group catches responses that succeeded even though the request was incomplete or invalid: a defect that ordinary metrics count as a success.
Before & after
From raw output to a plain-language diagnosis
Illustrative example with sample values, not client data.
{"type":"Point",
"metric":"http_req_duration",
"data":{
"time":"2026-03-04T10:21:07Z",
"value":2381.4,
"tags":{
"method":"POST",
"name":"/api/resource",
"status":"503"
}
}}
... + several thousand linesHigh Backend saturation under peak load
Root cause: 5xx errors rise with request rate while gateway capacity stays low: the backend, not the gateway, is the limit.
Evidence: backend duration accounts for most of total gateway time; 503s cluster in the busiest seconds.
Fix: scale out the backend service and add caching for read-heavy endpoints.
What every report contains
Executive summary
Plain-language overview and an overall health rating: Excellent, Good, Moderate or Critical.
Bottlenecks
Each with severity, root cause, the metrics behind it, a fix and expected impact.
Throughput prediction
Estimated maximum requests per second, confidence level and the limiting factor.
Prioritised fixes
P1–P3 priority, low/medium/high effort, and category: code, config, infra, caching or monitoring.
Resilience testing
Testing what happens when things go wrong
Dedicated scenarios check failure-handling behaviour, and each one classifies its own outcome.
Retry → failover
The primary endpoint fails, one retry follows, then traffic moves to a secondary endpoint. Latency is measured at each hop.
Non-retriable errors
Confirms that 4xx responses do not trigger retries or failover.
Gateway retry policy
5xx retries must stop when a 4xx interrupts them, and recover within the maximum attempt window.
Consolidated report
All scenarios merged into one pass/fail summary, with a targeted fix for each failure.
How we think
Five engineering decisions that make the AI trustworthy
Deterministic first, AI second
Code does the counting, so the model gets reliable facts. AI is used for interpretation and prioritisation, never arithmetic.
Schema-constrained output
A fixed JSON contract makes results renderable, comparable between runs, and replaceable by the fallback.
Evidence is required
Every bottleneck must cite the metrics behind it, so the AI’s claims can be checked.
“Green but wrong” detection
2xx responses to invalid payloads count as defects: bugs that ordinary dashboards miss.
Risk vs. fail
Only real HTTP failures break the build; latency breaches appear as warnings. No alert fatigue in CI.
Tech stack
Tools we used
Load & resilience testing
Grafana k6: scenarios, custom metrics, thresholds
Analysis & reporting
Node.js 18+, Anthropic Claude, GPT-4o via GitHub Models, HTML/CSS reports
Monitoring
PowerShell resource monitor, Azure Monitor for API Management and App Service
CI/CD
Azure DevOps YAML pipeline with typed parameters and credential masking
Auth
Azure Identity / Azure CLI, OAuth2 client credentials
Config
One JSON source of truth, with dynamic test data from Azure Blob Storage
Outcomes
What changed for the team
Zero manual triage
Every run goes from raw output to a ready-to-share diagnosis report automatically.
Hidden bug found
Uncovered an API input-validation gap: invalid payloads accepted with 2xx, which standard metrics missed.
Repeatable resilience checks
Circuit-breaker, failover and gateway retry policies are now verified on demand.
Testing for everyone
Non-developers run tests through pipeline parameters, without touching code.
Spending too long working out why tests failed?
We build test frameworks that explain their own results, so your team can go straight to the fix.