Portfolio · Case study · AI-assisted failure analysis

Load tests that diagnose themselves.

We built a k6 performance testing framework with an AI analysis stage. After every run it reads the raw results alongside client and gateway telemetry, groups failures by likely cause, and writes a plain-language report with root causes, severity and a prioritised fix list, so the team spends its time fixing, not triaging.

Client
Confidential enterprise
Service
Performance engineering & AI analysis
Platform
k6 · Node.js · Azure DevOps
AI models
Claude or GPT-4o

The challenge

Plenty of numbers. Very few answers.

After every load test, engineers had to work out why requests failed by hand:

  • Reading thousands of lines of raw k6 output
  • Telling real server faults apart from bad test data, throttling (HTTP 429) and client-side limits
  • Matching client latency against gateway and backend metrics in separate dashboards
  • Writing recommendations from scratch for every report

Worse, an API that returns 2xx for invalid input looks green in every standard metric: a silent failure no dashboard catches.

Our solution

Facts from code, judgement from AI.

  • Deterministic classification: a parser counts and groups every failure by type and endpoint before the AI sees anything.
  • Three layers of evidence: client metrics, API gateway telemetry and the failure profile, combined into one context.
  • Structured AI diagnosis: root causes with cited evidence, severity, and P1–P3 fixes in a fixed format.
  • Runs from the pipeline: anyone on the team can start a test and get a diagnosis without writing code.

Architecture

From test run to diagnosis report

Each run flows through five stages. If no AI key is available or the API call fails, a rule-based fallback produces the same report format, so the pipeline never ends without an answer.

  1. 01 · Load test

    k6 run

    Endpoints and test data from one JSON config; tokens generated automatically.

  2. 02 · Classify

    Failure profile

    Status codes, defect types, request rate per second and thresholds, counted in code.

  3. 03 · Context

    Combine evidence

    Client metrics, load-generator resources and API gateway telemetry are merged.

  4. 04 · Analyse

    LLM diagnosis

    Claude or GPT-4o returns root causes and fixes in a fixed JSON schema.

  5. 05 · Report

    Shareable HTML

    Published as a pipeline artifact, ready to share with the team.

Fallback No API key or AI call fails → rule-based analyser produces the same schema → report still generated.

How failures are grouped

5xx Server Error4xx Client Error4xx Data Validation Error2xx-but-invalid (API Validation Failure)429 Throttled (per second)

The “2xx-but-invalid” group catches responses that succeeded even though the request was incomplete or invalid: a defect that ordinary metrics count as a success.

Before & after

From raw output to a plain-language diagnosis

Illustrative example with sample values, not client data.

Before · raw k6 NDJSON (1 of thousands of lines)
{"type":"Point",
 "metric":"http_req_duration",
 "data":{
   "time":"2026-03-04T10:21:07Z",
   "value":2381.4,
   "tags":{
     "method":"POST",
     "name":"/api/resource",
     "status":"503"
   }
 }}

... + several thousand lines
After · AI bottleneck cardHealth: Moderate

High Backend saturation under peak load

Root cause: 5xx errors rise with request rate while gateway capacity stays low: the backend, not the gateway, is the limit.

Evidence: backend duration accounts for most of total gateway time; 503s cluster in the busiest seconds.

Fix: scale out the backend service and add caching for read-heavy endpoints.

P1effort: mediumcategory: infra

What every report contains

Executive summary

Plain-language overview and an overall health rating: Excellent, Good, Moderate or Critical.

Bottlenecks

Each with severity, root cause, the metrics behind it, a fix and expected impact.

Throughput prediction

Estimated maximum requests per second, confidence level and the limiting factor.

Prioritised fixes

P1–P3 priority, low/medium/high effort, and category: code, config, infra, caching or monitoring.

Resilience testing

Testing what happens when things go wrong

Dedicated scenarios check failure-handling behaviour, and each one classifies its own outcome.

Retry → failover

The primary endpoint fails, one retry follows, then traffic moves to a secondary endpoint. Latency is measured at each hop.

Non-retriable errors

Confirms that 4xx responses do not trigger retries or failover.

Gateway retry policy

5xx retries must stop when a 4xx interrupts them, and recover within the maximum attempt window.

Consolidated report

All scenarios merged into one pass/fail summary, with a targeted fix for each failure.

How we think

Five engineering decisions that make the AI trustworthy

01

Deterministic first, AI second

Code does the counting, so the model gets reliable facts. AI is used for interpretation and prioritisation, never arithmetic.

02

Schema-constrained output

A fixed JSON contract makes results renderable, comparable between runs, and replaceable by the fallback.

03

Evidence is required

Every bottleneck must cite the metrics behind it, so the AI’s claims can be checked.

04

“Green but wrong” detection

2xx responses to invalid payloads count as defects: bugs that ordinary dashboards miss.

05

Risk vs. fail

Only real HTTP failures break the build; latency breaches appear as warnings. No alert fatigue in CI.

Tech stack

Tools we used

Load & resilience testing

Grafana k6: scenarios, custom metrics, thresholds

Analysis & reporting

Node.js 18+, Anthropic Claude, GPT-4o via GitHub Models, HTML/CSS reports

Monitoring

PowerShell resource monitor, Azure Monitor for API Management and App Service

CI/CD

Azure DevOps YAML pipeline with typed parameters and credential masking

Auth

Azure Identity / Azure CLI, OAuth2 client credentials

Config

One JSON source of truth, with dynamic test data from Azure Blob Storage

Outcomes

What changed for the team

Zero manual triage

Every run goes from raw output to a ready-to-share diagnosis report automatically.

Hidden bug found

Uncovered an API input-validation gap: invalid payloads accepted with 2xx, which standard metrics missed.

Repeatable resilience checks

Circuit-breaker, failover and gateway retry policies are now verified on demand.

Testing for everyone

Non-developers run tests through pipeline parameters, without touching code.

Spending too long working out why tests failed?

We build test frameworks that explain their own results, so your team can go straight to the fix.

Book a free consultation