Features

Everything the dashboard measures, explained.

InferGauge answers five questions with evidence: is the app fast enough, how many users can it handle, where's the bottleneck, is performance degrading, and what does the workload cost.

Test types

Load

Steady, realistic concurrency to measure everyday performance.

Stress

Push past normal capacity to find where the app breaks.

Spike

Sudden bursts of users to see how gracefully you recover.

Endurance

Long-running load to catch drift, leaks, and slow degradation.

Core metrics

Latency (avg / p95 / p99)

Typical, bad-day, and worst-case response time. Set SLAs on p95, not the average.

Time to First Token

The wait before an answer starts, including queueing — the most user-felt number.

Inter-Token Latency

The gap between streamed tokens — rises when compute saturates.

Goodput

The share of requests meeting every SLO at once — deliberately strict.

Cost

Live spend, cost per request, and a realistic monthly projection.

Performance Score

A 0-100 composite of latency, errors, goodput, token performance, and cost.

Rule of thumb: high TTFT with healthy ITL means requests are queueing — a capacity/parallelism problem. Healthy TTFT with rising ITL means generation itself is compute-bound.

The core metaphor

Find the saturation knee.

Each point on the saturation curve is one second of the test: p95 latency against concurrent users. A flat line means headroom. The upward bend — the knee — is your real scaling ceiling, the direct answer to “how many users can we handle?”

SLA & CI/CD

Every SLA — p95 latency, TTFT, error rate, goodput, cost — is defined once in the test's YAML config and graded automatically. Anything you don't set is left “provisional” rather than silently passing.
Run --headless in CI for a summary, reports, and a pass/fail exit code. Add --baseline latest --max-regression-pct 15 to fail the build if any metric regresses more than 15% against the previous run.

Automated insights

  • Knee location and scaling ceiling
  • Rate limiting detection
  • TTFT vs. ITL bottleneck diagnosis
  • Performance drift across runs
  • Quality degradation under load
  • Unreachable endpoint / bad model diagnostics
  • Cost pressure warnings

Providers & the console

OpenAIAnthropicGroqTogether AIFireworks AIOllamaSelf-hosted / local

Teams running the console can build a test config from a form instead of hand-writing YAML — pick a provider preset, set the prompt your application actually sends, and it validates the test against your plan before generating a config. Sync results back with infergauge login and infergauge run --sync.

Ready to find your knee?

Get Started