InferGauge User Guide (Team Edition)

Local testing, Ollama testing, and how to read the dashboard. For v0.6.0+.

1. What InferGauge does

InferGauge load-tests AI applications and shows results on a live dashboard. It answers five questions with evidence: Is the app fast enough? How many users can it handle? Where is the bottleneck? Is performance degrading? What does the workload cost?

Everything runs on your machine — prompts, keys, and results never leave it. Every test is a YAML config, so any result can be re-run and verified by anyone.

2. Setup (Windows)

Check Python 3.10+ with python --version (install from python.org, tick "Add Python to PATH").

pip install infergauge-0.7.0-py3-none-any.whl
# or from the source zip (note the NESTED folder from Windows unzip):
cd Downloads\infergauge\infergauge
pip install .

If infergauge is "not recognized" afterward, use python -m infergauge ... everywhere — identical behavior.

2b. macOS install

The wheel is platform-independent (py3-none-any), so install failures on macOS are always environment issues. This one sequence avoids all of them:

python3 --version                 # must be 3.10+ (macOS ships 3.9 - see below)
python3 -m venv ~/infergauge-env
source ~/infergauge-env/bin/activate
pip install ~/Downloads/infergauge-0.7.0-py3-none-any.whl
infergauge init -y && infergauge run

Re-activate later with source ~/infergauge-env/bin/activate.

3. First test in 60 seconds (no API key)

python -m infergauge init -y
python -m infergauge run

Browser opens to http://localhost:8710 (open manually if not — the dashboard only exists while a test runs or replays). Keep the terminal open. Try the +/− user controls and Stop test; every manual action is annotated on the charts.

4. Testing with Ollama (free real model)

# install Ollama from ollama.com, then:
ollama pull llama3.2
ollama run llama3.2 "say hello"    # sanity check
ollama list                        # note the EXACT model name

Create the config (must be -Encoding ASCII — PowerShell's UTF8 adds a BOM):

@"
name: Ollama Load Test
application: Local Llama
test_type: load
duration_s: 90
users: 3
ramp_up_s: 15
think_time_s: 1.0

provider:
  kind: openai-compatible
  base_url: http://localhost:11434/v1
  api_key_env: NONE
  model: llama3.2
  stream: true
  max_tokens: 150
  timeout_s: 120
  prompt: "Explain in two sentences why the sky is blue."

sla:
  p95_latency_ms: 20000
  ttft_p95_ms: 5000
  error_rate_pct: 5
  min_goodput_pct: 80

pricing:
  input_per_1m: 0
  output_per_1m: 0
"@ | Set-Content -Encoding ASCII ollama-test.yaml

python -m infergauge run ollama-test.yaml

Expect: the first request can take up to a minute (model loading into RAM — the 120s timeout covers it), and TTFT balloons at 3 users because Ollama serves one request at a time by default. That's the expected finding, not a bug.

The parallel experiment (quantify a real fix):

$env:OLLAMA_NUM_PARALLEL = "3"
# restart Ollama (quit from tray or: taskkill /IM ollama.exe /F), relaunch, then:
python -m infergauge run ollama-test.yaml
python -m infergauge compare

TTFT collapses → queueing was the bottleneck. TTFT better but ITL worse → CPU now shared across generations: a measured capacity trade-off.

5. Everyday commands

5b. Using the web console to build a config

If your team runs the console (infergauge serve), you don't have to hand-write YAML. Sign in, open New test, fill in the form, and click Check plan & generate config. It validates the test against your plan first, then gives you a config to download.

Two fields are worth attention:

• Provider — presets for OpenAI, Anthropic, Groq, Together AI, Fireworks AI and Ollama fill in the base URL, a model, and the usual API-key variable name.
• Prompt — set this to the prompt your application really sends. Token counts, time-to-first-token and cost all scale with the prompt, so testing with a generic one tells you very little about your own workload. Multi-line prompts are fine.

Then, on the machine you want to measure:

infergauge login --server <your-console-url>   # once per machine
infergauge run infergauge.yaml --sync

--sync is what sends the result back to the console. The command must run in the same folder as the config, or pass the full path to it.

6. Understanding the dashboard

Header — test identity, elapsed, live users, req/s, status pill. +/− changes users mid-test; Stop ends it; both are marked on charts (amber lines).

Performance Score dial — 0–100 composite: Latency 30%, Errors 25%, Goodput 20%, Token performance 15%, Cost 10%. Components listed below the dial explain the number. 80+ healthy / 55–79 degraded / <55 critical.

Response card — avg/p95/p99: typical, bad-day, worst-case latency (set SLAs on p95, not averages). TTFT p95: wait before the answer starts, includes queueing — the most user-felt number. ITL p95: gap between tokens while streaming — rises when compute saturates.

Rule of thumb: high TTFT + healthy ITL = queueing (capacity/parallelism). Healthy TTFT + rising ITL = generation is compute-bound. Insights apply this automatically.

Load card — active/target users, in-flight, completed, error rate, Goodput (share of requests meeting EVERY SLO — deliberately strict) and Quality (sampled check pass-rate, when configured).

Tokens card — in/out totals, live tokens/sec, avg output/request (watch it drift up on long tests — context growth is a silent cost driver).

Cost card — live spend, $/request, projected monthly (assumption stated: current rate 24/7 — a planning ceiling, not a bill). Zero for local models.

Latency & load timeline — cyan p95, dim avg, violet users (right axis), red dashed SLA, amber event lines. Read: where does cyan cross red, and at what violet level?

Saturation curve — each dot is one second (newer = brighter): p95 vs users. Flat = headroom. The upward bend (the knee) is your scaling ceiling — the answer to "how many users can we handle?"

SLA panel — one PASS/FAIL row per threshold; these same verdicts drive CI exit codes.

Insights — rule-based findings with evidence and actions: knee location, rate limiting, TTFT-vs-ITL diagnosis, drift, quality-under-load degradation, unreachable-endpoint/bad-model diagnostics, cost pressure.

Errors & retries — rate limit (429): throttled → raise quota/backoff. server error (5xx): provider failing under load. timeout: cold start/overload/limit too tight. connection: endpoint down or wrong base_url (Ollama: http://localhost:11434/v1). other (4xx): usually wrong model name (ollama list) or missing key.

Exports — standalone HTML report (share anywhere) + JSON.

7. Worked example

Real first Ollama laptop run: 6 ok / 0 errors, avg 33.5s, TTFT p95 33.7s, ITL p95 150ms, goodput 0%, score 28, insight "Slow time-to-first-token, healthy streaming."

Reading: zero errors + healthy ITL = the model generates fine (~7 tok/s). Huge TTFT + fine ITL = the queueing signature — requests waited in line. FAILED verdict = the tool honestly reporting a laptop can't meet cloud-grade SLAs at 3 users: a capacity finding, not an error. A second (warm) run improved ~30% — re-run and compare.

8. Troubleshooting

9. Glossary

Goodput — share of requests meeting every SLO at once; the number to watch. · TTFT — wait before the answer begins (includes queueing). · ITL — gap between streamed tokens. · p95 — "95% of requests were faster than this." · Saturation knee — concurrency where latency inflects; your ceiling. · SLA/SLO — a measurable promise InferGauge turns into automatic PASS/FAIL.

Free tier & accounts. The Free plan allows up to 25 concurrent users, 120 s per run, and 10 runs per calendar month. Simulator tests are unlimited-by-account (no sign-in needed). Runs against a real endpoint on the Free plan require a free account and a live console authorisation (infergauge login) — the console keeps the monthly count. Paid plans (Developer, Team, Enterprise) raise or remove these limits and keep working offline for a 7-day grace window; see COMMERCIAL.md for the table.

Authoring test configs — the full reference

Every test is one YAML file. The complete schema, annotated (this is examples/simulator.yaml — runnable as-is with no API key):

# Start here: a load test against the built-in simulator. No API key, no cost,
# no network - it models realistic latency, tokens, and 429s so you can learn
# the dashboard instantly.  Run:  infergauge run examples/simulator.yaml
name: Simulator - Load Test
application: Example AI Application
test_type: load          # try also: stress | spike | endurance
duration_s: 120
users: 50
ramp_up_s: 30
think_time_s: 1.0

provider:
  kind: simulator
  model: gpt-4.1 (simulated)
  sim_ttft_ms: 320             # simulated time-to-first-token
  sim_itl_ms: 16               # simulated inter-token latency
  sim_output_tokens: 180
  sim_input_tokens: 950
  sim_saturation_users: 700    # latency degrades beyond this
  sim_rate_limit_users: 880    # 429s begin beyond this

sla:                # every threshold you set is graded; anything omitted leaves the score "provisional"
  p95_latency_ms: 8000
  ttft_p95_ms: 2000        # time to first token
  itl_p95_ms: 60           # inter-token latency (streaming smoothness)
  error_rate_pct: 1.0
  min_goodput_pct: 90
  max_cost_usd: 5.0        # fail the run if total cost exceeds $5

guardrails:         # HARD STOPS - abort the run in flight (SLAs only report)
  max_spend_usd: 5.0
  abort_on_rate_limit_pct: 25

pricing:            # USD per 1M tokens, for cost tracking
  input_per_1m: 3.0
  output_per_1m: 15.0

projection:         # realistic monthly cost from measured cost/request
  requests_per_day: 50000       # your expected production volume; omit -> 24/7 ceiling only
  days_per_month: 30.4167       # 365/12 = 730 hours

Rules of thumb:

• Set every SLA you care about. InferGauge grades only thresholds you define; omitted ones leave the score marked provisional.
• Paid endpoint ⇒ guardrails + pricing. guardrails.max_spend_usd and abort_on_rate_limit_pct stop a run in flight; pricing makes the cost readout real; projection.requests_per_day turns it into a realistic monthly estimate instead of the 24/7 ceiling.
• One config, many environments. INFERGAUGE_USERS, INFERGAUGE_DURATION_S, INFERGAUGE_MODEL, INFERGAUGE_BASE_URL etc. override the file at run time.

Ready-made variants in examples/: simulator.yaml (start here), ollama.yaml (free local), anthropic-claude.yaml (Claude), real-endpoint.yaml (any OpenAI-compatible API), demo-stress.yaml, demo-spike.yaml.