InferGauge Beta — Quick Start

Thanks for helping test InferGauge. This covers everything from "I have a download" to "I can see my first test result in the dashboard." It should take about 5 minutes.

The console

InferGauge's console lives at:

https://console.infergauge.com

You'll create an account there in step 3, and you'll come back to it after every test run to see results.

1. Requirements

Just a terminal — Terminal.app on macOS, PowerShell on Windows. No Python, no package manager, nothing else to install first. The binary is self-contained.

2. Download and set it up

Go to https://www.infergauge.com and download the build for your operating system.

macOS

That gets you a file like infergauge-macos-arm64.tar. Open a terminal, move to wherever it downloaded (usually your Downloads folder), and extract it there rather than by double-clicking in Finder — Finder's Archive Utility flags the file as quarantined, and the beta build isn't code-signed yet, so you'll hit a Gatekeeper warning later if you go that route.

cd ~/Downloads
tar -xvf infergauge-macos-arm64.tar

You should see x infergauge — that's the program, extracted. Make it runnable and confirm it works:

chmod +x infergauge
./infergauge --version

Don't double-click the infergauge file itself. It's a command-line program with no window of its own. Double-clicking it opens a Terminal window, runs it with no instructions, prints a usage error, and closes the window again — which looks like a crash but isn't. Always run it from a terminal you opened yourself, using the steps above.

To run it as just infergauge from any folder instead of ./infergauge every time:

sudo mv infergauge /usr/local/bin/

Windows

Download infergauge-windows-x86_64.zip and extract it — right-click it in File Explorer and choose Extract All, or from PowerShell:

cd $HOME\Downloads
Expand-Archive infergauge-windows-x86_64.zip -DestinationPath .

That produces infergauge.exe. Confirm it runs:

.\infergauge.exe --version

You'll likely see a blue "Windows protected your PC" SmartScreen warning. Expected for an unsigned beta build — click More info, then Run anyway. Safe to do if you trust where you got the file.

Don't double-click infergauge.exe in File Explorer. Same issue as macOS: it opens a console window, runs with no arguments, prints a usage error, and the window closes before you can read it. Always run it from a PowerShell window you opened yourself.

To run it as just infergauge from anywhere, add its folder to PATH via Settings → System → About → Advanced system settings → Environment Variables. Or skip that and just cd to the folder each time and run .\infergauge.exe.

If you tested an earlier beta that used pip install, you likely have a leftover copy that can silently take priority over the new binary — same command, different version, no error to tell you which one ran. Clear it out before continuing.

macOS / Linux:
rm -rf ~/infergauge-env
Windows — worth doing properly, not just deleting a folder. Pip can leave a copy in more than one place: inside a virtual environment (e.g. infergauge-env), and separately at user level in something like AppData\Roaming\Python\Python3xx\Scripts if pip install was ever run outside a venv. Find everything first:
Get-Command infergauge -All | Format-Table CommandType, Name, Version, Source
This lists every infergauge PowerShell can see and where each one lives. Two things about the output are easy to misread:
The Version column can show 0.0.0.0 for pip-installed shims. That's missing file metadata in the shim itself, not the tool's real version — it tells you nothing. Check the real version by calling each path directly: & "C:\path\from\Source\infergauge.exe" --version. If that prints a usage banner instead of a number, cross-check with pip show infergauge (run with that specific environment active) to find its actual installed version.

A venv copy and a user-level copy are removed differently. For a venv, activate it and run pip uninstall infergauge before deleting the folder — that also cleans up any secondary shim it created (some builds have shipped a second one named igauge.exe). For a user-level install, run pip uninstall infergauge without any venv active, so pip targets the right location.
Don't reach for rm -rf in PowerShell. rm is aliased to Remove-Item, but PowerShell doesn't support bundled single-letter flags — -rf isn't valid and the command will error out rather than delete anything. That's a safe failure, not a bug: if you were about to delete a venv you actually still needed (e.g. to upgrade it first), this stops you. Use the real parameter names instead:
Remove-Item -Recurse -Force $HOME\infergauge-env
Then open a new PowerShell window and re-run the Get-Command line — it should show exactly one infergauge, pointing at the binary you just extracted.

3. Create your account

Open https://console.infergauge.com in your browser and click Create account. Free tier needs no card and no license key: 25 concurrent users, 120 seconds per run, 10 runs a month.

infergauge login --server https://console.infergauge.com

Enter the email and password you just signed up with. This stores a token on your machine (~/.infergauge/session.json — on Windows, %USERPROFILE%\.infergauge\session.json) so future runs can sync automatically. You only need to do this once per machine.

5. Run your first test

The easiest path is through the browser:

In the console, go to the New test tab
Adjust anything you want (users, duration, provider), then click Check plan & generate config
Click Download infergauge.yaml (or Copy config and paste it into a file named infergauge.yaml yourself)
In your terminal, cd to wherever that file ended up — if you used the download button it's almost certainly your Downloads folder:
```
cd ~/Downloads
```
Run it:
```
infergauge run infergauge.yaml --sync
```

The command has to run in the same folder as the config file. If you get config not found: infergauge.yaml, you're in the wrong directory — check with ls (dir on Windows) whether the file is actually there. You can also pass a full path instead: infergauge run ~/Downloads/infergauge.yaml --sync.

A dashboard opens in your browser showing live metrics as the test runs. When it finishes, the result also appears in the console's Overview and Runs tabs — that's the --sync part working.

Getting a non-provisional score

You may see "score is provisional — only 75% of weight graded". That's not an error: InferGauge only grades a category when your config defines the threshold for it, rather than inventing one. The console's form doesn't yet expose two of them, so add these to the sla: block by hand for a fully graded score:

sla:
  itl_p95_ms: 100.0
  max_cost_usd: 5.0

Testing against your own model API

Everything above runs against InferGauge's built-in simulator — good for confirming the tool itself works, but it tells you nothing about your real endpoint. To test a live model API, the sequence is: create a key at your provider, export it into your terminal session, then run the config.

⚠ Never put your API key in infergauge.yaml

The key belongs in an environment variable, and nowhere else. Configs get committed to git, attached to bug reports, pasted into Slack, and shared with teammates — a key inside one will eventually leak, and a leaked key can be used to run up charges on your account.

There is no field in the config for a key, and this is deliberate. If you find yourself wanting to add one, export it instead. And if you've already pasted a key into a config file at any point, treat it as compromised: rotate it at your provider now, rather than deleting the file and hoping.

1. Create an API key in your model provider's own console (OpenAI, Anthropic, or whichever you're testing). InferGauge never creates or stores these — it just uses the one you give it.

2. Export it into your terminal.

macOS / Linux:

export OPENAI_API_KEY="your-key-here"

Windows (PowerShell):

$env:OPENAI_API_KEY = "your-key-here"

Use whichever variable name matches your provider — ANTHROPIC_API_KEY for Anthropic, and so on. If a run fails with an authentication error, the variable name is the first thing to check.

This lasts only for the current terminal window — the same catch as activating a virtual environment. Open a new tab, and the key is gone and your run will fail to authenticate. Re-export it, or set it permanently: add the export line to ~/.zshrc on macOS, or on Windows use [Environment]::SetEnvironmentVariable("OPENAI_API_KEY", "your-key-here", "User") and open a fresh window.

3. Generate a config pointed at your provider. In the console's New test tab, change provider from Simulator to your actual provider and fill in the model name it asks for, then Check plan & generate config as before. The downloaded YAML will carry a provider: block matching your choice instead of kind: simulator. It will not contain your key — it doesn't need to, because InferGauge reads that from the environment at run time.

4. Run it exactly as before:

infergauge run infergauge.yaml --sync

Cost and rate limits are real now. Start with a short duration and a handful of users before running anything larger. The max_cost_usd value in the sla: block is worth setting deliberately here rather than leaving at a default — it's your guard against a runaway bill.

Need more than the Free plan?

Free covers 25 concurrent users, 120 seconds per run, and 10 runs a month — plenty for trying things out. If you need spike or endurance tests, self-hosted endpoints, or unlimited runs, email support (link in the console) and we'll send you a license token.

Once you have it:

Sign in to the console
Click License (top right)
Paste the token in

Your account upgrades immediately — no restart, no waiting.

Getting an update later

Re-download from https://www.infergauge.com and repeat step 2 — extract over the old file, and on macOS chmod +x again:

cd ~/Downloads
tar -xvf infergauge-macos-arm64.tar
chmod +x infergauge

(Windows: re-run Expand-Archive over the old files.)

If you'd moved the old copy onto your PATH, move the new one there too to replace it.

Confirm it actually took:

infergauge --version

Plus, to check for a lingering older copy:

macOS:

which -a infergauge

Windows:

Get-Command infergauge -All | Format-Table CommandType, Name, Version, Source

(where.exe infergauge also works for a quick path list, but Get-Command is worth using here — it shows the type of each match, which tells you whether you're looking at a pip shim or a real binary. Note its Version column reads 0.0.0.0 for pip-installed shims regardless of actual version — that's a metadata gap, not the answer; get the real number with & "<path from Source>" --version.)

Whoever told you about the update can tell you which version number to expect. If this prints an older number, the command above tells you why — it lists every infergauge your system can see, in the order your shell would actually pick them. If it shows more than one, the first one wins regardless of which is newer.

Troubleshooting

A console window flashed open, showed a usage error, and closed You double-clicked the program file directly. It ran with no subcommand and the window closed itself. Nothing is broken — open a terminal yourself and run ./infergauge --help (macOS) or .\infergauge.exe --help (Windows).

zsh: permission denied: ./infergauge (macOS) The execute bit wasn't preserved during extraction. Run chmod +x infergauge.

"infergauge cannot be opened because the developer cannot be verified" (macOS) Gatekeeper blocking an unsigned beta build. Extracting with tar -xvf rather than double-clicking in Finder usually avoids this. If it's already happened and you trust the source:

xattr -d com.apple.quarantine infergauge

"Windows protected your PC" (Windows) SmartScreen blocking an unsigned beta build. Click More info, then Run anyway — expected for this build, not a sign anything is wrong.

The file you extracted has disappeared Archive tools aren't always predictable about where they put things. Re-extract from the terminal/PowerShell instead of double-clicking, which tells you exactly what came out and where. On macOS, if it's still missing: sudo find / -name "infergauge" -type f 2>/dev/null.

infergauge --version shows an older version than expected You have two copies and the wrong one is winning. Run which -a infergauge (macOS) or Get-Command infergauge -All (Windows) — the first entry is the one that actually runs. Common cause: a leftover install from an earlier beta round. See the callout in step 2 about clearing it out.

Windows: Get-Command infergauge -All shows a Version of 0.0.0.0 That's normal for pip-installed shims and doesn't mean anything is broken — the shim's file metadata just doesn't carry a version number. It tells you nothing about which build you're actually running. Get the real number by calling that specific path directly: & "C:\the\path\from\Source" --version. If that prints a usage banner instead of a version, check pip show infergauge in that shim's environment instead.

Windows: rm -rf fails with a parameter error PowerShell aliases rm to Remove-Item, but doesn't support bundled Unix-style flags — -rf isn't a valid parameter there. Use: Remove-Item -Recurse -Force <path>. Worth treating this failure as a prompt to double-check you're deleting the right thing before retrying, rather than just fixing the syntax and moving on.

Windows: an old install keeps coming back after you thought you removed it Pip can install to more than one place — inside a virtual environment, and separately at user level (something like AppData\Roaming\Python\Python3xx\Scripts) if pip install was ever run without a venv active. Deleting one doesn't touch the other. Run Get-Command infergauge -All to see every location before deciding what to remove, and use pip uninstall infergauge in the correct environment for each one rather than deleting folders by hand — some builds also register a second command name (igauge.exe) that a plain uninstall of infergauge alone won't catch.

command not found: infergauge / 'infergauge' is not recognized... Either the binary isn't on your PATH, or you're in a new window since moving it. Use ./infergauge (macOS) or .\infergauge.exe (Windows) from the folder you extracted it into instead.

A run against a real API fails with an authentication error Almost always the API key environment variable. Three things to check, in order: that you exported it in this terminal window (it doesn't carry across tabs), that the variable name matches your provider (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.), and that the key is still valid at the provider's end. Confirm the variable is actually set:

echo $OPENAI_API_KEY          # macOS / Linux
$env:OPENAI_API_KEY           # Windows PowerShell

Empty output means it isn't set in this session — re-export it.

Your terminal shows quote> or dquote> and won't accept new commands You're stuck mid-way through an unfinished quoted command, usually from a stray " or '. Press Ctrl+C to cancel it, then retry.

--sync requested but not signed in Run infergauge login --server https://console.infergauge.com again.

Nothing happens when you type into the terminal If infergauge serve (or anything else) is running in that same window, it's busy being that program. Open a new tab/window for anything else.

login failed - cannot reach ... [Errno 61] Connection refused Double-check you typed https://console.infergauge.com correctly. If it's still failing, the service may be down — check with support.

login failed - Incorrect email or password when you're sure it's right Confirm you're signing in with the same email you used to create the account at https://console.infergauge.com.

config not found: infergauge.yaml You're running the command from a different folder than the config file. cd to where it lives, or pass a full path: infergauge run ~/Downloads/infergauge.yaml --sync.

The console looks out of date after an update Browsers cache pages aggressively. Hard-refresh with Cmd+Shift+R (macOS) or Ctrl+Shift+R (Windows). Still stale? Try a private/incognito window.

login failed - Too many attempts. Try again later. [HTTP 429] After several failed sign-ins the account locks for 15 minutes — deliberate anti-brute-force protection, no way to skip it.

Questions or something doesn't match this guide? Reach out to support through the console.