[ local inference control plane ]

Flags are not a workflow. Profiles are.

Every inference engine speaks its own dialect of flag soup. infai gives them all one predictable structure — a profile: model + engine + config. Pick one, press enter. Run several models at once and watch every one of them from a single pane of glass.

$brew install infai Source on GitHub

llama.cpp · vLLM · one Go binary, zero daemons

[ 01 / the problem ]

Three engines.
Three dialects of flag soup.

Every runtime reinvents its own switches. Your shell history is not a configuration system.

llama.cpp

$ llama-server -m ~/models/qwen3.5-9b-q5_k_m.gguf \
    --ctx-size 65536 --n-gpu-layers 99 --flash-attn on \
    --cache-type-k bf16 --cache-type-v bf16 --jinja \
    --batch-size 2048 --ubatch-size 512 --temp 0.6 \
    --host 0.0.0.0 --port 8000

vLLM

$ vllm serve Qwen/Qwen3.5-9B --max-model-len 65536 \
    --gpu-memory-utilization 0.92 --max-num-seqs 128 \
    --enable-prefix-caching --port 8000

Same model. Same machine. Two incompatible vocabularies you're expected to memorize.

infai

$ infai
▸ qwen-coder-fast    Qwen3.5-9B  :8000
  gemma-vision       gemma-4-E4B :8001
  embed-server       nomic-v2    :8002
enter: launch · e: edit · d: delete

One muscle memory, whatever the engine underneath. That's the whole point.

[ 02 / the structure ]

One predictable shape
for every launch.

A profile is the whole idea. Define it once — infai handles everything downstream: the command line, the port, the process, the logs.

profile

qwen-coder-fast

=

model

Qwen3.5-9B-Q5_K_M.gguf

+

engine

llama.cpp

+

config

ctx 64K · ngl auto · fa on · temp 0.6

a

Saved, not typed

Profiles live in one SQLite database. No dotfiles, no YAML, no shell aliases rotting in your rc file.

b

Engine-agnostic by design

The structure doesn't change when the engine does. vLLM landed — your workflow didn't.

c

Collision-aware launches

infai knows what's already running and on which port before it starts anything. No zombie servers, no address-in-use surprises.

[ 03 / the glass ]

Many models.
One pane of glass.

A coder model on :8000, a vision model on :8001, embeddings on :8002 — all launched, monitored, and stopped from one screen. These are real screenshots, not mockups.

3.1

Configure once,
launch forever

Context size, GPU layers, KV-cache types, flash attention, sampling, extra flags — the editor exposes every knob with smart pickers, then remembers all of it. Relaunch is one keypress.

tab navigate · space toggle · ctrl+s save

infai profile editor with context size, GPU layers, cache types, flash attention and sampling fields
3.2

Watch inference
happen live

Per-run status, endpoint, uptime, and tokens-per-second. System and per-model CPU, RAM, and VRAM telemetry. Scrollable server logs. No tmux splits, no second monitor.

s stop · c clear logs · esc back

infai run screen streaming llama.cpp server logs with live system and model resource metrics

[ 04 / instrumentation ]

Everything on board.

F-01

Multi-run dashboard

Run several models concurrently. The Runs tab shows every process with status, profile, engine, uptime, and live TPS.

F-02

Model auto-discovery

Point infai at your directories. GGUF files are indexed, sized, and ready to bind to a profile instantly.

F-03

Live telemetry

System and per-model CPU, RAM, and VRAM, sampled in real time, rendered as meters right in the terminal.

F-04

Engine management

A dedicated Engines tab manages your inference-engine binaries and paths. Centralized, versioned, swappable.

F-05

Smart editors

Pickers for quantization and context units, checkboxes for toggles, sane empty-means-omit defaults everywhere.

F-06

One SQLite file

Profiles, paths, and settings in a single local database. Back it up with cp. Nothing phones home.

F-07

11+ terminal themes

Tokyonight, Everforest, Gruvbox, Rose Pine, One Dark and more — match the rest of your terminal.

F-08

Single Go binary

No runtime, no daemon, no Docker. Install it, type infai, and you're looking at your models.

[ 05 / deployment ]

Sixty seconds to first launch.

Homebrew recommended

macOS & Linux · pre-compiled · zero dependencies

Install script

Linux · installs to /usr/local/bin

Binary

Grab a pre-built binary from GitHub Releases — Linux & macOS, amd64 & arm64.

From source

Go 1.23+ and a C compiler (SQLite)

[ 06 / trajectory ]

llama.cpp today.
Every engine tomorrow.

The profile structure was built to outlive any single runtime. Here's where the control plane is headed.

shipped

llama.cpp engine

GGUF auto-detection, launch profiles, live logs, engine & path management.

shipped

Runs dashboard

Multiple concurrent models with per-run status, uptime, and live tokens-per-second — the single pane of glass.

shipped

Resource telemetry

System and per-model CPU, RAM, and GPU/VRAM meters, live in the run screen.

shipped

vLLM backend

Production-grade batched inference under the exact same profile structure. Your workflow didn't change; the engine did.

exploring

Hardware-aware autoconfig

infai reads a model's config.json and your hardware, then proposes the optimal launch configuration. You shouldn't need to know what ubatch means to get good tokens-per-second.

Your GPU is idle
right now.

$brew install infai Star it on GitHub