[ local inference control plane ]

Flags are not a workflow. Profiles are.

Every inference engine speaks its own dialect of flag soup. infai gives them all one predictable structure — a profile: model + engine + config. Discover or download a model, get a hardware-fit recommendation, press enter, and watch every run from a single pane of glass.

Experience infai

llama.cpp · vLLM · one Go binary, zero daemons

[ watch it in action ]

See infai take
the wheel.

A quick tour of the control plane for local inference.

[ 01 / the problem ]

Three engines.
Three dialects of flag soup.

Every runtime reinvents its own switches. Your shell history is not a configuration system.

llama.cpp

$ llama-server -m ~/models/qwen3.5-9b-q5_k_m.gguf \
    --ctx-size 65536 --n-gpu-layers 99 --flash-attn on \
    --cache-type-k bf16 --cache-type-v bf16 --jinja \
    --batch-size 2048 --ubatch-size 512 --temp 0.6 \
    --host 0.0.0.0 --port 8000

vLLM

$ vllm serve Qwen/Qwen3.5-9B --max-model-len 65536 \
    --gpu-memory-utilization 0.92 --max-num-seqs 128 \
    --enable-prefix-caching --port 8000

Same model. Same machine. Two incompatible vocabularies you're expected to memorize.

infai

$ infai
▸ qwen-coder-fast    Qwen3.5-9B  :8000
  gemma-vision       gemma-4-E4B :8001
  embed-server       nomic-v2    :8002
enter: launch · e: edit · d: delete

One muscle memory, whatever the engine underneath. That's the whole point.

[ 02 / the structure ]

One predictable shape
for every launch.

A profile is the whole idea. Define it once — infai handles everything downstream: the command line, the port, the process, the logs.

profile

qwen-coder-fast

=

model

Qwen3.5-9B-Q5_K_M.gguf

+

engine

llama.cpp

+

config

ctx 64K · ngl auto · fa on · temp 0.6

a

Saved, not typed

Profiles live in one SQLite database. No dotfiles, no YAML, no shell aliases rotting in your rc file.

b

Engine-agnostic by design

The structure doesn't change when the engine does. vLLM landed — your workflow didn't.

c

Collision-aware launches

infai knows what's already running and on which port before it starts anything. No zombie servers, no address-in-use surprises.

[ 03 / the glass ]

Many models.
One pane of glass.

A coder model on :8000, a vision model on :8001, embeddings on :8002 — all launched, monitored, and stopped from one screen. These are real screenshots, not mockups.

3.1

Configure once,
launch forever

Context size, GPU layers, KV-cache types, flash attention, sampling, extra flags — the editor exposes every knob with smart pickers, then remembers all of it. Relaunch is one keypress.

tab navigate · space toggle · ctrl+s save

infai profile editor with context size, GPU layers, cache types, flash attention and sampling fields
3.2

Watch inference
happen live

Per-run status, endpoint, uptime, and tokens-per-second. System and per-model CPU, RAM, and VRAM telemetry. Scrollable server logs. No tmux splits, no second monitor.

s stop · c clear logs · esc back

infai run screen streaming llama.cpp server logs with live system and model resource metrics
3.3

Find and import
the right model

Search Hugging Face, choose a quantization, or scan local GGUF and SafeTensors files. Provenance and metadata stay attached.

3.4

Know the fit
before launch

infai reads your hardware and estimates memory fit, context limits, and engine requirements before starting a server.

3.5

One profile,
any engine

Engine-aware options keep llama.cpp and vLLM settings in the same workflow, without memorizing a new flag dialect.

[ 04 / instrumentation ]

Everything on board.

F-01

Multi-run dashboard

Run several models concurrently. The Runs tab shows every process with status, profile, engine, uptime, and live TPS.

F-02

Model intelligence

GGUF and SafeTensors files are indexed with metadata, quantization, architecture, context limits, and multimodal projector detection.

F-03

Live telemetry

System and per-model CPU, RAM, and VRAM, sampled in real time, rendered as meters right in the terminal.

F-04

Engine management

A dedicated Engines tab manages your inference-engine binaries and paths. Centralized, versioned, swappable.

F-05

Engine-aware editors

Each profile exposes the right options for llama.cpp or vLLM, with hardware-fit recommendations before launch and explainable warnings.

F-06

One SQLite file

Profiles, paths, and settings in a single local database. Back it up with cp. Nothing phones home.

F-07

11+ terminal themes

Tokyonight, Everforest, Gruvbox, Rose Pine, One Dark and more — match the rest of your terminal.

F-08

Single Go binary

No runtime, no daemon, no Docker. Install it, type infai, and you're looking at your models.

F-09

Hugging Face imports

Search the Hub, choose a GGUF quantization, and download models directly into your library with resumable, atomic transfers.

F-10

Explainable fit checks

Typed hardware inventory and memory estimates show whether a model fits, why, and which launch settings will work best.

[ 05 / deployment ]

Sixty seconds to first launch.

Homebrew recommended

macOS & Linux · pre-compiled · zero dependencies

Linux package manager recommended

Add the infai package repository for your distribution.

Debian Ubuntu Fedora openSUSE

Install script

Linux · installs to /usr/local/bin

[ 06 / shipped ]

The whole local workflow.
Already on board.

From finding a model to launching a fit-checked profile, infai keeps the work in one predictable control plane.

shipped

llama.cpp engine

GGUF auto-detection, launch profiles, live logs, engine & path management.

shipped

Runs dashboard

Multiple concurrent models with per-run status, uptime, and live tokens-per-second — the single pane of glass.

shipped

Resource telemetry

System and per-model CPU, RAM, and GPU/VRAM meters, live in the run screen.

shipped

vLLM backend

Production-grade batched inference under the exact same profile structure. Your workflow didn't change; the engine did.

shipped

Hugging Face model imports

Search repositories, select quantizations, and download GGUF or SafeTensors models with resumable and atomic transfers.

shipped

Typed model intelligence

Model metadata, provenance, architecture details, and hardware-fit estimates turn a file on disk into a launch-ready profile.

shipped

Hardware-fit recommendations

infai reads model metadata and your hardware, estimates memory fit, and proposes a launch configuration with context and engine guidance before you start.

Your GPU is idle
right now.

$brew install infai Star it on GitHub