Architecture & Safety

Safe Bushpilot — No Crashes, No Guessing

Coding agents send large system prompts and repository context. On unified memory, guessing a quantization or context size often leads to swap thrashing or an unresponsive Mac. Bushpilot is built around safety guardrails:

Live hardware detection

Reads your chip and currently-free unified memory via sysctl and kernel memory counters (kern.memorystatus_level).

Safe model filtering

Greys out models that don't fit your live free memory and computes a safe context size with OS headroom reserved.

Speed estimate upfront

Shows estimated tok/s (from chip memory bandwidth and calibrated family factors) before you commit to a download.

Continuous memory monitoring

Polls free memory and kern.memorystatus_vm_pressure_level every few seconds while the server runs.

Debounced automatic teardown

Shuts down cleanly if free memory stays below ~10% for consecutive polls or macOS reports critical memory pressure.

Bounded concurrency

Explicitly sets --max-num-seqs to what free memory can hold, preventing unbounded concurrent KV allocations.

How It Works

Detect

Scan chip and currently-free memory

Recommend

Filter models that fit; estimate tok/s

Serve

Start MLX server + hot-swap router

Monitor

Track memory & tear down if critical

Technical Details

Components

Component	Purpose
`start_mlx_server.sh`	Orchestrates hardware detection, model selection, server startup, dashboard, and memory/liveness backstops
`model_advisor.py` / `bushpilot/`	Detects hardware specs and computes safe context, APC blocks, concurrency, and tok/s estimates
`launch_tool.sh`	Attaches Claude Code, OpenCode, Aider, Cline, or Pi (auto-splitting a terminal pane to start the server if needed)
`bushpilot/router.py`	HTTP router proxy on port 38472 holding client connections open while hot-swapping the internal MLX backend (port 38473)
`mlx_vlm.server` / `mlx-dspark`	OpenAI & Anthropic-compatible MLX inference backends (multimodal default or text-only speculative decoding)

Token Speed Estimation

The tok/s shown per model is a calculated estimate (not a live per-model benchmark at startup):

estimated tok/s = (chip's memory bandwidth / model's active weight size) × correction factor

Calibrated against real decode-speed benchmarks on Apple Silicon hardware (documented in HARDWARE_SPEED_ESTIMATION.md).

Speculative Decoding: Two Backends

Some catalog entries pair the main model with a small drafter model for real measured speedups (1.5x to 3x+):

-dflash (text-only via mlx-dspark): Fastest option (~15–20% faster than native drafter support after one-time kernel calibration), drops image/video input.
-dflash-vlm / -mtp (multimodal via mlx_vlm.server): Smaller speedup while keeping full vision support.
Plain entry: Unchanged baseline, safest when deterministic byte-for-byte output matters.

Bounded Concurrency

Auto (default): Sizes context to fill free memory for 1 sequence first, then checks whether rounding leftover fits extra identical slots (capped at 4).
Forced (concurrency in config.json): Shrinks per-slot context so all requested slots fit safely.

Why MLX Instead of Ollama?

Ollama is a great cross-platform inference server. Bushpilot is built specifically for Apple Silicon Macs + coding agents:

Aspect	Ollama	Bushpilot (MLX)
Hardware & context sizing	Generic server; manual context/model sizing.	Built-in: detects chip + live free memory, filters models, and computes safe context + tok/s.
Memory backstop	None built-in; relies on OS handling OOM.	Continuous: polls free memory & OS pressure, shutting down cleanly before critical pressure.
Speed on Apple Silicon	Fast (includes a native MLX backend option).	In one internal precision-matched (~4-bit) test, `mlx_vlm.server` ran ~23% faster than Ollama's MLX backend (methodology).
Platform scope	Cross-platform: Linux, Mac, Windows.	Apple Silicon macOS only, by design.

On this page