Lizard LLM benchmark CLI and HTTP guide

Run reproducible benchmarks through Lizard Native or Caterpillar directly and through the local OpenAI-compatible HTTP API. Compare with llama.cpp only when both runtimes load the same physical GGUF file.

Quick start

node .\src\cli.js models list --source local
node .\src\cli.js models activate --model llama-3.2-3b-instruct
node .\src\cli.js serve --provider lizard-native --model active --port 8099
Invoke-RestMethod http://127.0.0.1:8099/health

Replace lizard-native with caterpillar to exercise Caterpillar. The health response and benchmark artifact must identify the requested provider.

Same-weight HTTP benchmark against llama.cpp

Lizard Native command

npm run bench:native:http -- `
  --provider lizard-native `
  --model llama-3.2-3b-instruct `
  --gguf "$env:USERPROFILE\.lizard-llm\models\llama-3.2-3b-instruct.gguf" `
  --batch "16,8,4,1" `
  --tokens 64 `
  --label lizard-native-llama32-same-weights-http

Caterpillar command

npm run bench:native:http -- `
  --provider caterpillar `
  --model llama-3.2-3b-instruct `
  --gguf "$env:USERPROFILE\.lizard-llm\models\llama-3.2-3b-instruct.gguf" `
  --batch "16,8,4,1" `
  --tokens 64 `
  --label caterpillar-llama32-same-weights-http

The harness verifies canonical paths and refuses an apples-to-oranges run. Never quote a ratio from Llama 3.2 on one side and Gemma on the other.

Adaptive concurrency: B=16, then B=8

Target B=16 for aggregate throughput.
Use B=8 when memory/KV capacity requires it or measured B=8 is faster.
Use B=4 and B=1 only when both primary lanes are unavailable.
Never promote a failed or token-unmeasured lane.

This policy is shared by Lizard Native and Caterpillar. B=1 remains useful for interactive latency, but it is not the preferred throughput baseline.

Benchmark command categories

Throughput

npm run bench:native:http -- --provider caterpillar --model active --batch "16,8" --tokens 128 --skip-llama --label caterpillar-primary-throughput

Latency

npm run bench:native:http -- --provider lizard-native --model active --batch "1" --tokens 64 --skip-llama --label lizard-native-b1-latency

Serving

node .\src\cli.js serve --provider caterpillar --model active --host 127.0.0.1 --port 8099

Offline

node .\src\cli.js models list --source local
node .\src\cli.js benchmark --model active --prompt "Summarize local inference."

Prefix-cache workload

npm run bench:native:http -- --provider lizard-native --model active --batch "16,8" --tokens 64 --prompt "Shared fixed prefix. Explain repeated-prefix reuse." --skip-llama --label repeated-prefix-workload

A repeated prefix does not by itself prove a cache hit. Require explicit telemetry and a controlled cache-off run before claiming acceleration.

Sequence length and concurrency

Keep long prompts version-controlled, hold output tokens constant, and include fallback lanes when the workload may reduce KV capacity.

Stress and production

Repeat complete sweeps, preserve every artifact and failure, record background load, and report median plus spread rather than the single fastest result.

Metrics and interpretation

Metric	Meaning
Aggregate tokens/sec	Completion tokens divided by wall time across successful concurrent requests.
Requests/sec	Successful requests divided by wall seconds; compare only similar request work.
Median latency	Middle complete non-streaming HTTP response duration.
TTFT / TPOT	Not reported by this non-streaming harness; require timestamped streaming.
Completed / failed	Request outcomes; failed lanes are ineligible for automatic selection.
Provenance	Canonical candidate/control files and actual runtime provider.

CLI and console workflow

Scan hardware → list and activate a model → start Lizard Native or Caterpillar → verify health → benchmark direct or HTTP → inspect metrics and provenance → retain the artifact.

Key categories include server, benchmark, engine, model, quantization, execution, backend, logging, and debugging. Common advanced overrides are --threads, --ctx-size, --gpu-layers, --batch-size, --ubatch-size, --backend, --timeout, --no-mmap, and --mlock.

Environment variables

Advanced controls include LIZARD_CONFIG_PATH, LIZARD_RUNTIME_DIRECTORY, LIZARD_PORT, LIZARD_CHAT_PORT, LIZARD_CHAT_PROVIDER, LIZARD_NATIVE_RUNNER, LIZARD_NATIVE_GPU_ADAPTER_LUID, LIZARD_NATIVE_INFERENCE, LIZARD_CATERPILLAR_INFERENCE, CATERPILLAR_CPU_THREADS, CATERPILLAR_GPU, and CATERPILLAR_SPECULATIVE. Prefer public CLI flags and record every override in the artifact or run notes.

Multi-GPU, tensor parallel, and data parallel

Lizard does not currently expose public tensor-parallel or data-parallel model-sharding flags. Multi-machine remote benchmarking compares independent paired computers; it does not combine their GPUs into one model execution.

Production checklist and troubleshooting

Record version, provider, model, GGUF hash/path, quant, hardware, driver, prompt, tokens, lane, fallback state, and time.
Warm providers consistently and stop competing model servers.
Use the same physical GGUF for every same-weight ratio.
Prefer HTTP evidence for claims about user-facing serving speed.
Keep 5.15 tok/s only as historical local context with its artifact, never as a universal guarantee.

Read the benchmark methodology, native-engine architecture, remote workflow, general documentation, and the official vLLM Benchmark CLI reference that informed this page's documentation structure. vLLM flags are not Lizard flags.