researchInference

Putting It to Work: Serving Speculative Decoding with vLLM and SGLang

Everything in the previous posts, now running. A practical guide to enabling EAGLE, n-gram, and Medusa speculation in production inference engines.

Conscious EnginesJune 25, 2026

Part 6 of 6 — Speculative Decoding series. Start with The Need for Speed.

The Need for Speed — Why LLMs are memory-bandwidth-bound and how speculation exploits the GPU's idle compute.
Speculative Decoding, Formally — The draft-then-verify algorithm, the rejection-sampling proof, and the metrics that matter.
A Field Guide to Speculative Decoding Methods — Separate models, Medusa, n-gram, and trees: a taxonomy of every drafting approach.
The EAGLE Family — How predicting hidden states instead of tokens reshaped the field, across three generations.
Parallel Drafting with Block Diffusion — DFlash collapses γ draft passes into one; DDTree turns that pass into a 7× verification tree.
Putting It to Work (this post) — Enabling EAGLE, n-gram, and Medusa speculation in vLLM and SGLang, with a guide to measuring real speedup.

Speculative decoding makes an LLM generate faster while provably leaving its output unchanged — a cheap drafter proposes several tokens and the target verifies them in one parallel pass (lossless, via rejection sampling), cashing in the fact that decoding is memory-bandwidth-bound and the GPU is otherwise idle. The drafters have grown steadily more capable — separate small models, Medusa heads, feature-level EAGLE predictors, and parallel block-diffusion drafters — all to raise the accepted-tokens-per-step that drives the speedup.

This final post is operational. The good news is that you almost never implement speculative decoding by hand anymore — the two dominant open-source inference engines, vLLM and SGLang, ship it as a configuration flag. Below is how to turn it on, how to choose a method, and how to confirm the speedup is real on your workload.

Version caveat. Speculative-decoding APIs in both engines have moved fast. The snippets below reflect the stable interfaces as of early 2026 (vLLM's speculative_config on the V1 engine; SGLang's --speculative-* flags). Always cross-check against the docs for your installed version — argument names and supported methods change between releases.

A decision guide: which method?

Before any code, pick a method to match your situation. The trade-offs are exactly the ones from Speculative Decoding, Formally's speedup formula $S \approx \bar k / (1 + \gamma\rho)$ and A Field Guide to Methods's taxonomy:

If you…	Use	Why
Have a small model in the same family	Draft model (vanilla SD)	Zero training, lossless, simple
Want the best general speedup, can grab a checkpoint	EAGLE-3	Top acceptance, plug-and-play, only a lightweight draft head to host (no separate full model)
Generate repetitive/structured output (code, SQL, JSON, agents)	N-gram / prompt-lookup (or SuffixDecoding-style)	Free, no model, huge wins on repetition
Are latency-bound at low batch size / low load	Any of the above	SD shines when the GPU is idle
Are throughput-bound at high batch size	Reconsider	At large batches the GPU is already compute-bound; SD's extra verify work can hurt

That last row is the single most important operational caveat, and it follows directly from The Need for Speed: speculation pays for the GPU's idle capacity. Under heavy batched load the idle capacity is gone, so the wasted-draft overhead can dominate. Speculation is primarily a latency optimization for low-to-moderate load, and most engines let you disable it dynamically as load rises.

vLLM

Offline (the `LLM` class)

In vLLM, speculative decoding is configured through a single speculative_config dict. The output is identical to non-speculative decoding for the lossless methods — only the latency changes.

EAGLE-3 draft (recommended general-purpose):

python

1from vllm import LLM, SamplingParams
2
3llm = LLM(
4    model="meta-llama/Llama-3.1-8B-Instruct",
5    tensor_parallel_size=1,
6    speculative_config={
7        "method": "eagle3",
8        "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B",
9        "num_speculative_tokens": 5,   # draft length γ
10    },
11)
12
13params = SamplingParams(temperature=0.0, max_tokens=256)
14out = llm.generate(["Explain speculative decoding in one paragraph."], params)
15print(out[0].outputs[0].text)

Classic separate draft model (vanilla SD):

python

1llm = LLM(
2    model="meta-llama/Llama-3.1-70B-Instruct",
3    tensor_parallel_size=4,
4    speculative_config={
5        "model": "meta-llama/Llama-3.2-1B-Instruct",  # the draft
6        "num_speculative_tokens": 5,
7    },
8)

N-gram / prompt-lookup (free, great for repetitive output):

python

1llm = LLM(
2    model="meta-llama/Llama-3.1-8B-Instruct",
3    speculative_config={
4        "method": "ngram",
5        "num_speculative_tokens": 5,
6        "prompt_lookup_max": 4,   # longest n-gram to match
7        "prompt_lookup_min": 2,   # shortest n-gram to match
8    },
9)

This method drafts by copying continuations of n-grams it has already seen in the prompt + generated text — no model, no memory cost. It is astonishingly effective for summarization, code editing, and RAG, where the output echoes the input.

Online server

Everything above maps onto vllm serve with --speculative-config taking the same JSON:

bash

1vllm serve meta-llama/Llama-3.1-8B-Instruct \
2  --speculative-config '{"method": "eagle3", "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B", "num_speculative_tokens": 5}'

The server then exposes the standard OpenAI-compatible API — clients need no changes:

bash

1curl http://localhost:8000/v1/completions \
2  -H "Content-Type: application/json" \
3  -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "def fibonacci(n):", "max_tokens": 128, "temperature": 0}'

SGLang

SGLang exposes speculative decoding through --speculative-* launch flags. Its EAGLE integration is mature and well-tuned; the three numbers to know map directly onto the tree concepts from earlier: num-steps is the draft depth, eagle-topk the branching factor, and num-draft-tokens the total tree nodes verified per step.

EAGLE-3 server:

bash

1python3 -m sglang.launch_server \
2  --model-path meta-llama/Llama-3.1-8B-Instruct \
3  --speculative-algorithm EAGLE3 \
4  --speculative-draft-model-path jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B \
5  --speculative-num-steps 5 \
6  --speculative-eagle-topk 8 \
7  --speculative-num-draft-tokens 64 \
8  --dtype float16

Checkpoint formats differ between engines. SGLang's EAGLE3 expects a SpecForge-format draft checkpoint (e.g. jamesliu1/sglang-EAGLE3-Llama-3.1-Instruct-8B or the lmsys/SGLang-EAGLE3-…-SpecForge series), not the yuhuili/EAGLE3-… checkpoints that vLLM and the reference EAGLE repo load. Using the wrong format is a common first-run failure — match the checkpoint to the engine.

EAGLE (v1/v2 checkpoints) use the same flags with --speculative-algorithm EAGLE and the matching draft checkpoint. SGLang also exposes an OpenAI-compatible endpoint on :30000 by default:

bash

1curl http://localhost:30000/v1/completions \
2  -H "Content-Type: application/json" \
3  -d '{"model": "default", "prompt": "Write a haiku about GPUs.", "max_tokens": 64, "temperature": 0}'

Tuning the tree. The three speculative knobs trade acceptance against verification cost — exactly the tension Parallel Drafting with Block Diffusion was about. Larger num-steps and eagle-topk raise $\bar k$ but inflate num-draft-tokens and thus verification cost. Sensible starting points are num-steps 3–6, eagle-topk 4–8, num-draft-tokens 32–64; sweep them on your traffic and keep the configuration that maximizes measured tokens/second, not acceptance rate.

Measuring the speedup honestly

Speculative Decoding, Formally warned that acceptance rate is necessary but not sufficient — only the clock pays the bills. Always benchmark with and without speculation on your prompts, at your batch size. A minimal offline A/B:

python

1import time
2from vllm import LLM, SamplingParams
3
4prompts = [...]  # a representative sample of YOUR traffic
5params = SamplingParams(temperature=0.0, max_tokens=256)
6
7def bench(llm, label):
8    t0 = time.perf_counter()
9    outs = llm.generate(prompts, params)
10    dt = time.perf_counter() - t0
11    n_tok = sum(len(o.outputs[0].token_ids) for o in outs)
12    print(f"{label}: {n_tok/dt:,.1f} tok/s  ({dt:.2f}s for {n_tok} tokens)")
13
14baseline = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
15bench(baseline, "baseline")
16
17spec = LLM(
18    model="meta-llama/Llama-3.1-8B-Instruct",
19    speculative_config={"method": "eagle3",
20                        "model": "yuhuili/EAGLE3-LLaMA3.1-Instruct-8B",
21                        "num_speculative_tokens": 5},
22)
23bench(spec, "eagle3")

For server deployments, drive load with a realistic concurrency sweep (e.g. vLLM's benchmark_serving.py or SGLang's bench_serving) and watch per-request latency and tokens/second as a function of concurrency. You will typically see the speculative speedup shrink as concurrency rises — the expected behavior from The Need for Speed, and the reason many production stacks gate speculation on current load.

Checklist for an honest benchmark:

Use your prompt distribution — acceptance is workload-dependent (Speculative Decoding, Formally).
Match temperature to production — lower temperature usually accepts more.
Sweep batch size / concurrency — the speedup is load-dependent.
Report tokens/second and p50/p99 latency, not just acceptance rate.
Verify correctness: with temperature=0 and a lossless method, speculative and baseline outputs should match token-for-token.

Conclusion: a free lunch, served

Speculative decoding is the rare optimization that asks you to give up nothing. The output distribution is provably unchanged (Speculative Decoding, Formally); the only currency you spend is some wasted draft compute, drawn from a pool — the GPU's idle decode cycles (The Need for Speed) — that was going to waste anyway.

The throughline of this series is a single asymmetry: generation is sequential, verification is parallel, and modern hardware is starved for parallel work during decoding. Everything else — Medusa's heads, EAGLE's feature predictor, Sequoia's optimal trees, DFlash's block-diffusion drafting — is a more and more sophisticated way to manufacture good guesses cheaply and check them in bulk. The field's trajectory has been steadily toward simpler deployment (training-free, plug-and-play) and broader scope (reasoning, SSMs, retrieval, vision), and its center of gravity has moved from research code into the inference engines you can pip install today.

So the practical advice is short. Pick a method that matches your workload, turn it on with a flag, and measure the clock. For most latency-sensitive deployments, a few lines of configuration buy a 2–4× speedup for free. That is about as close to a free lunch as systems engineering ever gets.

References

vLLM — Speculative Decoding documentation. https://docs.vllm.ai/ (see the speculative_config / --speculative-config reference for your version).
SGLang — Speculative Decoding (EAGLE) documentation. https://docs.sglang.ai/
Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. ACM SOSP. (vLLM)
Li, Y., Wei, F., Zhang, C., Zhang, H., et al. (2025). EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv:2503.01840.
Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast Inference from Transformers via Speculative Decoding. ICML. arXiv:2211.17192.
Saxena, A. (2023). Prompt Lookup Decoding. (n-gram / prompt-lookup drafting). https://github.com/apoorvumang/prompt-lookup-decoding
NVIDIA. TensorRT-LLM — speculative decoding support. https://github.com/NVIDIA/TensorRT-LLM

Apr 23, 2026

Speculative Decoding in MLX using DFlash

An empirical evaluation of speculative decoding on Apple Silicon, across a 300-run parameter sweep

apple siliconInference

Sabesh B

A decision guide: which method?

vLLM

Offline (the LLM class)

Online server

SGLang

Measuring the speedup honestly

Conclusion: a free lunch, served

References

Keep reading

Speculative Decoding, Formally: The Algorithm, the Proof, and the Metrics That Matter

The Need for Speed: Why LLMs Are Slow and What Speculation Promises

A Field Guide to Speculative Decoding Methods

The EAGLE Family: Speculating in Feature Space

Parallel Drafting with Block Diffusion: DFlash and DDTree

Speculative Decoding in MLX using DFlash

Offline (the `LLM` class)