AI-Native Updated 2026-09 11 min read View as Markdown

Local inference economics: where the cost is memory bandwidth

Nobody calls a hosted model from C. If C is in your stack, you are running the model — and the currency is not tokens, it is bytes moved per second.

Every other page in this series is about a bill from an API. This one is not, because if you are writing C in an LLM context you are almost certainly on the other side of that API: llama.cpp, ggml, a custom runtime, an embedded deployment. The hosted economics still apply to whatever calls you — but your own costs are hardware, and they behave completely differently.

The single most useful thing to internalise: generating tokens is memory-bandwidth bound, not compute bound. Almost every performance and cost decision follows from that one fact, and it is the opposite of most people's intuition.

Why bandwidth, not FLOPs#

Generating one token with a dense transformer requires reading essentially every weight in the model from memory. The arithmetic per weight is trivial — one multiply-accumulate — so the processor spends nearly all its time waiting for memory.

That gives you a back-of-the-envelope bound that is usually within a factor of two of reality:

tokens/sec  ≈  memory bandwidth (GB/s)  /  model size in memory (GB)

/* A sanity check worth running before you optimise anything. */
static double roofline_tok_s(double bandwidth_gb_s, size_t weight_bytes) {
    return bandwidth_gb_s * 1e9 / (double)weight_bytes;
}

/* 7B params at 4-bit ≈ 3.8 GB of weights.
 * A machine with ~200 GB/s of usable bandwidth:
 *   200e9 / 3.8e9  ≈  52 tokens/sec, single stream.
 * If you are getting 50, you are at the hardware limit and no amount of
 * SIMD tuning will help. If you are getting 12, you have a real problem. */

Run that calculation before you spend a week on kernels. Most "slow inference" turns out to be either a model that does not fit in fast memory, or a KV cache that has quietly grown past it.

Prefill and decode are different workloads#

Conflating them is the most common analysis error, and it leads to optimising the wrong half.

So a long prompt with a short answer is a compute problem, and a short prompt with a long answer is a bandwidth problem. They want different optimisations and they should be measured separately.

/* Report them separately or the number is meaningless. */
printf("prefill: %6.1f tok/s (%zu tokens, %.2f s)\n",
       n_prompt / prefill_s, n_prompt, prefill_s);
printf("decode : %6.1f tok/s (%zu tokens, %.2f s)\n",
       n_gen / decode_s, n_gen, decode_s);
printf("ttft   : %6.0f ms\n", prefill_s * 1000.0);

Time to first token is prefill; tokens per second is decode. A user perceives those as two different qualities, and a system tuned for one can be poor at the other.

The KV cache is the thing that surprises people#

Weights are fixed. The KV cache grows with every token in the context, per sequence, and it is where "it worked in testing and OOMs in production" comes from.

/* bytes = 2 (K and V) * layers * kv_heads * head_dim * ctx_len * bytes_per_elem
 * (grouped-query attention reduces kv_heads well below the attention head count) */
static size_t kv_cache_bytes(int layers, int kv_heads, int head_dim,
                             int ctx_len, int elem_bytes, int n_seq) {
    return (size_t)2 * layers * kv_heads * head_dim
         * (size_t)ctx_len * elem_bytes * n_seq;
}

Two properties matter:

• It is linear in context length. Doubling your context window doubles this allocation, on top of the weights.
• It is linear in concurrent sequences. Batching four requests means four KV caches.

Which is why serving eight concurrent 32k-context sessions can need more memory for cache than for the model itself. Compute it up front, at startup, and refuse a configuration that will not fit rather than discovering it under load:

size_t need = weights_bytes
            + kv_cache_bytes(cfg.layers, cfg.kv_heads, cfg.head_dim,
                             cfg.ctx_len, cfg.kv_elem_bytes, cfg.max_seq)
            + workspace_bytes(cfg);

if (need > budget_bytes) {
    fprintf(stderr,
            "config needs %.2f GB, budget is %.2f GB\n"
            "  reduce --ctx (%d), --parallel (%d), or quantise the KV cache\n",
            need / 1e9, budget_bytes / 1e9, cfg.ctx_len, cfg.max_seq);
    return -EINVAL;
}

Quantising the KV cache to 8-bit halves that allocation for a usually small quality cost, and it is the first thing to try when you are memory-limited rather than bandwidth-limited.

Quantisation is the main dial#

Fewer bits per weight means fewer bytes read per token means more tokens per second — and less memory. It is the rare optimisation that improves both axes at once.

The practical guidance is unglamorous: 4-bit is the default for a reason, and below it you are trading real capability for diminishing returns. A larger model at 4-bit generally beats a smaller model at 8-bit for the same memory — which is the decision that actually matters when you are sizing a deployment.

Measure quality when you change quantisation. Perplexity on a held-out set is the cheap proxy; a task-specific eval set is what you should actually gate on.

Batching: the throughput lever#

Because decode reads all the weights per step regardless, processing several sequences in the same step gets the extra ones nearly free in bandwidth terms.

1 sequence   →  50 tok/s total
8 sequences  → ~300 tok/s total   (≈37 each: slower per user, 6x the throughput)

That trade — per-request latency for aggregate throughput — is the whole design question for a serving deployment. Continuous batching, where a finished sequence is replaced immediately rather than waiting for the whole batch, is what makes it practical.

The limit is memory: each concurrent sequence needs its own KV cache. Batch size is bounded by (total memory − weights) / kv_bytes_per_sequence, which is why the sizing function above is worth writing.

Measure the right things#

typedef struct {
    double  prefill_tok_s, decode_tok_s, ttft_ms;
    size_t  peak_rss_bytes, kv_bytes;
    double  achieved_gb_s;      /* vs your hardware's theoretical peak */
} bench_t;

achieved_gb_s divided by your hardware's rated bandwidth is the number that tells you whether further optimisation is worth attempting. Above roughly 70% of peak you are near the roof and the remaining work is in reducing bytes, not in faster kernels. Below 40%, something structural is wrong — a bad memory layout, a model that spills out of the fast memory tier, an allocation in the hot loop.

perf stat -e cache-misses,cache-references,instructions,cycles ./infer --prompt-file p.txt

A high cache-miss rate with low IPC is the signature of a bandwidth-bound workload behaving as expected. It is also what makes profiling here different from ordinary C profiling — see the performance page.

Cost, in the units you actually pay in#

amortised cost per 1M tokens
  =  (hardware cost / useful lifetime hours + power kW × electricity rate)
     ÷ (tokens per hour at your achieved throughput)
     × 1,000,000

Two things fall out of that formula that people consistently get wrong:

• Utilisation dominates. A machine at 10% utilisation costs ten times as much per token as the same machine at 90%. Batching and queueing improve your unit economics far more than kernel tuning does.
• Compare honestly against hosted. Include the machine, the power, and the engineering time to keep it running. Local inference wins clearly on privacy, on latency, on offline capability and at high sustained utilisation. At low or spiky volume, hosted is usually cheaper once you count everything.

Common questions#

Why is my inference slower than the roofline says?#

In order of likelihood: the model does not fit in the fastest memory tier and is being paged; the KV cache has grown past what fits alongside the weights; you are allocating or copying inside the decode loop; or your threads are fighting over memory bandwidth rather than adding throughput. Check memory placement before you touch a kernel.

More threads did nothing. Why?#

Because you are bandwidth-bound, not compute-bound. Past the point where the memory controller is saturated, extra threads add contention rather than throughput. The optimum is often well below the core count, and it is worth measuring rather than assuming.

Should I quantise the KV cache as well as the weights?#

If you are memory-limited rather than bandwidth-limited, yes — 8-bit KV roughly halves that allocation for a small quality cost, and it is what lets you raise the context length or the batch size. If you have memory to spare, leave it.

Is local inference cheaper than an API?#

At high sustained utilisation, and where privacy, latency or offline operation matter, clearly yes. At low or bursty volume, usually not once you count hardware amortisation, power and the engineering time — the honest comparison includes the third one, which is the one people leave out.