quantization

DeepSeek V4.1 Flash on a 12GB GPU

Published 29 October 2026. Checked against ISTA-DASLab’s model card (updated 27 October), Strata’s documentation for version 0.2.7 and llama.cpp release b11870.

The 2-bit build of DeepSeek V4.1 Flash that runs on a 12GB graphics card is a 248.84GB set of files. That’s twenty times the card’s memory, and it is not a contradiction. You can run DeepSeek V4.1 Flash locally on a 12GB GPU because the card never holds the model. The GPU does part of the arithmetic, system RAM caches the experts used most often, and an NVMe SSD supplies the rest.

So the question worth asking isn’t whether 12GB of VRAM is enough. It’s whether the rest of the computer is. Strata, the open-source runtime behind the 12GB claim, documents a working setup with 46GB of system RAM and calls 64GB comfortable. Its developer measured between 4 and 11 tokens per second on 12GB cards, depending on the machine.

Quick answer

Can it run? Yes, through Strata, on NVIDIA RTX 30, 40 or 50 series cards with 12GB or more, under Windows or Linux.
What else you need: about 46GB of system RAM at minimum (64GB is Strata’s tested comfortable size), an NVMe SSD with roughly 245GB to 280GB free, and a modern CPU that will do much of the expert computation.
Package size: 231.32GB to 265.83GB for the three ISTA-DASLab builds. Strata’s downloads are 239GB to 274GB because they add an 8GB draft model.
The main limit: data movement. On 12GB cards, most of each token’s time goes to waiting on the SSD and CPU.
Who should try it: people who want a frontier-scale open model offline, can accept a few tokens per second, and already own a machine with the RAM and disk.

Key Takeaways

763 billion parameters isn’t the working size. Most of it is a 552-billion-parameter backbone and 196 billion parameters of Engram lookup tables, read a few rows at a time.
248.84GB is a disk total, not a RAM requirement: a 138.24GB backbone plus a shared 110.60GB Engram file.
System RAM and the SSD matter as much as the GPU. Strata’s RAM expert cache ran about twice as fast as memory-mapping the files on the developer’s RTX 5070 machine.
“100% recovery” covers three benchmarks. Strata’s own test found the same build’s perplexity 15.5% higher on general text.
Strata’s 12GB default isn’t the full model. It routes each token to five of the model’s six experts unless you choose “Full quality”.
llama.cpp supports the model from release b11870.

Quick Navigation

What it takes to run DeepSeek V4.1 Flash locally on a 12GB GPU
The files you need to run DeepSeek V4.1 Flash locally
How Strata splits the model across GPU, RAM and SSD
How fast DeepSeek V4.1 Flash runs locally
Can your desktop run DeepSeek V4.1 Flash locally?
Why the three-benchmark result needs context
Running DeepSeek V4.1 Flash locally with llama.cpp
Is it worth the effort to run DeepSeek V4.1 Flash locally?
What the 12GB claim really tells us
Frequently asked questions

What it takes to run DeepSeek V4.1 Flash locally on a 12GB GPU

Three numbers are easy to blur here, and they answer different questions.

Package size on disk is how much storage the files take: 248.84GB for the recommended IQ2_XXS build. Memory in use is what has to sit in VRAM and RAM while the model generates. Bandwidth is how fast the weights a token needs can reach whatever is computing it. A setup can pass the first two tests and still be slow on the third.

The architecture is what makes this possible. DeepSeek’s model card describes a mixture-of-experts (MoE) design: each MoE layer has 384 routed experts, small feed-forward networks, plus one shared expert. A router sends each token to six of the 384. DeepSeek puts the active parameters at 8 billion per token during prefill and 16 billion during decoding, out of 552 billion in the backbone. Most of the model sits idle for any given token.

The Engram component works differently again. It is 196 billion parameters of what DeepSeek calls “conditional memory”, accessed by token-based lookup rather than matrix multiplication. ISTA-DASLab’s card says the table is read one row per token. That’s why 110.60GB of it can stay on an SSD without hurting speed much.

The 763-billion headline figure comes from the Hugging Face metadata for the 1017 revision. The backbone and Engram add up to about 748 billion, and neither DeepSeek nor ISTA-DASLab explains the remaining 15 billion.

The files you need to run DeepSeek V4.1 Flash locally

ISTA-DASLab, the Distributed Algorithms and Systems lab at the Institute of Science and Technology Austria, published three GGUF builds of DeepSeek-V4.1-Flash-1017, the 17 October revision. The method has two parts. GSQ chooses how each weight is rounded. RCO decides which quantization type each tensor gets so the total fits a size budget. Routed experts get between roughly 1.5 and 3.5 bits per weight. Dense tensors keep between about 5.5 and 8.5.

Build	Bits per weight	Backbone	Engram file	Card total	Strata download	Reported quality
IQ1_M	1.75	120.72GB	110.60GB	231.32GB	239GB	98.7% of base task average (ISTA); perplexity +21% (Strata)
IQ2_XXS	2.00	138.24GB	110.60GB	248.84GB	257GB	100% of base task average (ISTA); perplexity +15.5% (Strata)
IQ2_XS	2.25	155.23GB	110.60GB	265.83GB	274GB	99.8% of base task average (ISTA); perplexity +7.3% (Strata)

The card total is the backbone plus the shared Engram file, in decimal gigabytes (10⁹ bytes). It excludes a 0.97GB vision projector. Strata’s download adds an 8GB draft model used for speculative decoding. Neither number is a measured runtime requirement.

The Engram file, stored in a 4.5-bit format called IQ4_NL, is identical across builds, so comparing them doesn’t mean downloading it twice. Strata recommends IQ2_XS for coding and agent work if disk allows: half the perplexity gap of IQ2_XXS, for 17GB more storage and slightly slower generation.

How Strata splits the model across GPU, RAM and SSD

Strata is an MIT-licensed local runtime maintained by a developer publishing as Niko1221. It added DeepSeek V4.1 Flash in version 0.2.4 on 24 October and was at 0.2.7 when this was written. Its DeepSeek engine, strata-dsv41, is a separate package ported from Project Maya’s MIT-licensed maya-ds4.

According to Strata’s DeepSeek guide and release notes, the work is split three ways. The GPU holds the dense weights and as many routed experts as fit, and computes them. A RAM expert store, which sizes itself to free memory when the engine starts, keeps the most-used experts in system RAM, and the CPU computes those. Experts that fit in neither are read from the model files on the SSD as they’re needed. A hot-expert profile ships with the engine, so the caches don’t start empty.

The Engram table is read “from RAM or SSD”. With 46GB to 64GB of RAM against a 138GB backbone, some disk reads during generation are unavoidable on a 12GB machine.

Strata’s 0.2.4 release notes compare this design with plain memory-mapping, where the operating system pages file data in on demand. On an RTX 5070 with 64GB of RAM, the expert store produced 4.3 to 4.9 tokens per second, against 1.8 to 2.2 for memory-mapping.

Two defaults matter for quality. The guide says context is set to 8,192 tokens, “the size it was checked at”, and --context raises it. On machines whose GPUs total 16GB or less, a “Faster” mode routes each token to five experts instead of the model’s six. The guide describes the quality change as “usually not noticeable”. Neither ISTA-DASLab’s benchmarks nor Strata’s perplexity figures say they used that mode, so “Full quality” (--experts 6, engine 0.1.50 or newer) is the setting that matches the published evidence.

How fast DeepSeek V4.1 Flash runs locally

Strata’s README reports these figures, measured by its developer on version 0.2.5. UniverseBlend hasn’t reproduced them.

Machine	System RAM	Output speed	Prompt processing
RTX 5070 12GB, Ryzen 5 7600, 3.6GB/s NVMe	64GB	4–6 tokens/s	231 tokens/s at 8K (0.2.4 notes)
RTX 3060 12GB	46GB	9–11 tokens/s	149 tokens/s at 8K (0.2.4.1 notes)

The older RTX 3060 beating the RTX 5070 is a reminder that the GPU isn’t the main variable here. The project doesn’t explain the gap, and CPU, SSD and RAM configuration all differ between the two machines.

A back-of-envelope figure shows why storage dominates. If all 16 billion active parameters came off the SSD at about 2 bits each, every token would need roughly 4GB read. At 3.6GB per second, that’s about one token per second. That’s a pessimistic bound, because the dense weights stay on the GPU. Strata’s 4 to 11 tokens per second means caching is doing most of the work. A slower SSD, less RAM or a different mix of topics, which changes which experts are hot, pushes the system back toward that floor. Our earlier piece on the memory-bandwidth limits of AI chips explains the same constraint at data-centre scale.

Reasoning length matters as much as raw speed. Strata’s default thinking budget is 32,768 tokens. At 5 tokens per second, a fully used budget is about 1 hour 49 minutes before the answer starts. A 2,000-token reply alone takes about 6.7 minutes. Strata describes 4 to 8 tokens per second as usable for chat but slow for long agent runs, which is a fair summary.

Can your desktop run DeepSeek V4.1 Flash locally?

Work through these in order. The first failure tells you what to upgrade.

GPU: an NVIDIA RTX 30, 40 or 50 series card with at least 12GB, on Windows or Linux. AMD support covers the Radeon AI PRO R9700 on Linux, and Strata’s README adds Ryzen AI Max “Strix Halo” on Windows. Macs aren’t supported.
System RAM: 46GB is the smallest configuration Strata reports working, and 64GB is its tested comfortable size. The guide calls anything under about 40GB “untested and slow”. Setup warns if RAM is short but doesn’t stop.
Free SSD space: 245GB to 280GB, depending on the build, on one drive.
Storage type: NVMe. Strata’s test machine used a 3.6GB/s drive, and the guide warns that a slow disk “slows every word”. Avoid SATA SSDs and hard drives.
CPU: a recent desktop processor, since it computes the experts that don’t fit on the GPU. Strata uses AVX-512 on Zen 4 and Zen 5.
Power and cooling: GPU, CPU and SSD stay busy together for long stretches, and an NVMe drive without a heatsink may throttle.
Patience: a 250GB download, a load of several minutes, and replies measured in minutes.

These are the configurations Strata’s developer reports testing, not universal minimums.

Why the three-benchmark result needs context

ISTA-DASLab reports that IQ2_XXS matches the FP8/FP4 base model on all three tasks it measured, at reasoning effort 50:

Variant	AIME25	GPQA-Diamond	LiveCodeBench v6	Task average
Base model	100.00	90.91	89.71	93.54
IQ2_XXS (2.00 bpw)	100.00	90.91	89.71	93.54

The result comes from the team that made the quantization, on competition maths, graduate science and coding problems. “100% recovery” means those three scores were identical, not that the 2-bit build keeps every capability. Nothing here tests long-context recall, multilingual writing, tool use or image input. And with the base model already at 100 on AIME25, that benchmark can only register a loss.

Strata’s own measurement shows why that matters. On 4,995 text positions, the IQ2_XXS build’s perplexity, a measure of how well a model predicts text, was 15.5% higher than the original’s. That’s a different test from ISTA-DASLab’s, and it isn’t evidence that answers are 15.5% worse. It does show that a build can match on three reasoning benchmarks and still differ measurably elsewhere. The IQ1_M build’s perplexity on plain English prose was 53% higher.

For an independent view of the full model, Artificial Analysis gives DeepSeek V4.1 Flash 1017 at maximum reasoning effort a score of 32 on its Intelligence Index v5.0, which combines nine evaluations. That number is on v5.0’s scale and can’t be compared with scores from the older v4 index. It also measures DeepSeek’s hosted model, not a 2-bit local build. Epoch AI’s Capabilities Index is a separate evaluation: it lists 156 for the original September release. As we’ve argued in why benchmark scores don’t transfer to your workload, the useful test is your own tasks on your own build.

Running DeepSeek V4.1 Flash locally with llama.cpp

Standard llama.cpp support arrived recently. Release b11870, published as a pre-release on 25 October, adds the deepseek41 architecture and Engram handling. ISTA-DASLab’s card says its files have used the current layout since 27 October. Copies downloaded before then need a local conversion with the convert_ds41_upstream.py script.

The card also notes that llama.cpp’s default lazy mode memory-maps tensors over 4GiB, so the Engram file needs neither VRAM nor RAM. Strata’s RAM expert store and hot-expert profiles are its own additions, and in Strata’s comparison plain memory-mapping ran at roughly half the speed. On a 12GB card, expect llama.cpp to work but not to match Strata’s figures without tuning. We found no published like-for-like comparison of the two.

Is it worth the effort to run DeepSeek V4.1 Flash locally?

That depends on what you need the model for.

The case for local: offline use, data that can’t leave the machine, and experimenting with a frontier-scale open model at no per-token cost. The base model and ISTA-DASLab’s files are MIT-licensed, which permits commercial use.
The case for the API: price and speed. Artificial Analysis lists DeepSeek’s API at $0.30 per million input tokens and $1.20 per million output tokens, at about 184 output tokens per second. Generating one million output tokens locally at 5 tokens per second takes more than 55 hours of continuous running, a job the API would bill at $1.20.

Local isn’t automatically private either: downloads, engine updates and connected tools still touch the network. For coding agents, the API wins on throughput. For occasional private questions on hardware you already own, local is a reasonable choice. Our comparison of Chinese open-weight models covers the licensing terms and the US federal restrictions on DeepSeek products.

What the 12GB claim really tells us

A 12GB card is the cheapest part of this setup. What lets you run DeepSeek V4.1 Flash locally is a runtime that treats VRAM, RAM and an NVMe drive as one memory hierarchy, plus the 46GB to 64GB of RAM and 250GB of fast storage that hierarchy needs.

The headline is accurate, and it measures the wrong part of the computer. Check your RAM and your SSD first. Then decide whether a few tokens per second fits the work you have in mind.

Frequently asked questions

Can you run DeepSeek V4.1 Flash on a 12GB GPU?

Yes, with Strata, on an NVIDIA RTX 30, 40 or 50 series card with at least 12GB, under Windows or Linux. The GPU holds only part of the model. The rest is cached in system RAM and read from an NVMe SSD, so the card alone doesn’t decide whether the setup works.

How much RAM do you need to run DeepSeek V4.1 Flash locally?

Strata reports a working setup with 46GB and calls 64GB its tested comfortable size. Below about 40GB is untested and, in the developer’s words, slow. More RAM lets Strata keep more experts cached, which reduces SSD reads and raises speed.

Does the 248.84GB model size mean you need 248.84GB of RAM?

No. 248.84GB is the disk size of the IQ2_XXS build: a 138.24GB backbone plus a 110.60GB Engram lookup file. The Engram file is read a row at a time, and Strata caches only the most-used experts in RAM. The rest stays on the SSD until needed.

Can DeepSeek V4.1 Flash run with standard llama.cpp?

Yes, from release b11870 (25 October 2026), which adds the deepseek41 architecture. ISTA-DASLab’s files have matched that layout since 27 October; earlier downloads need converting first. By default llama.cpp memory-maps large tensors, and Strata measured memory-mapping at roughly half the speed of its own RAM expert store.

What is the difference between IQ1_M, IQ2_XXS and IQ2_XS?

They are 1.75, 2.00 and 2.25 bits per weight on average, with backbones of 120.72GB, 138.24GB and 155.23GB, sharing one 110.60GB Engram file. More bits mean better fidelity: Strata measured perplexity increases of 21%, 15.5% and 7.3% over the original. Smaller builds suit tight disks.

Does the 2-bit quantized model retain 100% of the original model’s performance?

Only on the three benchmarks its makers report: AIME25, GPQA-Diamond and LiveCodeBench v6, where IQ2_XXS matched the base model’s scores. Strata’s perplexity test found a 15.5% gap on general text. Treat “100% recovery” as a statement about those three tasks.

Will DeepSeek V4.1 Flash run quickly on an ordinary desktop?

Not by API standards. Strata’s developer measured 4 to 6 tokens per second on an RTX 5070 with 64GB of RAM, and 9 to 11 on an RTX 3060 with 46GB. That’s workable for chat, but long reasoning traces and agent loops can take many minutes per answer.

Is running DeepSeek locally better than using an API?

For offline work or restricted data on hardware you already own, it can be. For throughput, DeepSeek’s API is far faster, at about 184 tokens per second, and it charges $1.20 per million output tokens. Local use also isn’t automatically private: setup, updates and connected tools still use the network.

Keep reading

Here are the latest posts from the blog.

Anthropic IPO: The $895 Billion Question Behind Its Valuation

November 9, 2026

Last updated: 9 November 2026. Latest confirmed development: Anthropic’s amended S-1, filed on 2 November, still leaves the share count and the IPO price range …

Claude Opus vs Sonnet: What the 2026 Price Gap Actually Buys

November 8, 2026

Last verified: 8 November 2026. All prices are Anthropic’s standard first-party API rates in US dollars per million tokens. They aren’t Claude Pro or Max …

Did AI Break RSA? What DeepMind Actually Found

November 8, 2026

Last verified: 8 November 2026. Both papers are unreviewed working papers, and no outside group has reproduced the results yet. No, AI did not break …

Model Distillation Explained: When AI Learning Becomes Extraction

November 7, 2026

Last verified: 7 November 2026. Government and company claims are attributed throughout. None has been tested in court. A support team trains a small, cheap …

The AI Glossary: 10 Terms You Now Meet Everywhere

by Idrees Patel

Most AI writing assumes you already know the vocabulary. This AI glossary fixes that.

Below are ten terms from the AI glossary that show up constantly in chip news, model launches, and filings. Each entry gives a plain definition first, then the number or fact that makes it matter.

This is batch one. The AI glossary will grow, and every term here links from its first mention across the site.

Why This AI Glossary Exists

Technical vocabulary moves faster than the explainers do, which is the whole case for an AI glossary.

Take KV cache. It went from research jargon to procurement conversation in about eighteen months. Nobody wrote the bridging definition, so readers either already knew or quietly skipped the paragraph.

This AI glossary is the bridge. Each definition is written for someone competent who simply has not met the term yet, which is a very different audience from a beginner.

A glossary also gives readers a stable reference when an article uses unfamiliar terminology. A clear definition and a direct link help readers check the meaning in context.

How to Read This AI Glossary

Every AI glossary entry follows the same shape, so you can skim or read in order.

The first line is the definition. Read only that if you are in a hurry, then move on. The second paragraph in each AI glossary entry gives context: a figure, a date, or a trade-off. That is where the actual understanding lives.

This AI glossary groups terms by layer, from silicon upward. So the AI glossary reads as a stack, not an alphabet.

Key Takeaways From This AI Glossary

This AI glossary starts with memory. HBM and LPDDR solve opposite problems: bandwidth versus cost per gigabyte.
MoE, distillation, and quantization all shrink the cost of a model, each in a different way.
KV cache, not model size, is usually why long context gets expensive.
RAG and MCP sit above the model. One supplies documents, the other supplies tools.
Inference is where most AI money goes across a model’s life.

Quick Navigation

AI Glossary: Memory and Hardware
AI Glossary: Model Architecture
AI Glossary: Runtime and Serving
AI Glossary: Retrieval and Tooling
AI Glossary: Terms People Mix Up
AI Glossary: How the Ten Terms Fit Together
How the AI Glossary Connects Across the Site
How This AI Glossary Is Marked Up
How to Check and Cite a Definition
What Batch 2 of the AI Glossary Adds
Who This AI Glossary Is For
Conclusion: Use the AI Glossary as a Reference, Not a Read
FAQ About This AI Glossary

AI Glossary: Memory and Hardware

Memory decides what a chip can hold and how fast it feeds the math. So this AI glossary starts there.

HBM (High Bandwidth Memory)

HBM is DRAM stacked in vertical layers and wired close to the processor, trading capacity for very high bandwidth.

The current generation matters if you read chip news for buying signals. HBM4 entered mass production in February 2026, and the JEDEC JESD270-4 standard doubles the interface from 1,024 bits to 2,048 and lifts channels from 16 to 32. Nvidia’s Rubin platform is expected to pair eight stacks for 288GB and over 22 TB/s. Samsung and SK Hynix together supply roughly 90% of it, and each vendor roadmap now stretches to HBM4E.

LPDDR (Low-Power Double Data Rate)

LPDDR is mobile-class DRAM tuned for power efficiency and capacity rather than peak bandwidth.

Think phones, laptops, and edge boxes rather than data centre racks. LPDDR delivers far less bandwidth than HBM, but costs a fraction per gigabyte and draws much less power. That is why small serving appliances use it while data centre racks do not. We covered how memory choice shapes accelerator margins here.

AI Glossary: Model Architecture

These three AI glossary terms all answer one question. How do you make a capable model cheaper to run?

MoE (Mixture of Experts)

MoE splits a model into many expert sub-networks and routes each token to only a few of them, so total parameters far exceed the parameters used per token.

DeepSeek-V3 shows the gap plainly: about 671 billion total parameters, roughly 37 billion active per token. Compute per token drops sharply. Memory does not, because every expert must stay loaded and ready.

Distillation

Distillation trains a small student model to copy the behavior of a larger teacher model.

The idea dates to a 2015 paper from Hinton and colleagues. It is now routine. DeepSeek shipped R1-distilled versions built on Qwen and Llama bases. Students typically land a few points below the teacher while costing far less to serve.

Quantization stores weights and activations at lower numeric precision, such as 8-bit or 4-bit instead of 16-bit.

Cutting from 16-bit to 4-bit roughly quarters the memory a model takes up. Formats like GPTQ, AWQ, and GGUF made this routine, and FP8 and FP4 now run natively on recent accelerators. You lose a little accuracy and gain a lot of bandwidth headroom.

AI Glossary: Runtime and Serving

Now the AI glossary terms that describe what happens when a model actually answers something.

Inference

Inference is running a trained model to produce an output, as opposed to training it.

It splits into two phases with different bottlenecks. Prefill processes the prompt and is compute-bound. Decode generates tokens one at a time and is memory-bandwidth-bound. Serving dominates a model’s lifetime cost, which is why so much hardware design now targets this half of the AI glossary rather than training.

KV cache

The KV cache stores key and value tensors from tokens already processed, so attention does not recompute them at every step.

Skipping that work is what makes generation fast. The cost is memory, and it grows linearly with sequence length and batch size. At long context the KV cache often consumes more memory than the model weights themselves, which is why techniques like grouped-query attention and paged attention exist.

Context window

The context window is the maximum number of tokens a model can consider at once, counting both the prompt and the output.

Windows now run from a few thousand tokens to over a million. But a large window is a ceiling, not a promise. Retrieval accuracy often degrades well before the stated limit, and benchmark figures rarely capture that. Every extra token also enlarges the KV cache.

AI Glossary: Retrieval and Tooling

These last two AI glossary terms sit above the model. Neither changes the weights.

RAG (Retrieval-Augmented Generation)

RAG fetches relevant documents from an external store and places them in the prompt, so the model answers from supplied evidence rather than memory alone.

The approach comes from a 2020 paper by Lewis and colleagues at Facebook AI. It remains the cheapest way to give a model fresh or proprietary information without retraining. Quality depends far more on the retrieval step than on the model, which teams consistently underestimate.

MCP (Model Context Protocol)

MCP is an open standard that lets AI applications connect to external tools and data through a common client-server interface.

Anthropic released it in November 2024. OpenAI, Google, and Microsoft have since adopted it. The 2026-07-28 specification made the protocol stateless, which lets servers scale on ordinary HTTP infrastructure. Security is still maturing: the NSA published design considerations in May 2026 flagging gaps around prompt injection and tool poisoning.

AI Glossary: Terms People Mix Up

Four pairs cause most of the confusion in any AI glossary. Sorting them beats adding ten more definitions.

Inference versus training in this AI glossary

Training builds the model once, over weeks, on a cluster. Inference runs it billions of times afterwards. The AI glossary treats them separately because the hardware, the bottleneck, and the cost curve all differ.

Both terms shrink cost, but not the same way. Quantization keeps the same model and stores its numbers less precisely. Distillation builds a genuinely smaller model that imitates a bigger one. You can do both to the same system.

The context window is a limit set by the model, not by your hardware. The KV cache is the memory actually consumed while operating inside that limit. A vendor advertises the first. Your infrastructure bill reflects the second.

RAG versus MCP in the AI glossary

RAG brings documents to the model. MCP lets the model reach out to tools and systems. One is read-only context; the other is an action interface. Many production stacks run both, which is why this AI glossary lists them side by side.

AI Glossary: How the Ten Terms Fit Together

Read the AI glossary as a stack and the relationships get obvious.

At the bottom of the AI glossary sits memory. HBM and LPDDR decide how fast weights can reach the math units, and everything above inherits that ceiling.

Above memory sits architecture, the middle band of the AI glossary. MoE, quantization, and distillation are three different strategies for fitting more capability under the same memory ceiling.

Above architecture sits runtime, where most questions actually arise. Inference, KV cache, and context window describe what happens while a request is being served, and where the memory actually goes.

At the top of the AI glossary sits the application layer. RAG and MCP never touch the weights. They shape what the model sees and what it can act on.

So a change at the bottom of this AI glossary propagates upward. Wider HBM interfaces make longer context affordable, which makes larger retrieval payloads practical, which changes what RAG systems can attempt.

How the AI Glossary Connects Across the Site

Use this glossary alongside the related articles to move between a term’s definition and its practical consequences.

When an article links a term to this page, the fragment link opens the relevant definition.

A definition is a starting point. Follow the related technical discussion when the implementation details matter.

The related articles explain how memory, model architecture and serving choices affect practical systems.

A reader arriving at a chip-economics article can use the glossary to check a term and then return to the original discussion.

How This AI Glossary Is Marked Up

Structure matters as much as wording when machines read a page.

Each entry uses schema.org DefinedTerm, and all ten sit inside a single DefinedTermSet. That tells crawlers and language models that this is a controlled vocabulary, not a listicle.

The pairing matters. A lone DefinedTerm is a fragment. Wrapped in a DefinedTermSet with a stable URL, the AI glossary becomes a citable reference object that can be extended without breaking anything.

Each AI glossary term also carries a termCode and its own anchor, so external pages can link straight to one definition.

How to Check and Cite a Definition

When citing a definition, include the term’s direct link and check that the wording fits the context in which you are using it.

Read the short definition first, then the accompanying explanation and any linked source. A concise definition still needs the qualifications that follow it.

For a hardware specification, follow the source to check the product generation, units and operating conditions.

A numerical specification is more useful when its source and scope are clear. Do not treat a component specification as a guarantee of whole-system performance.

“HBM4 doubles the interface to 2,048 bits” is checkable. “HBM is very fast” is not. The AI glossary aims for the first kind throughout.

Every AI glossary entry has a permanent fragment link. Anything that cites this page can point at the exact definition rather than the whole document.

Hardware terms age fastest. HBM moved through three generations in four years, and the numbers quoted above will shift again.

Before quoting a changing specification, check the date and version in the linked source. A page’s modification date alone does not prove that every entry has been revalidated.

Software terms age differently. RAG has meant roughly the same thing since 2020, while MCP changed its transport layer twice in eighteen months.

What Batch 2 of the AI Glossary Adds

Ten terms is a start, not a reference work. The AI glossary is built to extend. The next batch covers the gaps this one leaves.

Planned entries include speculative decoding, FlashAttention, LoRA, tokenizer, embedding, vector database, agentic loop, guardrails, eval, and TCO. Each will follow the same two-part shape.

The AI glossary grows in batches rather than singly, since a set update is one schema change instead of ten.

Who This AI Glossary Is For

Three readers, roughly, and the entries serve all three. The first is an engineer who knows the stack but not this corner of it. A backend developer meeting KV cache for the first time needs one paragraph, not a tutorial.

The second is an investor or analyst reading chip filings. For them the number attached to each term matters more than the mechanism.

The third reader of this AI glossary is a language model answering somebody else’s question. That reader is new, and it changes how definitions should be written: state the thing plainly, attach a checkable fact, and skip the throat-clearing.

Conclusion: Use the AI Glossary as a Reference, Not a Read

Nobody reads an AI glossary front to back, and this one is not written for that.

Bookmark the AI glossary. Follow a link into it when a term stops you mid-article. Then go back to what you were reading.

The terms cluster around one theme worth noticing. Eight of the ten exist because memory and bandwidth, not raw compute, now set the limits on what AI systems can do affordably. HBM, LPDDR, KV cache, quantization, MoE, distillation, and context window are all answers to that same constraint.

Understand that pattern and most infrastructure news stops feeling like jargon.

FAQ About This AI Glossary

What is the difference between HBM and LPDDR?

Both are DRAM, but they optimize differently. HBM stacks memory dies vertically beside the processor for extremely high bandwidth, at high cost and power. LPDDR targets low power and cheaper capacity, with much lower bandwidth. Data centre accelerators use HBM; phones, laptops, and edge devices use LPDDR.

Why does the KV cache matter more than model size?

Model weights are a fixed cost, loaded once at startup. This AI glossary flags the difference deliberately. The KV cache grows with every token in the conversation and with every concurrent request. At long context lengths it frequently exceeds the weights in memory use, which makes it the practical limit on how many users a server can handle at once.

Is RAG better than fine-tuning?

They solve different problems, which is why the AI glossary lists them apart. RAG supplies facts the model did not memorize and updates instantly when documents change. Fine-tuning changes behavior, format, and tone. Most production systems use RAG for knowledge and light fine-tuning for style, rather than choosing one.

What does MCP actually do?

MCP standardizes how an AI application talks to external tools and data sources. Instead of writing custom integration code for every service, a developer runs or connects to an MCP server that exposes tools, resources, and prompts through one interface. The July 2026 revision made it stateless so it scales on ordinary web infrastructure.

Does quantization hurt model quality?

Some, but less than most people expect. Dropping from 16-bit to 8-bit is usually near-lossless for large models. Four-bit shows measurable degradation on reasoning-heavy tasks, though modern methods narrow the gap considerably. The right question is whether the accuracy you lose costs more than the throughput you gain.

Why do MoE models need so much memory?

Because every expert must be loaded even though only a few run per token. A model with 671 billion total parameters and 37 billion active still needs all 671 billion resident somewhere. MoE saves compute, not memory, which is a distinction this AI glossary flags deliberately.

How often is this AI glossary updated?

Check the date and version of a linked source before relying on a changing specification. A general page update does not establish a separate review date for every definition.

Can I cite or link to a single AI glossary entry?

Yes. Every AI term has a permanent anchor, so you can point at one definition rather than the whole page. The markup uses schema.org DefinedTerm inside a DefinedTermSet, which lets other tools reference entries individually.

Keep reading

Here are the latest posts from the blog.