Large language models, or LLMs for short, are AI models that understand and generate human language. Well-known examples are GPT from OpenAI, Llama from Meta and the models from Mistral. LLMs are part of generative AI. They answer questions, summarize documents, translate or write program code.
How does an LLM work?
During training, an LLM learns from huge amounts of text how words and sentences relate to each other. In the process, it adjusts billions of parameters. These parameters hold the knowledge of the model. When it receives a request, the model calculates token by token which continuation fits best. The foundation is the transformer architecture, on which most language models have been based since 2017.
What are tokens, context and context length?
A token is the smallest unit of text an LLM works with. It can be a whole word, part of a word or a punctuation mark. In English, one token roughly corresponds to four characters, and a longer word often consists of several tokens.
The context covers everything the model takes into account for a request: your question, the previous chat history, attached documents and the answer generated so far. The context length indicates the maximum number of tokens the model can process at once. Current models often handle 128,000 tokens or more, which corresponds to several hundred pages of text. The longer the context, the more memory the model needs.
What is the difference between training, fine-tuning and inference?
Training creates the model from scratch. This takes weeks and often occupies thousands of GPUs. With fine-tuning, companies adapt a finished model with their own data, for example to technical terms or internal documents. Inference is the ongoing operation in which the model answers requests.
What hardware does an LLM need?
The size of a model determines its memory requirements. As a rule of thumb, each parameter needs two bytes at 16-bit precision and only one byte at 8-bit precision (FP8). Since many current models are released in FP8, this is the practical basis for calculation: a model with 70 billion parameters occupies around 70 GB for the model weights alone (at 16 bits it would be 140 GB).
On top of that comes the KV cache. This is the buffer in which the model stores the intermediate results (keys and values) of all previous tokens in the context. This way, it does not have to recalculate them for each new token. The KV cache grows with the context length and the number of parallel requests and can occupy several tens of GB when there are many users.
A single NVIDIA H200 has 141 GB of GPU memory. A 70B model in FP8 fits on one card, including a generous KV cache. Larger models, by contrast, usually run distributed across several GPUs, for example in a GPU server with eight GPUs.
A more cost-effective alternative is the NVIDIA RTX PRO 6000 Blackwell with 96 GB of GDDR7 memory. A 70B model in FP8 fits on a single card with around 70 GB. That leaves about 26 GB for the KV cache, enough for long contexts and several parallel requests. If you need more context or parallelism, you can switch to 4-bit formats such as NVFP4, which Blackwell supports natively. They halve the weights to around 35 GB, with slight quality losses, especially in complex reasoning. With two to four cards in one server, even models with more than 100 billion parameters can be run without needing a large eight-GPU system. Thanks to its PCIe connection and air-cooled variants, the card can also be integrated into standard racks.
Many companies run LLMs locally in their own data center so that sensitive data stays in-house.
In the MEGWARE Benchmark Center, you can test which hardware your model needs.
FAQ LLM
Parameters are the numerical values that a model learns during training. They determine how the model weights and links words. A model with 70 billion parameters therefore has 70 billion such values stored.
An LLM is the language model itself. ChatGPT is an application from OpenAI that is based on the GPT models. It adds a chat interface, safety filters and other functions to the model.
An LLM hallucinates when it generates plausible-sounding but false statements. This happens because the model calculates probable word sequences and does not check facts. Companies reduce this risk by connecting the model to verified internal documents. This method is called retrieval augmented generation (RAG).
Yes. Open models such as Llama, Mistral or Qwen also run on your own servers. A single GPU is often enough for small models, while large models are distributed across several GPUs. Running them locally keeps sensitive data in your own data center.
Providers train large models over weeks to months on thousands of GPUs. For companies, fine-tuning is usually what matters. It lets you adapt a finished model to your own data, which takes hours to days depending on the model size and hardware.