Modern AI chatbots are built on large language models (LLMs), a different paradigm from the pattern tricks in Pre AI Chatbot Tricks. This article will try and cover the core concepts behind how these things work.
An LLM does not read whole words. Text is broken into tokens: chunks that can be whole words, parts of words, or single characters, depending on how common they are.
Everything downstream (cost, context limits, generation) is counted in tokens, not words or characters. We pay for tokens.
The transformer architecture (2017) is what the LLMs of today are built on. The primary mechanism, attention, lets the model weigh in every other token in the input when processing a token. This lets it tell that "it" refers to "the skibidi toilet" three sentences back, rather than just looking at the words nearby.
This replaced older architectures that used to process text left to right, one word at a time, which would struggle to connect "ideas" across long stretches of text.
Training will happen in stages. Pre-training feeds the model a bunch of text and will train it on one task... predict the next token. Nothing more: no instructions, no conversation.
Fine-tuning comes next. Additional training happens on smaller datasets that teach the model to follow instructions, hold a conversation and become an assistant rather than just really good at predicting the next token.
RLHF (Reinforcement Learning from Human Feedback) is a fine-tuning step where real humans will rank different model outputs, and the model is trained to produce more of what was ranked highly. This is a huge part of what differentiates a raw pretrained model from a chatbot that gives helpful, on-topic answers. The term for that broader goal is alignment.
The context window is the max number of tokens that a model can use at once. Your messages, the replies, and anything else fed in like docs/pics/instructions. Once the conversation exceeds the context window, earlier content must be dropped or summarized to make room.
At each of these steps, the model will output a probability for each possible next token, not one fixed answer. Sampling is how one gets picked. Temperature controls how random the pick is. Low temperature will stick close to the most likely token every time (more repetitive, more predictable); high temperature will pick from further down the probability list more often (more varied, morelikely to get weird.
A prompt is the text sent to the model to create a response. A system prompt is a special, mostlikely hidden, prompt set by the application (not the user) that sets the model's role, tone, and rules even before the conversation starts.
Prompt engineering is the art of wording your prompts to get more reliable output.
A hallucination is a confidently incorrect output: a fake citation, a straight up lie, code that doesnt work. It happens because the model is actually doing what it is designed to. Giving you what you want to hear. gotta be careful about this stuff. This is the nature of predictie text.
RAG reduces the hallucination by fetching relevant documents from an external source like a website at query time and inserting that into the prompt. This lets the model ground its answer with current text instead of only what was stored/curated during training.
this is kinda close to the retrieval matching used by old chatbots, except the retrieved text becomes an input to a generative model, not the straigt up output.
An agent is a model that is given access to external tools like websites/databases/email/whatever and given the ability to run without input from a human. Can be Sus if you dont know what you are doing or if you give it too much access.
A multimodal model can handle more than just text; it can handle images, audio or video by converting them into the same kind of token that we talked about earlier. This lets one model reason across "modalities" instead of needing a separate model for each format.