Academy / Modern AI Concepts
← Back to Neverot

Modern AI Concepts

How the AI chatbots of today actually work
Modern AI (LLMs)
Time period
2017 to present
Field
Machine learning
Famous examples
GPT, Claude, Gemini, OLlama
Main idea
Predict the next token from the patterns learned across massive amounts of text
Related
Pre AI Chatbot Tricks (the past or something) · AuthN & Identity Concepts (how AI agents authenticate to stuff)

Modern AI chatbots are built on large language models (LLMs), a different paradigm from the pattern tricks in Pre AI Chatbot Tricks. This article will try and cover the core concepts behind how these things work.

Contents
  1. Tokens
  2. Transformers / attention
  3. Pretraining / fine-tuning
  4. Reinforcement Learning from Human Feedback (RLHF)
  5. Context window
  6. Temperature / sampling
  7. Prompting / system prompts
  8. Hallucination
  9. Retrieval-augmented generation (RAG)
  10. Agents / tool use
  11. Multimodal models
  12. Glossary
  13. References

1. Tokens

An LLM does not read whole words. Text is broken into tokens: chunks that can be whole words, parts of words, or single characters, depending on how common they are.

Text: chatbots can be unpredictable Tokens: "chat" "bots" " are" " un" "predict" "able"

Everything downstream (cost, context limits, generation) is counted in tokens, not words or characters. We pay for tokens.

2. Transformers / attention

The transformer architecture (2017) is what the LLMs of today are built on. The primary mechanism, attention, lets the model weigh in every other token in the input when processing a token. This lets it tell that "it" refers to "the skibidi toilet" three sentences back, rather than just looking at the words nearby.

This replaced older architectures that used to process text left to right, one word at a time, which would struggle to connect "ideas" across long stretches of text.

3. Pretraining / fine-tuning

Training will happen in stages. Pre-training feeds the model a bunch of text and will train it on one task... predict the next token. Nothing more: no instructions, no conversation.

Fine-tuning comes next. Additional training happens on smaller datasets that teach the model to follow instructions, hold a conversation and become an assistant rather than just really good at predicting the next token.

4. RLHF / alignment

RLHF (Reinforcement Learning from Human Feedback) is a fine-tuning step where real humans will rank different model outputs, and the model is trained to produce more of what was ranked highly. This is a huge part of what differentiates a raw pretrained model from a chatbot that gives helpful, on-topic answers. The term for that broader goal is alignment.

5. Context window

The context window is the max number of tokens that a model can use at once. Your messages, the replies, and anything else fed in like docs/pics/instructions. Once the conversation exceeds the context window, earlier content must be dropped or summarized to make room.

6. Temperature / sampling

At each of these steps, the model will output a probability for each possible next token, not one fixed answer. Sampling is how one gets picked. Temperature controls how random the pick is. Low temperature will stick close to the most likely token every time (more repetitive, more predictable); high temperature will pick from further down the probability list more often (more varied, morelikely to get weird.

Familiar territory: this is the child of the weighted randomness trick covered in Pre AI Chatbot Tricks §6: dice rolls, but the pool and its weights are handled by the model instead of manually coded.

7. Prompting / system prompts

A prompt is the text sent to the model to create a response. A system prompt is a special, mostlikely hidden, prompt set by the application (not the user) that sets the model's role, tone, and rules even before the conversation starts.

Prompt engineering is the art of wording your prompts to get more reliable output.

8. Hallucination

A hallucination is a confidently incorrect output: a fake citation, a straight up lie, code that doesnt work. It happens because the model is actually doing what it is designed to. Giving you what you want to hear. gotta be careful about this stuff. This is the nature of predictie text.

9. Retrieval augmented generation (RAG)

RAG reduces the hallucination by fetching relevant documents from an external source like a website at query time and inserting that into the prompt. This lets the model ground its answer with current text instead of only what was stored/curated during training.

this is kinda close to the retrieval matching used by old chatbots, except the retrieved text becomes an input to a generative model, not the straigt up output.

10. Agents / tool use

An agent is a model that is given access to external tools like websites/databases/email/whatever and given the ability to run without input from a human. Can be Sus if you dont know what you are doing or if you give it too much access.

11. Multimodal models

A multimodal model can handle more than just text; it can handle images, audio or video by converting them into the same kind of token that we talked about earlier. This lets one model reason across "modalities" instead of needing a separate model for each format.

See also

Glossary
Agent
AI model that is given the ability to use apps and stuff.
Alignment
Training a model to be helpful and answer correctly instead of just predicting the next thing to say.
Context window
The maximum number of "tokens" that a model can look at at a time
Hallucination
A confident incorrect response from an AI model.
LLM
Large language model: a model which is trained to predict text at scale.
RAG
Retrieval augmented generation: the fetching of outside documents into the prompt to help pad an answer.
RLHF
Reinforcement Learning from Human Feedback: training a model thatt uses human rankings of its outputs.
Token
A chunk of text like a word, part of a word, or character: the basic unit that a model reads and generates.
Transformer
The architecture behind LLMs
References
  1. Vaswani, A. et al. (2017). "Attention Is All You Need." NeurIPS. Introduced the transformer architecture. arxiv.org/abs/1706.03762
  2. Sennrich, R., Haddow, B., & Birch, A. (2016). "Neural Machine Translation of Rare Words with Subword Units," subword/BPE tokenization. arxiv.org/abs/1508.07909
  3. Brown, T. et al. (2020). "Language Models are Few-Shot Learners" (GPT-3). NeurIPS. arxiv.org/abs/2005.14165
  4. Ouyang, L. et al. (2022). "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT), RLHF. arxiv.org/abs/2203.02155
  5. Lewis, P. et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." NeurIPS. arxiv.org/abs/2005.11401
  6. Ji, Z. et al. (2023). "Survey of Hallucination in Natural Language Generation." ACM Computing Surveys. arxiv.org/abs/2202.03629
  7. Schick, T. et al. (2023). "Toolformer: Language Models Can Teach Themselves to Use Tools." arxiv.org/abs/2302.04761