Chain of Thought
A prompting technique that improves accuracy on complex tasks by having the LLM explicitly generate intermediate reasoning steps.
What is Chain of Thought
Chain of Thought (CoT) is a prompting technique that improves accuracy on complex tasks by explicitly having an LLM generate intermediate reasoning steps.
Understanding Through a Concrete Example
For a problem such as "There are 3 apples and 5 oranges. What is the total?", instead of having the LLM answer "8" directly, it is guided to output the intermediate process: "3 apples + 5 oranges = 8." The difference is hard to notice with simple addition, but accuracy improves significantly for problems involving multi-step reasoning or conditional branching—such as determining whether legal requirements are satisfied.
Simply adding "Please think step by step" to a prompt can be effective. This is called Zero-shot CoT.
Relationship with Reasoning Models
Reasoning models are designed with CoT built into the model itself, automatically generating a chain of thought without any prompting. On the other hand, CoT can also be elicited from standard LLMs through prompt engineering, so the practical approach is to first try it on the prompt side, and switch to a reasoning model if the accuracy is insufficient.
One important caveat: CoT increases the number of output tokens, which raises costs. Rather than applying it to all requests, the smart operational approach is to limit its use to queries where accuracy is critical.
Articles covering this term
- Latency Budget Design for AI Agents — How to Control the Trade-off Between Thinking Time and Response TimeExplore how "thinking time" causes response delays in multi-step reasoning agents, and compare latency budget allocation strategies and implementation patterns based on task complexity.
- What Is a Token Trap? Practical Token Consumption Management to Prevent Hidden Cost Explosions in AI AgentsLearn how "token traps" cause billing spikes in high-frequency agent loops, and how to prevent cost explosions with budget caps, throttling, and smart loop design.
- What is Inference-Time Scaling? How to Optimize the Trade-off Between AI Inference Cost and AccuracyLearn how Test-Time Compute scaling works and how to optimize the tradeoff between inference cost and accuracy. A practical guide to LLM operations and cost design in the age of reasoning models.
- What is Dynamic Prompt Routing? A Design for Optimally Selecting LLMs Based on QueryBeyond single LLM limits: compare routing layer mechanisms that dynamically assign optimal models and templates based on query content, with implementation patterns.
Related Terms
Context Engineering
Context Engineering is a technical discipline focused on systematically designing and optimizing the
System Prompt
A system prompt is an instruction given to an LLM before the start of a conversation with a user, de
AI Red Teaming (AI Red Teaming)
An evaluation method that systematically tests AI system vulnerabilities from an attacker's perspect
Grounding
A technique that cross-references LLM outputs with external data sources and search results to gener