Multi-step reasoning

Limitations of Simple Question Answering

A factual lookup such as "What is the revenue?" can be completed in a single step. However, a question like "What initiatives did the person in charge of the division with the highest year-over-year revenue growth introduce?" cannot be answered without going through multiple intermediate steps: comparing sales data → identifying the division → identifying the person in charge → searching for information on the initiatives.

Multi-step reasoning refers to an approach in which an LLM internally decomposes such complex questions and solves them incrementally. It is an extension of Chain-of-Thought (CoT) prompting, but differs in that, when combined with RAG, a search of external data sources is inserted at each step.

Relationship with Agentic RAG

Agentic RAG can be understood as an implementation of multi-step reasoning as an agent loop. The agent determines "what should be investigated next" and cycles through search → evaluation → re-search. Multi-step reasoning is the design pattern for that thought process, while Agentic RAG is the architecture that executes it.

Accuracy vs. Speed Trade-off

The more steps involved, the greater the comprehensiveness of the answer; however, the LLM inference cost and search latency accumulate at each step. In practice, many systems are designed with an upper limit on the number of steps (typically 3–5) and an early termination mechanism that triggers once sufficient information has been gathered.