What Is Chain-of-Thought Prompting?

Chain-of-thought prompting is a technique used to guide large language models (LLMs) to complete tasks using a logical sequence of steps. Even though AI models attempt to mimic humans, they require detailed instructions, called prompts, to create high-quality, accurate, and relevant output. Prompt engineering is the art and science of creating good prompts. Chain-of-thought (CoT) reasoning is a prompt engineering technique that breaks down complex tasks into a series of simpler, more manageable steps to arrive at an answer. It gives users greater control and insight into how the LLM arrives at its responses.

Chain-of-thought example

“Solve the following math word problems step by step.

If you have 7 bananas and give 4 bananas to your friend, then you receive 5 more bananas, and afterward, throw out 3 rotten bananas; how many bananas do you have left?”

Chain-of-thought output

Let’s break it down step by step:

1. You start with 7 bananas.

2. You give 4 bananas to your friend, so you have: 7 - 4 = 3 bananas left

3. You buy 5 more bananas, so you now have: 3 + 5 = 8 bananas

4. You throw out 3 bananas, so you have: 8 - 3 = 5 bananas left

Therefore, you have 5 bananas left.

Chain-of-thought prompting works well for complex reasoning tasks with many steps or other rule-based reasoning tasks.

Mathematical reasoning

Mathematical equations are a popular use case for chain-of-thought prompting. Equations have many rules that apply in various orders and contexts, so solving one requires the LLM to use these rules correctly. Guiding the order and context via CoT prompting helps the LLM arrive at the correct answer.

Software coding and debugging

Software coding and debugging often require multiple steps to solve a problem. In coding, CoT can suggest the best coding strategy and algorithmic next steps to the LLM. In debugging, chain-of-thought prompting guides the LLM to start at the observed bug and step back through breakpoints to determine where the bug originated. CoT prompting is also used to discover and patch software vulnerabilities.

Commonsense reasoning

Commonsense reasoning uses a humanized, contextual, logical process to solve a problem. It becomes necessary for common sense questions like ‘Why can’t you place a baby alone in a swimming pool?’ CoT can guide the prompt by providing cultural and human context so that it reasons correctly.

Symbolic reasoning

Symbolic reasoning involves applying rules to solve problems represented using symbols, expressions, and relationships—for example, decoding a message using a multi-step cipher or analyzing visual data. CoT can guide the LLM in processing these symbols systematically, applying logical operations to transform inputs into meaningful outputs. Unlike statistical or probabilistic methods, which rely on data patterns, symbolic reasoning depends on explicitly defined rules to make deductions.

Complex question reasoning

Many questions have multiple parts that need to be broken down to answer. Complex tasks with multi-step logical reasoning are a good fit for chain-of-thought prompting. An example of a question that requires complex reasoning is: “Write a cold marketing email from my company outlining how we can help improve our client’s market position. Evaluate their business using the SNAP Product Strategy and base your email on evaluation results.”

Chain-of-thought prompting works by setting up a reasoning chain or logical thought process sequence in the LLM prompt. The reasoning chain outlines the steps the LLM needs to follow to arrive at the best solution. The LLM may also be asked to output intermediate reasoning steps for debugging.

Designing the CoT prompt

Instruct the LLM to explain its thought process before reaching a conclusion. Ensure the prompt explicitly tells the model to think through each step logically. Ambiguous or vague prompts may lead to incomplete or incorrect reasoning.

For example, “Explain why the sky appears blue, starting with how sunlight interacts with the atmosphere and ending with how human eyes perceive color.”

For multi-step problems, guide the model to break them down into smaller components before synthesizing an answer. Give examples where possible. You can also prompt the AI to justify its response by explaining its logic. This reduces hallucinations and ensures the model follows a structured reasoning path.

Example: “Is 153 a prime number? Explain your reasoning by testing divisibility rules step by step.”

CoT phrases in the prompt

Consider using the following phrases when writing the prompt

• Explain why

• Explain/give/state your reasoning

• First determine ____ then identify _____

• Follow these steps before you_____

• If ______then do _______ else _______

You can implement multiple chain-of-thought strategies depending on the LLM and the problem.

Multimodal CoT prompting

Multimodal CoT prompting extends the concept of chain-of-thought reasoning to media other than text, like photos, video, or sound. Some complex questions may require other media, so multimodal CoT prompting can incorporate these media into the prompt and sometimes the output. For example, “Use Jen’s method from the video to outline a text-based strategy for developing our app.”

Auto-CoT prompting

Many modern LLMs incorporate automatic chain-of-thought prompting into their process, with an output explaining step-by-step how it arrived at a response. Auto CoT prompting is achieved by adding a “Provide step by step reasoning” pre-prompt into every prompt.

Least-to-most CoT prompting

Least-to-most CoT prompting decomposes complex questions to solve smaller problems before combining them to solve the entire question. To do this, it needs to generate more minor questions and come to the answer to each of its own smaller questions. An example is determining which investments to make based on a person’s profile, goals, financial state, and markets.

CoT is one among several advanced prompt engineering techniques.

CoT prompting vs. standard prompting

Standard prompting does not provide the LLM with further instructions on processing, structuring, or conducting a query; it simply returns an answer. In contrast, CoT prompting breaks down prompts into steps for output. CoT prompting is more accurate and debuggable than standard prompting. However, the lengthy output it generates may not be needed for straightforward tasks.

CoT prompting vs. zero shot prompting

Zero-shot prompting occurs when the user provides no examples of how the LLM is to do its job. For example, the prompt “Translate ‘My name is’ into Finnish” is a zero-shot prompt. When zero-shot prompting does not result in an accurate response, the user often needs to add more context to the prompt to achieve an acceptable output.

CoT prompting can be combined with zero-shot prompting to solve a problem based on the model’s inference alone, known as Zero-Shot CoT prompting.

CoT prompting vs. few-shot prompting

Few-shot prompting is when the user provides a few examples to help the LLM arrive at a solution. For example,

Based on the following names and roles, write a sentence introducing how we can do business together with my app startup:

Mike, Software Engineer -> “Hi John, I thought I’d reach out to see if you were interested in collaborating on our app project.”

Jen, Recruiter-> “Hi Jen, My app is currently looking for developers, and I was wondering if you could help me with recruitment?”

Ben, Investor ->

Few-shot prompting guides the LLM in producing the desired output the user requires. It can be combined with CoT for complex problems needing further user guidance. This is known as the few-shot chain-of-thought prompting.

CoT prompting has the following limitations.

Requires larger LLM models

Smaller LLM models perform poorly with chain-of-thought prompting tasks, ranking similarly to standard prompting. Instead, larger models with around 100 billion parameters are more reliable for accuracy in CoT prompting.

The final answer may not always reflect the chain of reasoning

When a step-by-step answer is given, the user trusts the answer implicitly. However, the LLM’s final answer may sometimes not match the chain of reasoning steps provided in the output. Thus, users must always check the steps and the answer for reliability.

Not suitable for all tasks

There are plenty of use cases where chain-of-thought prompting is unnecessary, including when performing pattern recognition, creative writing, and simple factual answers.

Generative AI services on AWS allow you to innovate faster with new capabilities, a choice of industry-leading AI models, and infrastructure that pushes the envelope to deliver the highest performance while lowering costs. For example,

• Amazon Q is a generative AI assistant that transforms how work gets done in your organization.

• Amazon Bedrock is a fully managed service that accelerates the development of generative AI applications using language models through an API without managing infrastructure. Amazon Nova is a new generation of state-of-the-art (SOTA) foundation models (FMs) available exclusively on Amazon Bedrock.

• Amazon SageMaker AI is a service to build, train, and deploy AI models for any use case with fully managed infrastructure, tools, and workflows. For example, you can work with popular models like OpenLLaMA, RedPajama, MosaicML’s MPT-7B, FLAN-T5, GPT-NeoX-20B, and BLOOM. Many models available on AWS incorporate Auto-CoT.

Get started with chain-of-thought prompting on AWS by creating a free account today.