Enhancing Text-to-SQL Agents with Step-by-Step Reasoning

Unlock Deeper Thinking in non-CoT LLM

One cool outcome of the DeepSeek R1 release is that LLM is now starting to show the Thinking <think> tokens in the response, similar to ChatGPT-o1 and o3-mimi. Encouraging an LLM to think more deeply has a lot of benefits:

• No more black-box answers! You can see the reasoning behind your LLM’s responses in real-time.
• Users get insight into how the model reaches its conclusions.
• Spot and fix prompt mistakes with clarity.
• Transparency makes AI decisions feel more reliable.
• When humans and AI share reasoning, working together becomes effortless.

So here we are, I’ve built a RAG that brings a similar reasoning process (CoT responses) to the LangGraph SQL agent with tool calling. It is a ReAct agent (Reason + Act) that combines LangGraph’s SQL toolkit with a graph-based execution. Here’s how it works:

Now, let’s understand the thinking process.

The agent starts with a system prompt that structures its thinking:

I’ve mapped out the exact steps our SQL agent takes, from the moment it receives a question until it returns the final query: