Created by the Author using AI and Figma

Structured, Traced, Versioned: Adding AI Like a Pro

How I learned to love LangChain, Langfuse & Azure OpenAI

5 min readJul 22, 2025

Motivation

Do you remember the first time you tried to get an LLM to return structured JSON? Are you still screaming at it?

“No backticks or extra symbols.”
“Make sure its valid JSON. ONLY JSON.”
“FOLLOW THE FORMAT”

And what did you get back?
Not what I wanted for sure.
A JSON type structure wrapped in Markdown. With bonus trailing commas.
Wonderful…NOT.

Let’s be honest - adding AI to your app used to be a total pain.

But in 2025, it’s actually kinda fun and easy! What?? YES!!!

Let me show you how I went from shouting at prompts to sipping coffee while my application reasons, responds, and even follows my damn schema — FINALLY!

With a little help from:

LangChain → structure-obsessed, TypeScript-loving toolkit
Langfuse → my AI debugger and prompt control master chief center
Azure OpenAI → production-grade, enterprise-ready AI models

And yes, there’s little bit of code.

LangChain: Turning Messy LLM Responses into Valid JSON — Like a Boss!

I used to spend hours trying to get LLMs to return {"email": "me@example.com"} instead of "Sure! Here's your email:\n- me@example.com". or JSON`{"email":"me@example.com"}`

Then LangChain’s SDK entered our chat.

With LangChain, you just define the structure you want, and it handles the rest. No more begging. No more regex. Just structured results!

Real-world Example: Parsing a user message into structured data:

This is my favorite trick. I used to be begging. Now I control them.

LangChain also supports multi-step workflows, function calling, retrievers, tools, memory, agents and what not…but start here first.

Langfuse: I can finally spy on my Prompts

Let’s play a simple Q&A game.

How much does each LLM call in your app cost?

Don’t know?

What’s the average latency of your prompt pipeline?

Not sure?

Which prompt version caused your customer to email you with
“WHY IS THE AI NOT WORKING”?

Exactly. My. Point.

That’s why I love Langfuse — it’s like having x-ray vision for your AI backend and prompts. And it makes prompt tweaking fun.

What does Langfuse do?

Logs every prompt and response
Tracks token usage and $$$ cost
Let’s you version and test prompts without redeploys — OMG!!!
Works beautifully with LangChain

I think you are sold…

Now you’ve got:

A trace in the UI
A full audit trail for what was sent and received
Token cost and latency per call

Huge Bonus: Prompt versioning without Redeploys

Langfuse lets you define and tweak prompts right in the dashboard.
No more shipping your app because you wanted to change “You are a helpful assistant.” to “Be helpful and brief.”

You’ll never write prompts in const prompt = again. Trust me.

Azure OpenAI — Because it’s good.

So you’ve got your structured calls and observability.
Now you need a solid, scalable, secure place to actually run those LLMs.

Enter: Azure OpenAI.

Why I use it:

Built-in enterprise security, networking & logging
Plays REALLY well with LangChain
Great for production workloads

Setting it up is genuinely straightforward — no arcane CLI rituals.

⚡️ Plug it into LangChain like this:

import { AzureChatOpenAI } from "@langchain/openai";

const model = new AzureChatOpenAI({
  azureOpenAIApiKey: process.env.AZURE_OPENAI_API_KEY!,
  azureInstanceName: "my-ai-instance",
  deploymentName: "gpt-4",
  apiVersion: "2024-05-01-preview"
});

const result = await model.invoke(prompt);

Scale when you need to. Stay compliant by default.
Bonus: no weird limits like some other OpenAI clones out there.

Start Simple. Scale Later.
Don’t Go Full Agentic (Yet).

You don’t need to build an AGI overlord from day one.

Start with:

A few structured prompts (LangChain)
Real-time tracing and control (Langfuse)
Secure, scalable infra (Azure OpenAI)

Then, if your app grows into needing tool use, memory, agents and so on…all that fancy stuff — LangChain has your back.
But 90% of value comes from just getting structured, observable responses into your UX.

Think MVP — not Matrix.

Finally AI That Doesn’t Make You Want to Quit Programming

AI used to feel like a black box. Now it’s genuinly a building block.

Thanks to these tools, you can now:

Build smarter apps
Cleanly integrate LLMs into your stack
Debug, trace, and optimize with confidence
Iterate quickly without freaking out

So go ahead. Add AI to your app.

You’ll be amazed how far you can get in a weekend!

Common Gotchas When Adding AI

Let’s be real: even with LangChain, Langfuse, and Azure OpenAI, AI is still weird sometimes and smokes too much of its good stuff.
Here’s what tripped me up several times:

Structure is NOT Perfect Responses
LangChain’s StructuredOutputParser is incredible but remember, the model can still hallucinate. Use temperature:0for more deterministic tasks and add few-shot examples to your prompt.
Invalid JSON is NOT Invalid AI.
Models love missing commas, markdown formatting, etc.
Always wrap model output parsing in try/catch, and use LangChain parsers that validate against zod.
Don’t trust raw JSON.parse() without LangChain.
Token Cost is Real
Debugging with GPT-4? Just one “oops” in your prompt loop and…“You’ve spent $89.99 today.”.
Use Langfuse to trace, monitor token usage, and test with smaller models before switching to the big guns. Add guardrails before shipping infinite loops.
Prompt Versioning is NOT Prompt Consistency
You updated a prompt in Langfuse. Your app is still using the old one. Or you have 12 variations and forget which is live.
Name your prompt versions clearly ("v1-email-parser", "v2-cv-summarizer") and sync them with branches or feature flags. Use environment labels and test them.
Agents = Overkill → until not
Everyone wants to use agents and tools and memory and action plans… until it turns into an unpredictable mess.
Start with single-shot prompts + LangChain flows. Move to agents only when your app genuinely needs planning, chaining, and decision trees. Until then? KISS (Keep It Simple, STUPID).
API with Azure OpenAI
The deploymentName isn’t the model name.
apiVersion is required.
Read the docs. Slowly. Twice.
Secrets, Keys & Managed Identity
Azure lets you use Managed Identity, but LangChain and Langfuse might still ask for API keys.
Store secrets in Key Vault and inject them securely.
Use DefaultAzureCredential wherever you can, and wrap LLM calls in your own abstraction if you're managing both local dev and cloud infra.