If you’ve built a basic RAG pipeline, you’ve done this: chunk your documents, embed them with a model like text-embedding-3-small, store the vectors, and at query time embed the user’s question and pull back the top-k most similar chunks.

It works. Sort of. Then one day a user asks a reasonable question, and your pipeline returns chunks that are about the right topic but don’t answer the question.

Sometimes worse. The user asks “Is ibuprofen safe to take during late pregnancy?” The pipeline returns this chunk as the top match:

“Ibuprofen is widely used and considered safe for general pain relief.”

The chunk that actually answers the question sits two ranks below:

“Ibuprofen is not recommended after 20 weeks of pregnancy due to fetal kidney risks.”

Both chunks are about ibuprofen safety, so vector search ranks both highly. But only one answers the question the user asked. The pipeline can’t tell the difference, and confidently hands back the wrong chunk.

This post explains why that happens, and how re-rankers fix it.

I published a companion GitHub repo with a runnable notebook that demonstrates three common ways vector search fails (negation, missed constraints, topic-over-answer) and shows the re-ranker fixing them.

The Library Analogy

A giant library. Every book has many sections, each holding facts, claims, definitions, maybe answers to questions someone will ask one day.

You need to make it searchable. So you hire a librarian to summarize every section on a small card. The catch: she writes the cards before any reader walks in, with no idea what readers will ask.

What’s the best she can do? Write a high-level summary. “This section discusses the causes of type 2 diabetes.” “This section covers lithium-ion battery thermal runaway.”

These cards capture the topic. They don’t capture every fact, every constraint, every nuance. A one-line summary loses detail by definition.

That’s what an embedding model does. Feed it a chunk of text and you get back a vector: a list of numbers, usually between 384 and 1536 long, depending on the model. The vector is the digital summary card. It captures roughly what the chunk is about, written before any question is asked.

A reader walks in with a question. She writes that on a card too. The librarian compares the question card against every section card and hands back the closest matches.

That’s vector search. Embed the question, compute similarity against the pre-computed chunk vectors, return the top matches.

How Embedding Models Work, and Where They Break

An embedding model is a transformer encoder, a model that reads text and produces numbers that represent it. In vector search, the encoder runs twice: once on each chunk (ahead of time, to build the index) and once on each query (at search time). The two outputs (vectors) are then compared by a similarity score computed outside the model.

This architecture is called a bi-encoder. The model never sees the query and chunk together.

The bi-encoder design is what makes vector search fast. You pre-compute every chunk vector once, store them in an index, and reuse them for every future query. At query time, you embed the question once and search across millions of vectors in milliseconds.

The architecture has two compressions, both lossy:

Chunks get compressed before anyone asks a question. A chunk holds several facts, possibly answers to several questions, plus context and noise. All of it squashes into one vector: roughly the aggregate of everything in the chunk, in the model’s coordinate space (often called “semantic space” because chunks with similar meanings land near each other). No fact gets its own representation. The model doesn’t know which fact will matter for a future question, so it can’t favor any.

Questions get compressed without knowing which document they’ll meet. Same problem on the question side. Intent, constraints, framing: all reduced to one topical fingerprint. When you embed “What was Apple’s revenue in Q3 2024?”, the vector mostly represents “Apple revenue recent quarter.” “Q3 2024” barely registers.

You’re comparing two lossy fingerprints. Nothing in your pipeline ever looks at the raw question alongside the raw chunk.

Three Common Ways Vector Search Fails

These are three canonical failure modes the companion repo shows. Each maps to a query in the demo notebook.

Negation gets lost.

• Question: “Is ibuprofen recommended late in pregnancy?”

• Chunk A: “Ibuprofen is widely used for general pain relief.”

• Chunk B: “Ibuprofen is not recommended after 20 weeks of pregnancy due to fetal kidney risks.”

Both chunks share the same vocabulary about ibuprofen, so vector search ranks them similarly. But only Chunk B answers the question about late pregnancy; Chunk A answers a different question, about general pain relief. The model can’t tell the difference.

Constraints get washed out.

• Question: “What was Apple’s revenue in Q3 2024?”

• Chunk A: “Apple’s Q2 2023 revenue grew significantly, reflecting strong iPhone sales.”

• Chunk B: “Apple reported Q3 2024 revenue of $85.8 billion.”

Both chunks share the same vocabulary about Apple and quarterly revenue, so vector search ranks them similarly. But only Chunk B answers the question about Q3 2024; Chunk A answers a different question, about Q2 2023. The exact quarter barely registers in either vector.

Topic match beats answer match.

• Question: “How do I reset my password?”

• Chunk A: “Password security is critical. Use strong passwords, enable 2FA, never share passwords...”

• Chunk B: “To reset your password, click ‘Forgot Password’ on the login page...”

Both chunks share the same vocabulary about passwords, so vector search ranks them similarly. But only Chunk B answers the question about resetting a password; Chunk A answers a different question, about password security best practices.

Every failure traces back to one source: vector search ranks chunks by topical overlap, but topical overlap doesn’t tell you which question each chunk actually answers. The fix is in the architecture.

The Re-Ranker

The architectural fix is to bring the question and the chunk inside the same model. Instead of comparing two vectors outside the model, a re-ranker reads the question and a candidate chunk together in one forward pass, and scores how well the chunk answers that question. It compares them word by word.

This architecture is called a cross-encoder, because the query and passage attend across each other inside the model. A re-ranker is the API around it: you send a query and a list of candidates, and the service pairs the query with each candidate internally, runs the cross-encoder on every pair, and returns the sorted list.

The cross-encoder sees which chunk answers the asked question, even when several chunks share the same vocabulary: late-pregnancy safety vs. general use, Q3 2024 vs. Q2 2023, password-reset instructions vs. password best practices. It returns a relevance score for every pair: higher means more relevant.

If you were searching by hand, you’d do the same thing. You wouldn’t write summary cards in advance. You’d grab a stack of candidate books, sit down with the question in front of you, and read each passage with the question in mind. Then you’d rank them.

Why joint processing works

The cross-encoder’s joint processing, reading the query and passage in one forward pass, works because of the transformer’s attention mechanism. At every layer, each token weights its relationship to every other token in the input. With the question and chunk in the same input, the model can match “Q3 2024” in the question against “Q3 2024” in one chunk and “Q2 2023” in another, and tell which one the question is asking about.

A bi-encoder runs two separate forward passes. The model has no chance to compare words across them. The comparison happens at the end, between two finished vectors, after the word-level details have been compressed away.

Joint processing means more information flowing through the same computation.

How does the model know which words are query and which are passage (answer chunk)?

If both texts share one input, how does the model tell query from passage?

Two special tokens. [CLS] marks the start of the input. [SEP] marks the boundary between query and passage. The combined input looks like this:

[CLS] is ibuprofen safe during late pregnancy ? [SEP] ibuprofen is not recommended after 20 weeks . [SEP]

The model processes both texts together, but always knows which words came from the query and which came from the passage.

Input and output

A re-ranker typically takes 10 to 100 candidate passages per query. It returns a relevance score per passage. Some models return values between 0 and 1; others return unbounded numbers, often in the -10 to +10 range, where higher means more relevant. Most also return a sorted index list so you know which score belongs to which passage.

You sort by score and keep the top N for your LLM.

One constraint: most cross-encoders cap input at 512 tokens for the combined query plus passage. If your chunks are long, the end gets truncated, which quietly degrades quality. Keep chunks under about 400 tokens, or pick a longer-context re-ranker like bge-reranker-v2-m3, which supports inputs up to 8192 tokens (though it’s fine-tuned at 1024, so very long inputs see diminishing returns).

Bi-encoder and cross-encoder, side by side

The bi-encoder and the cross-encoder are built on the same transformer encoder, used differently. The bi-encoder runs it twice and compares vectors outside the model; the cross-encoder runs it once and judges relevance inside.

The bi-encoder gives up precision to search at scale. The cross-encoder gives up scale to score with precision. Each makes up for the other’s weakness, which is why production RAG pipelines use both, in that order.

In your pipeline, the re-ranker call sits between vector search and the LLM:

Why Not Use Re-Rankers for Everything?

A bi-encoder lets you pre-compute every chunk’s vector once and index it. At query time, you embed the question once and search across millions of vectors in milliseconds.

A re-ranker can’t be pre-computed. Every (query, passage) pair runs through the full model at query time. Re-ranking a million chunks per query would take hours.

So you combine them:

1. Retrieval with a bi-encoder. Pull 50 to 100 candidates from the full corpus. Cast a wide net. The goal is recall: make sure the right chunks sit somewhere in those 100. (Recall measures how often the right answer appears in your results at all; precision measures how often the top result is right.)

2. Re-ranking with a cross-encoder. Score each of the 50 to 100 candidates against the question. Sort. Keep the top 5 or 10 for the LLM.

Speed at the recall stage, precision at the ranking stage. The two-stage approach mirrors the human librarian: narrow with rough signals, then read closely to pick the winners.

Why Not Just Let the LLM Filter for Relevance?

A reasonable question: why add a re-ranker at all? Why not send the top 50 chunks from vector search straight to the LLM and ask it to use only the relevant ones?

You can. It loses on accuracy, cost, and latency.

Accuracy. LLMs are bad at picking the most relevant chunks when many candidates sit in the same context window. The “Lost in the Middle” paper (Liu et al., 2023) showed that across GPT-3.5 and several other models, accuracy is highest when the relevant chunk sits at the very start or end of the context, and drops 20 to 30 percentage points when it sits in the middle. Follow-up work has confirmed the same pattern on GPT-4, Claude, and Gemini, even as their context windows grew past 100K tokens. A cross-encoder scores each (query, chunk) pair on its own. Position in a list never enters the math.

Cost. Sending 50 chunks (about 25,000 tokens) to a frontier LLM on every query runs roughly 10x the input-token cost of re-ranking first and sending only the top 5 (about 2,500 tokens). At 10,000 queries a day, the gap adds up.

Latency. Two stages are usually faster than one. A re-rank pass takes 100 to 300ms. Sending 25,000 tokens to a frontier LLM typically takes several seconds for the first token; 2,500 tokens takes one or two. Re-ranker plus a smaller LLM call ends up noticeably faster end to end.

A re-ranker is small, fast, and trained on millions of (query, doc, relevance) pairs to do one job. An LLM is trained to write fluent answers. Asking it to also act as a precise relevance classifier inside the same call moves the work to a slower, costlier, less accurate component.

The exception is small candidate sets. With only 3 to 5 chunks, position bias is mild and re-ranking adds little. Skip it. Past about 10 candidates, the gap opens fast.

When a Re-Ranker Isn’t Worth It

Re-rankers have their cost. Before you add one, check whether the cost is justified:

• Tight latency budgets. Re-ranking adds 100 to 500ms per query. If your end-to-end budget is under a second, that hurts.

• Mostly exact-keyword lookups. For product SKUs, error codes, and literal strings, BM25 (a keyword algorithm that scores documents by term frequency and rarity, used by most search engines before embeddings) often beats vector search on its own.

• High query volume. API re-rankers charge per call. At millions of queries per day, costs add up. Self-host an open-source re-ranker if volume is high.

If none of these apply, and your current pipeline returns “close but not quite” results, a re-ranker is worth exploring.

Re-Ranker Options

A short list of re-rankers in active use. They fall into two buckets: hosted APIs (call an endpoint, pay per query) and open-weights models (download and run yourself).

API-based:

• IBM watsonx.ai Rerank. Hosts several cross-encoder models.

• Cohere Rerank. Current models include rerank-v3.5 (4096-token context) and rerank-v4.0.

• Jina Reranker. Available as API or self-hosted.

• Voyage Rerank. API-based; current models include rerank-2 and rerank-2.5.

Open weights, self-host:

• BAAI/bge-reranker-v2-m3. Multilingual; supports inputs up to 8192 tokens.

• mixedbread-ai/mxbai-rerank-large-v2. Supports inputs up to 32K tokens.

• cross-encoder/ms-marco-MiniLM-L-6-v2. 6-layer MiniLM cross-encoder trained on MS MARCO; commonly used as a starter model.

Summary

Vector search compares fingerprints, not specifics. It compresses every chunk and every question into their vectors, then matches them by topical overlap. That works for “what is this about?” It fails when negations, exact constraints, or specific entities decide the answer.

A re-ranker reads the question and the chunk together, word by word, and scores how well one answers the other. Vector search narrows the corpus to about 50 candidates; the re-ranker picks the winners. Rough filtering, then careful reading.

The payoff is concrete: a correct chunk ranked 30th by embedding similarity gets promoted to position 1 or 2 by the re-ranker. That’s the precision gap the cross-encoder recovers.

If your RAG pipeline returns “close but not quite” answers, try a re-ranker next.

Try This

• Clone the companion repo: github.com/shaikhq/rag-reranker-demo. Run the notebook end to end, see the three failure modes, see them fixed.

• Swap in your own PDF and rerun. The same lift usually shows up on real data.

• Wire a re-ranker (watsonx.ai, Cohere, or a self-hosted BGE model) into your pipeline behind a feature flag.

• Measure top-1 and top-5 accuracy on a small eval set, with and without re-ranking.

• Tune the candidate count (50 vs 100 vs 200) and watch the precision-versus-latency curve.

I walked out of that meeting with a long list of gaps and one thought in my head: how am I going to pull all of this together?

I had just presented my plan for adding vector storage and search to Db2 at the Architecture Review Board, the committee of senior technical leaders who must approve every major new project before a line of code is written. I had walked in confident. I walked out humbled. The committee had found problem after problem in my plan that I had not considered. The list kept growing. Each item was a real use case that a customer would hit, and my plan had nothing to say about any of them.

I had spent nineteen years building software at IBM. I thought I knew what the job required.

I was wrong. Not about the technology. About what it actually takes to own a major AI project from concept to delivery inside a large, mature software product.

My Two Recent AI Projects at Db2

In the past two years I built and delivered two major AI projects at Db2.

The first was vector storage and similarity search, built directly into the database engine. This lets AI applications store collections of vector data inside the database and search through it to find similar items quickly. It is a core capability for modern AI use cases like recommendation engines and semantic search.

The second was connecting large language model APIs to Db2. This project lets application developers call these AI models directly from inside the database using standard SQL commands. Application developers no longer need to write separate code outside the database to do this.

These projects ran from the initial proposal to the final delivery under my technical ownership. More than twenty developers across multiple teams. Tight deadlines. Five Db2 platforms and many more deployment configurations. Db2 is a product used by companies for over thirty years, so a mistake is not just a bug report. It breaks the daily workflow of teams that have depended on it for decades.

The first project shipped to customers last year. The second has completed development and is scheduled to ship in the next few months.

These two projects taught me more about technical leadership than the previous nineteen years combined. Here are the five lessons I learned from leading these projects.

Lesson 1: A new feature must work with everything the product already has and does.

When we started building vector support, I thought the job was simple. Add something new. A new data type for storing vectors. A new way to search those stored vectors. Clean, contained work. Like building a new room in a house without touching the rest.

Users have built numerous business workflows around Db2’s existing features. There are tools for copying data out of the database to a file. There are tools for moving data from one database to another. There are SQL commands for every common task a user needs to do.

When we added the vector data type, users did not care about our internal design. They cared about their work. They expected the new vector type to work with all the same tools they already used every day with other data types. If the copy tool worked with varchar data type, they expected it to work with vectors too.

We were not just adding something new. We were taking on the responsibility of fitting into everything that already existed, without breaking the ecosystem. I had not understood that responsibility before I walked into the Architecture Review Board.

That is where the list of gaps I described in the opening came from. Each gap the committee found was a real use case that a customer would hit, and my plan had nothing to say about any of them.

I went straight to the senior architects and most experienced developers. We went through the list together, gap by gap. I asked for their guidance. We rewrote the plan several times until it was solid. A new shared plan, built together with the senior developers in my squad.

The method that came from those sessions: find the existing feature in Db2 most similar to the one you are adding. Search the entire codebase. Look at every place in the code that handles that existing feature, one by one, and decide whether your new feature needs to do the same thing. This gives you a map before you write a line of code. In a product built over many years, the people who built it will always find problems you missed. Bring them in before you think you are ready.

Lesson 2: A good design does not coordinate teams. A leader does.

The second project, integrating large language model APIs into Db2, made this lesson unavoidable. The feature required work from three component teams in sequence.

To add one of the new functions, we first needed new entries added to the database catalog, a central store that holds information about every object in the database. The catalog team had to finish their work before the query compiler team could start theirs. The query compiler team had to finish before the runtime team could finish theirs. Each team was waiting on the team before them. Like links in a chain.

If the catalog team finished one week late, the compiler team was stuck for a week doing nothing useful. If the compiler team was late, the runtime team was stuck too. The whole project fell behind.

I did not track these connections carefully at the start of the project. I assumed each team would manage their own work and speak up when stuck. They did not. Halfway through the project I could see that work was piling up in the wrong order. Teams were waiting on each other but not saying so out loud. Things were quietly falling apart.

I stopped the usual work and took charge of tracking every dependency myself. I asked each team one direct question: what do you need from another team before you can finish your work? I asked this at every standup meeting. If any work was not helping clear a dependency, I stopped it and pointed the team at what actually needed to happen next.

When a dependency could not be fixed in time, we found a workaround. The runtime team needed specific data from the catalog team, but the catalog work was still two weeks away. We hardcoded a temporary number as a stand-in. The runtime team used that number to keep building, completing 80 percent of their work without waiting. Once the catalog work was done, we swapped the hardcoded number for the real value. The team kept moving. The deadline held.

As a technical leader, you have to find the dependencies between teams, identify them clearly, write them down, and make sure every team can see what they are waiting for, and what other teams are waiting for from them. Do this early, or the project will slow down for reasons that are invisible until it is too late.

Lesson 3: A squad is not formed by putting people on the same project. It is built.

I formed my squad by pulling developers from different teams inside Db2. Some were experts in the runtime layer. Some worked on the catalog. Some worked on the query compiler. Every one of them was skilled and experienced in their own component.

What I did not expect was the resistance: not to the project itself, but to working outside their own component. Developers who spend years on one part of a codebase become very attached to that component. When asked to work in a different part, some resist. Not because they are difficult people, but because it is unfamiliar and they do not want to take responsibility for components owned by another team.

In a project where every layer of the product needs to work together, that reluctance causes real problems.

I had to build a new team culture through action, not words. I asked a runtime developer to write test cases for the query compiler section. I asked a compiler developer to write code for the catalog section. Every time someone stepped outside their usual area and helped another team, I called it out and thanked them. I wanted that behavior to feel normal, not unusual.

There was pushback. One situation stands out. The runtime team was waiting on a piece of work from the query compiler team. But the compiler team was in the middle of something that, if interrupted, would slow the whole project. Simply stopping their work was not the answer.

So we went deeper. We identified the specific code paths shared between the runtime and query compiler components, the intersection where both teams had a stake. Then we brought a runtime developer into that shared territory. They worked on the overlapping piece directly, consulting the compiler developer as needed and including them in the pull request review. Nobody had to abandon their current work. The dependency got cleared by finding where the two domains met and working from that common ground.

That became a pattern. Whenever two components had a dependency, I looked for the intersection, the code or interface they shared, and found a way to bring both developers to that point together. It was slower than simply assigning tasks. It was faster than waiting for one team to finish before the other could start.

I also made a point to listen. When someone had a concern about working outside their component, I heard them out and worked with them to find a way to contribute that felt manageable. People work better toward a shared goal when they feel heard.

Slowly the pushback faded. The squad started acting like a team. Once that happened, the speed of the work increased noticeably.

You cannot create a real team just by putting people on the same project. You build one by giving people a shared goal, listening to them, and reminding them, over and over, of what they are building together, until working as one team feels more natural than working apart.

Lesson 4: Testing runs alongside development. Not after it.

We wrote the code. We ran tests on our own development machines. Everything passed. We delivered the code to the release branch. I thought we were ready.

What I did not know was that Db2 has a central automated testing system. Think of it as a shared dashboard that runs thousands of tests on every piece of delivered code across all platforms and all deployment setups, automatically and continuously. Anyone can look at it and see which tests are passing and which are failing.

I had never looked at it. I did not fully understand what it showed.

Our QA lead, Darrin Woodard, showed it to us one day. The first time I saw the results for our delivered code, it was bad. Test failures everywhere. Red marks across every platform and every setup we needed to support.

Db2 runs on five different OS platforms: standard Linux, Linux on Power, Linux on Z, AIX, and Windows. Our code had to work correctly on all five. Db2 also runs in different deployment setups. Some installations spread across multiple nodes, others optimized for different types of workloads such as analytical or transactional. Every setup type can expose different bugs.

I had not thought carefully about any of this when planning the project. I underestimated how many test cases we needed to write. I underestimated how long it would take to investigate each failure and fix it.

The Db2 Chief Architect, Mike Springgay, regularly reminded me of this throughout the project. He consistently brought up the importance of estimating the QA effort separately from the development effort. Eventually that lesson landed. I began to treat QA planning as its own work stream, not an afterthought.

After seeing the QA dashboard for the first time, I requested the QA lead to join our regular standup meetings. Twice a week, we opened the dashboard together and looked at every new test failure for the features we had delivered for vector support. Before the meeting ended, every failure had a name next to it: the developer in my squad responsible for investigating and fixing it.

Week by week, the failures went down. More green, less red. Then, before our internal delivery date, everything was green.

Writing code that works on your own development machine is only half the job. The other half is proving it works on every platform, in every deployment setup, in every situation a user might encounter. When it did land, it changed how I planned every subsequent release. QA was no longer something that happened after the work. It was part of the work.

Looking back, Lesson 1 and Lesson 4 are two sides of the same coin. Lesson 1 brought ecosystem awareness to the design phase. Lesson 4 confirmed that the implementation actually honored that ecosystem. One is about seeing clearly before you build. The other is about proving it after you do.

Lesson 5: Writing code made me a better technical leader.

For a long time I believed that leading a technical project meant staying at the level of architecture and direction, and that keeping my distance from the code made me more effective.

It did not. It kept me comfortable.

I was running design meetings. Writing technical proposals. Talking with directors, architects, and customers.

But all of it kept me away from writing code for the actual product features we were shipping. The developers on my team were writing code every day. I was managing, reviewing designs, making decisions, and representing the project to people outside the team. I understood the system at a high level: what it did, how the pieces fit together. I did not understand it at the level where the real problems live: inside the code.

I gave myself a coding assignment. Real product code: the kind that gets reviewed, approved, merged into the main codebase, and shipped to customers.

I started with a relatively simpler function: the vector dimension count function. This calculates the number of dimensions in an input vector. I had built an earlier version of this as a user-defined function, a separate script outside the database. Now I needed to build it properly as a native part of Db2 itself.

I wrote the SQL grammar rules that let users call the function. I wrote the code that actually runs the calculation. I built it the same way every developer on my team builds their features: design it, write it, test it, find the bugs, fix them, write more tests, repeat.

Implementing this first function took me nearly four weeks. I was not working on it full time. I was still doing everything else too. And I was learning this part of the codebase, much of it for the first time. I wrote notes about everything as I progressed: what I tried, what worked, what did not, what I learned. I kept a running log in the Git issue tracker. Screenshots. Code comments. A record of how the feature was built from nothing.

Then I sent the code for review. To a senior developer on my team, someone whose work I had been reviewing and approving for months.

He came back with more than fifteen things that needed to change.

I had thought the code was good. It ran. The tests passed on my machine. But “it runs” and “it is ready for shipping” are very different bars. A developer who has worked in this codebase for years sees things a newer person cannot see. I read every comment. I made every change. I sent it back. More comments. More changes. Several rounds. It was hard, not because the feedback was unfair, but because it was right.

Everything I wrote down during the first feature paid off on the second. The steps were similar. I had a working template for developing new features. Four weeks became two. On the third feature, more complex than the first two, I blocked time on my calendar and focused. I opened a pull request with complete, working code in just two days.

Not because the reviewers stopped looking carefully. Because the code I was writing was getting better.

I was no longer an outsider, a non-developer technical leader telling others what to build. I was one of them. One of the developers in the front line, co-creating the feature code. When I sat in design discussions, my questions were more specific, more tied to the actual code, more connected to how things would actually work. The team’s relationship with me changed too, not because of my job title, but because they had looked at my code line by line, told me what to fix, and watched me fix it the same way I asked them to fix their own work.

The day I stopped thinking of myself only as the leader and started being a developer again. That was the day I became a better leader.

What I Know Now

On any given day across these projects I was the architect, the project manager, the developer, the QA reviewer, the stakeholder manager, and the face of the project to customers and senior leadership. Sometimes all in the same day. That is what technical ownership of an AI project at this level actually looks like. It is not one job. It is many jobs, held together by a clear goal and the discipline to keep moving toward it.

These projects gave me something harder to build than technical knowledge: real confidence, strong working relationships across many teams, and the clarity to take on what comes next. I can now lead AI projects at a scale and complexity I could not have handled two years ago. There is more to build. I plan to build it.

When I walked out of that Architecture Review Board meeting with a long list of gaps, I did not have what I have now. I had nineteen years of experience and a plan that was not ready.

What I have now came from two years of doing the work, getting things wrong, and rebuilding my understanding one lesson at a time. It came from tracking dependencies no one else was tracking. From writing code that got sent back with fifteen things to fix. From saying the same thing to the same team dozens of times until it finally landed.

These are not lessons I read in a book or learned in a course. They are the kind that only come from doing the work, from standing in the middle of something hard and finding your way through it.

I shared them because I wish someone had shared them with me.

Last week, I hit a bug in code I was writing for Db2. The same category of problem that once kept me up until mid morning. I shared the error with an AI coding agent. Root cause identified. Fix applied. Done. Under two minutes.That is not a small improvement. That is a different kind of work.

Last Saturday I shared both of those stories with 35 university students from India, most of them computer science majors, preparing for careers in AI engineering. Then I showed them exactly what sits between those two moments, the tools, the habits, and the mindset shifts that made the difference.

In the session we covered three things. First, how I currently hold a full-time role as AI Architect at IBM Db2 and complete a PhD in AI at the same time, using AI tools to run both tracks in parallel. Second, three prompting habits that immediately change what you get back from any AI tool, including how I use AI when reading something new. Third, two free tools you can open today: Claude and NotebookLM.

The session also opened up questions from the audience that made it richer. Someone in the audience asked how a student studying neuroscience, not computer science, can start using AI. We talked about how AI makes you a sharper learner, not just a faster coder. And we closed with three specific tasks you can act on this weekend.

If you are an engineering student preparing for a career in AI, this session was made for you.

My first day at IBM was in 2005. I was a student intern, excited, prepared technically, and completely lost by the end of the day. Not lost in the code. Lost in everything around it. I joined full time shortly after graduating and the feeling did not go away. If anything it got sharper.

This is the post I wish had existed before I walked through those doors.

Where this came from

Before IBM I had done internships at smaller organizations in New Brunswick, a government office and a few small IT companies. Good experiences. But nothing that prepared me for what IBM actually was. A few friends had interned there before me but nobody told me what it was like on the inside. The only thing I thought I needed to prepare was my technical skills.

I was wrong.

Day one I was lost. Not after a few weeks of settling in. Day one. The scale of it, the complexity of it, the sheer number of things I did not understand hit me immediately. The people, the team dynamics, the processes, the unspoken expectations. Smart, experienced professionals all around me who seemed to know exactly how everything worked. I did not know how to navigate any of it. I did not know what was expected of me beyond the technical work. I did not know how to read the room, how to ask the right questions, or how to exist in that environment without feeling like I did not belong there.

The technology was never the problem. I could handle that.

What I couldn’t handle was everything else.

I struggled to communicate, really communicate. Listening in meetings without tuning out. Writing clearly under pressure. Speaking with any confidence in a room full of people who seemed to know exactly where they stood. Reading the room. Navigating across personalities. Knowing when to push and when to wait.

I didn’t know how to exist in that environment. I didn’t believe I belonged there.

I didn’t know the path forward. I just knew I was lost.

Three people changed that.

My supervisor Tahsin Rouf, my first IBM manager Kent Klaas, and my second manager Ed Van Gennip. Their support in my early years at IBM was not incidental. It was foundational. The trajectory I have been able to build over the past 20 years, everything I have learned, taught, and contributed, sits on the base they helped me build. Tahsin saw something in me at exactly the right moment and offered guidance when I had no idea how to ask for it. Kent supported me consistently and introduced me to mind mapping, a thinking tool I still use today for planning and problem solving. Ed believed in me deeply and gave me the kind of continuous feedback, support, and direction that most early-career people never get. I want to name them here because people like this rarely get named. They should.

And I joined Toastmasters.

That decision opened something I did not expect. Not just better communication. A deep curiosity about the foundational skills nobody had ever taught me. How to learn better. How to think more critically. How to listen deeply, and build the kind of discipline that actually compounds over a career. I started taking IBM’s internal courses. I started building personal frameworks. I started testing things on myself.

Then came the moment that pulled everything together.

About a year or two into my time at IBM, I volunteered to run Think Friday. A weekly internal knowledge-sharing session for our team. I wanted to do it well, not just show up and lead it. So before I ran my first session I read a book called “Not Another Meeting” by Frances A. Micale. That book gave me something specific: a method. How to prepare with a clear outcome in mind, how to structure the discussion, how to manage debate productively, how to take notes that were actually useful, and how to close with clear action items organized by topic and next step.

I ran Think Friday that way for two to three years.

Looking back, those sessions taught me more than I realized at the time. How to bring a group of people toward a common objective. How to hold a productive discussion without letting it drift. How to capture what actually matters and make it useful afterward. How to lead a room, not just occupy it.

That weekly meeting was a turning point. Not because it was high-stakes. Because it was consistent. I got the repetitions. And the confidence that came from those repetitions was real in a way that no single course or credential ever produced.

What was missing

Looking back, I was technically prepared and foundationally underprepared. Both things were true at the same time.

The technical skills got me hired. They kept me moving in the first few weeks. But the moment I needed to work across a team, recommend a clear solution, listen in a meeting without missing half of what was actually being said, or simply believe I had a right to be in the room, the technical skills went quiet. They had nothing to say.

Nobody had ever taught me those things. Not in school. Not in any course. Not in any internship prep session I had ever sat through.

I had to figure them out the hard way, mostly alone, mostly by making mistakes I did not understand until much later. The difference was that I eventually found people who believed in me, and I found a way to practice. Most students never find either.

The framework, step by step

After two decades of learning this myself and then teaching it to students, I have found the foundational skills fall into a small number of areas that show up again and again. These are not soft skills in the vague, resume-filler sense. They are specific and learnable.

1. Communication that actually works in a corporate environment. Not presentation polish. The ability to write a clear email, summarize a meeting, give feedback without creating defensiveness, and ask a question without sounding lost. This also includes thinking tools. Kent Klaas showed me mind mapping early in my career as a way to organize thinking before writing or speaking. It changed how I approach any problem that feels too big to tackle in a straight line. These skills are different from anything you practiced in school and they matter from day one.

2. Learning how to learn on the job. School gives you structured problems with known answers. Work gives you ambiguous problems with moving targets. Building a personal system for learning new things fast, the way I built one by reading a single practical book before running my first meeting, is a skill in itself.

3. Navigating the environment, not just the work. Understanding how decisions get made. Reading team dynamics. Knowing when to speak up and when to listen. This is not political in the negative sense. It is situational awareness, and it determines whether your technical work ever lands.

4. Believing you belong there. This one sounds soft. It is not. Self-doubt is not a feeling to push through. It is a pattern that shapes every decision you make, who you ask for help, whether you volunteer for harder problems, how you handle feedback. The students who address it directly move faster than the ones who try to outwork it.

5. Building discipline that compounds. The habits you build in your first year set the baseline for everything that follows. Not productivity hacks. The small consistent things: how you prepare for meetings, how you follow up, how you manage your own energy, how you stay curious when the work gets repetitive. Running Think Friday every week for two to three years was not glamorous. It was the repetition that built the skill.

The two assets most students skip

Following this kind of framework produces two things that matter more than any project portfolio.

The first is self-awareness about where you actually are. Not where you think you are, not where you hope you are. A clear read on what is working and what is not, grounded in real situations from your actual job.

The second is a personal operating system. A small set of principles and habits you have tested on yourself and trust. Not borrowed from a book. Yours. Mine started with a practical meeting guide and a weekly Friday session. It grew from there.

Why this works

Corporate environments, and AI teams in particular, move fast and reward people who can operate with ambiguity, communicate clearly under pressure, and keep building without waiting to be told what to do next.

The students I have seen thrive earliest are not always the most technically brilliant. They are the ones who understood early that the job was bigger than the code. They put time into the foundational skills deliberately, not because someone told them to, but because they could see the gap and decided to close it.

That gap is real. It is common. And unlike a lot of things in your career, it is entirely within your control to work on.

After 20 years at IBM, mentoring and helping more than 100 students across undergrad, masters, and PhD programs, I still see the same gap show up in the same way. Not because students are underprepared. Because nobody told them it existed.

That is exactly why I am sharing this with you now.

Students reach out to me regularly. They share their portfolios, ask for feedback, want to know why they are not hearing back from applications.

I look at their work. And almost every time, I see the same pattern.

A RAG pipeline. An AI agent. A classification model. Clean code, decent README, maybe a demo video. And then it stops there.

Building something real takes effort, and that effort is worth acknowledging. But there is a next step most students have not been shown, and that gap is what is holding them back.

That is what this post is about.

The question that separates candidates

When I review a portfolio as an AI architect, I am not only asking can you build this? I am also asking do you know if what you built is any good?

Those are not the same question. And in my experience, most portfolios only answer the first one.

Think about what the job actually involves. You will not be handed tutorials to replicate. You will be handed a system that partially works and asked to improve it. Or you will build something from scratch and be responsible for demonstrating it meets a quality standard before it ships.

The question your portfolio needs to answer is: can you take something that partially works and make it measurably better?

What this looks like in practice

Let us use RAG as the example. It is the most common AI portfolio project I see right now, and a good one to learn from.

You built a RAG pipeline. It retrieves documents and generates answers. You tested it with a few queries and the responses looked reasonable.

But do you actually know how well it is working?

There is an open source evaluation framework called RAGAS, published at EACL 2024, built specifically to evaluate RAG systems. It gives you measurable scores on dimensions that matter in production:

Faithfulness measures whether the model’s answer is grounded in the retrieved documents, or whether it is generating claims the documents never supported. This is how you detect and quantify hallucinations.

Answer Relevancy measures whether the response actually addresses what the user asked.

Context Precision and Recall measures whether your retriever is surfacing the right document chunks in the first place.

You run your system through RAGAS and get a faithfulness score of 0.61. That means roughly 4 in 10 claims your system generates are not supported by the retrieved documents. For most production use cases, that is not acceptable.

[What most portfolios show vs. what hiring managers are also looking for]

So you investigate. You find your chunk size is too large and the retriever is returning loosely related content alongside relevant content. You reduce the chunk size. Faithfulness moves to 0.74. You introduce a re-ranker to improve which chunks reach the model. Now it is 0.84.

You document every step. The baseline score, what you changed, why you made that change, and what improved as a result.

That documented process, not just the final system, is what demonstrates engineering thinking.

This applies beyond RAG

If you are building a classification model, it is worth asking: do you know your precision, recall, and F1 score, and do you understand which one matters most for your specific use case?

If it is a regression model, can you explain what your RMSE is telling you about your system’s real world behaviour, beyond the formula itself?

These are not advanced topics. They are foundational to working with models professionally. And developing fluency with them early will serve you well throughout your career.

There is a principle worth holding onto: you cannot confidently claim something works if you have no definition of what working means.

I came across this idea early in my career through a book called Exploring Requirements: Quality Before Design by Donald Gause and Gerald Weinberg. I read it years ago and it shaped how I think about building systems. Their core argument is straightforward: define quality before you design, not after. The book was written in 1989 about software engineering. The principle applies just as directly to AI systems today.

What production ready actually means

"Production ready" gets mentioned constantly in job descriptions. Here is what it actually means in practice.

Reliability means your system handles unexpected conditions without breaking. What happens when the retriever returns nothing useful? When an API call times out? A production system fails gracefully and recovers.

Observability means you can see what is happening inside your system after deployment. Are you logging inputs, outputs, and quality scores in a way that lets you diagnose problems when they occur?

Cost means your system stays economically viable at scale. How many LLM calls does it make per query? Does that remain reasonable if usage doubles or triples?

Edge cases are the inputs your system handles poorly. Every system has them. The difference between a student project and a production ready system is often just whether those edge cases were found and documented before users encountered them.

You do not need to have solved all of this as a student. No one expects that. But if your README shows you have thought about these dimensions, identified the limitations of your own system, and have a reasoned view on what you would address first, that communicates something important about how you think. It signals that you are ready to operate on a real engineering team.

That kind of thinking is what I have seen make a genuine difference when students move from portfolio to interview to role.

One thing to do this week

Take any project in your portfolio. Add one section to the README called Evaluation and Known Limitations. Answer these three questions:

## Evaluation and Known Limitations

**1. What metrics did you use, and what did the numbers show?**
e.g. "Faithfulness: 0.84 after two iterations. Context Recall: 0.71, identified as the next area to improve."

**2. Where does the system underperform, and under what conditions?**
e.g. "Retrieval degrades on queries shorter than 5 words. Fails gracefully but returns low-confidence responses."

**3. What would you address first to improve it, and why?**
e.g. "Improve Context Recall by experimenting with hybrid retrieval. Currently retriever is keyword-only."

The engineers who stand out are not always the ones who built the most projects. They are the ones who can examine a system honestly, articulate where it falls short, and think clearly about how to make it better.

That is the skill worth developing. And it is one you can start demonstrating today.

If you are preparing for an AI role and want feedback on your portfolio, feel free to reach out or leave a comment below. And if you have worked with evaluation frameworks or metrics in your own projects, I would be glad to hear what you found useful.

If you are an engineering student preparing for an AI internship and not sure what to build for your portfolio, this post is for you.

I want to share an approach that turned a single open-source tutorial into three publishable assets over a couple of weeks. It is not a shortcut. It is a deliberate process, and it taught me more than starting from scratch would have.

Where this came from

About a year ago, after a decade of working in AI and ML, I decided to learn agentic RAG properly. Not just read about it. Build something real with it.

I found a well-structured tutorial in the LangChain repo. It builds an agentic RAG system that loads web content, chunks it, indexes it for semantic search, and uses a LangGraph agent to decide when to retrieve versus when to answer directly. Clean, well-organized, a good foundation.

My first instinct was not to modify it. It was to understand the tutorial code completely.

So I ran it as written. Worked through every component. Traced each step in the workflow. Asked myself why each design choice was made, not just what it did. Why that retrieval strategy. How the agent decides when retrieved documents are relevant enough to generate an answer, and when to rewrite the query and try again. Where the boundaries between components are drawn and why.

That process took time. But it was the only reason I could later see where the gaps were. You cannot see what is missing in something you have only skimmed.

What the tutorial was missing

Once I had that foundation, the gaps became clear.

The tutorial used an in-memory vector store, fine for a prototype but not for production. It loaded content from public blog posts, useful for a demo but not connected to any real enterprise use case. There was no evaluation of retrieval quality, no baseline numbers to measure anything against, and no way to run any of it outside a Jupyter notebook.

Those were my extension targets.

I replaced the in-memory vector store with IBM Db2 as the vector database. Swapped the public blog content with a technical article I had written myself, so I could evaluate the outputs against answers I already knew. Added local embedding generation using a Granite model so the entire pipeline could run offline. Then decomposed the notebook into three production microservices: a data ingestion API, a search and generation API, and a unified gateway that runs both.

Having built RAG pipelines with my team, I can tell you that decomposition step is where most of the real engineering decisions surface. Error handling, input validation, service boundaries, modularity. None of that appears in a notebook.

My full repo is here: shaikhq/agentic-rag-db2

The framework, step by step

Here is the process in the order I followed it. It applies to any AI tutorial you want to extend, not just RAG.

1. Run the original first, without touching it. Understand every component and every design choice before you form an opinion about what to change. This is the step many students skip. They skim the tutorial, get the general idea, and immediately start modifying things. That is why their extensions feel shallow. Deep understanding is what lets you see the gaps.

2. Record baseline numbers before changing anything. Retrieval accuracy, latency, response quality, whatever is measurable for the use case. You need a starting point. Without a baseline, any improvement you make later is invisible. This is also one of the most common gaps in tutorials themselves, and closing it alone is a meaningful contribution.

3. Swap the data source with something real to you. Replace the tutorial’s sample data with your own content, your own domain, something you can evaluate honestly. I used a technical article I had written. The pipeline’s responses immediately became more meaningful to read because I already knew what a correct answer looked like.

4. Close one gap with a targeted improvement. Better retrieval logic, a smarter chunking strategy, an evaluation step the original skipped, a production-grade vector store. Pick one improvement at a time, implement it, and measure the difference against your baseline from step 2. That comparison is the substance of your portfolio, not the code alone.

5. Convert the notebook to APIs. This is the step that separates a prototype from something deployable. I split the notebook code into a data ingestion API and a search and generation API, then combined them behind a gateway. It forces decisions about error handling, input validation, and service boundaries that notebook code never surfaces. Most students never get there, which is exactly why it is worth doing.

6. Write the README while the work is fresh. Not a usage guide. A record of what you found in the original tutorial, what you changed, and what the numbers showed before and after. Include a workflow diagram. This is the document an interviewer will ask you to walk them through. Writing it also forces you to articulate what you actually did and why, which is harder than it sounds.

The two assets that most students skip

Once the code is done, produce two more assets from the same work before you close the project. These are the ones that actually get you noticed.

Write a post about what you extended and what you measured. Not a tutorial explaining the technology. Your own account of the extensions you made, what surprised you, and what the numbers showed. A hiring manager can tell the difference between a post written from experience and one assembled from documentation. This kind of writing is harder to produce from AI alone, which is exactly what makes it credible.

Record a short video walking through the repo. Many students applying for internships will not do this. I published two walkthroughs covering the above sample work: Deploying Agentic RAG to Production, Part 1: FastAPI Data Ingestion and Deploying Agentic RAG to Production, Part 2.

A hiring manager who watches three minutes of you explaining your own work has a clearer picture of how you think than one who reads a README alone.

One tutorial. A couple of weeks of deliberate work. Three assets: an extended repo, a written reflection, and a video walkthrough. Each one builds on the same foundation and tells a consistent story about how you think and work.

Why this works

The reason this approach produces better portfolio projects than building from scratch is not that it is easier. It is that it mirrors how engineering work actually happens. In any real job, you will inherit code, understand it, identify what is missing, and improve it. The tutorial gives you something to push against. The gaps give you something real to solve. The baseline numbers give you a way to show that the solution worked.

A portfolio project that shows you can do that is more convincing to a hiring manager than one that shows you can follow a course and build what the instructor built.

If you try this approach, I would be curious to hear which step felt hardest, finding a tutorial worth extending, deciding which gap to close first, or getting the post and video out after the code was done. Those are usually where people get stuck, and they are worth talking through.

When you build an application that uses embedding models - whether for semantic search, retrieval-augmented generation, or any other vector-based workflow - measuring performance is easy to put off. You get the feature working, the embeddings look right, and you move on.

Then someone asks: “How long will it take to vectorize our 50 million documents?”

If you have not done the performance work by that point, the question becomes a problem without a quick answer. I have been there. This post is about doing that work before you need the answer, building a repeatable performance testing approach rather than figuring it out under pressure.

One clarification on scope: this guide focuses specifically on batch embedding generation - sending multiple text inputs to an embedding API in a single call and measuring the throughput. If you are calling an embedding API one input at a time, some of this still applies, but the biggest gains described here come from batching. That context matters as you read.

I will walk through the performance testing approach I have been developing and refining through real experiments against cloud embedding APIs and a self-hosted vLLM endpoint. The work ranged from small single-request tests all the way to running embedding generation over datasets at million-row scale.

Why Embedding APIs Warrant Their Own Performance Testing Approach

Embedding APIs are used differently from text generation APIs, and that difference shapes how you should think about performance.

With a generation API, you typically process one request at a time - a user sends a prompt, the model responds. With an embedding API, you are almost always working in bulk. A product catalog with 10 million descriptions needs all 10 million vectorized. A document corpus gets re-indexed every night. The volume is large, and it often runs as a background job where throughput matters more than single-request response time.

This bulk processing nature means that small configuration choices - like batch size - can have large effects on the total time and cost to complete a job. For example, running with a poorly chosen batch size can cut your throughput significantly compared to an optimized one, and that difference compounds across every run, every day, at every scale. It is not the only thing that matters, but it is one of the most controllable levers you have.

The performance testing approach covered here focuses on four questions:

1. What limits does your embedding API put on each call?

2. At what batch size does throughput peak for your model and data?

3. How does throughput drop when you go past that batch size?

4. What breaks when you run the same job at production scale?

Step 1: Know Your API’s Hard Limits Before You Test Anything

Before running a single test, check the documentation for your embedding provider and answer one question: how does this API limit what you can send in a single call?

There are two common patterns:

Input count limits. Some providers cap the number of text inputs you can include in one API call. Once you exceed that number, the request is rejected. Always verify the current limit in the provider’s documentation, as these can change.

Token budget limits. Other providers do not set an input count limit directly, but cap the total number of tokens across all inputs in a single call. For example, if the limit is 300,000 tokens and your texts average 450 tokens each, that budget translates to roughly 666 inputs per call. The limit is not stated as an input count, but it effectively becomes one based on your data.

These two approaches require different batching logic. With an input count limit, you can set a fixed maximum batch size. With a token budget, your effective batch size depends on the actual token length of your inputs, so you need to calculate it dynamically for each batch rather than hardcoding a number. Getting this wrong - for example, assuming all inputs are short when some are long - can cause unexpected failures partway through a large job.

All of the APIs I tested were REST APIs. The limits below reflect what the REST API accepts per call. These figures come from publicly available documentation at the time of writing. Always check the provider's current docs before using them, as limits can change.

• WatsonX AI — Input count limit. Observed up to 1,000 inputs per call in my own testing. IBM Cloud API Docs

• OpenAI — Token budget. 300,000 tokens total across all inputs per call. OpenAI API Reference

• Azure OpenAI — Input count limit. Up to 2,048 inputs per array per call. Azure OpenAI docs

• AWS Bedrock (Titan) — Single input only. The InvokeModel REST API accepts one input string per call — batch arrays are not supported. AWS docs

• vLLM (self-hosted) — Depends on deployment. The server operator controls limits at startup via --max-num-batched-tokens and --max-num-seqs. In my own testing I was able to send up to 10,000 inputs in a single call. vLLM Engine Arguments

A note on counting tokens before you send them. If your provider uses a token budget, you need a way to estimate token counts on your side before constructing each batch. Calling the API itself to count tokens is expensive and defeats the purpose. A practical alternative is to use an open-source tokenizer locally. For OpenAI-compatible APIs, the tiktoken Python library can count tokens offline at no API cost. For other models, the transformers library from Hugging Face provides tokenizers that run locally. Neither requires a network call, and both run quickly enough to be part of your batching logic at scale.

Step 2: Measure the Right Things - Break Down Where Time Goes

Before you start adjusting batch sizes, set up your measurement code to separate request time into three distinct parts. This is easy to skip, but it is where the most useful diagnostic information comes from.

Payload preparation is the time your code spends building the request - formatting your list of input texts into the JSON structure the API expects. For small batches, this is negligible. For large batches with many long texts, it can add up and become worth optimizing separately.

API wait is the time from when your request leaves your machine to when the response starts coming back. This includes network travel time in both directions, any time the request spends waiting in a queue on the provider’s side, and the time the model actually takes to compute the embeddings. This phase typically takes the most time, and it grows with batch size - though not in a straight line, as you will see in Step 3.

Response parsing is the time your code spends reading the response and extracting the embedding vectors. At larger batch sizes, responses get large. A batch of 1,000 embeddings at 768 dimensions, stored as 32-bit floats, contains about 3 MB of numerical data to read and process. Measuring this separately helps confirm whether response handling is ever a bottleneck.

When you break latency into these three parts, you can see where to focus. If API wait is the dominant cost, then batching strategy is your main lever. If payload prep is surprisingly slow, look at how you are constructing requests. If response parsing takes longer than expected, look at how you are reading and storing results.

Step 3: Find Where Throughput Peaks - and Where It Falls Off

This is the core of the performance testing work. The goal is to find the batch size where you get the most inputs processed per second, and to understand what happens on both sides of that point.

When you plot throughput against batch size, you get a curve with a predictable shape: it rises quickly at first, peaks somewhere in the middle, and then gradually declines. Three things drive that shape:

Why throughput rises as batch size increases. Every API call has a fixed cost you pay regardless of how many inputs are in it: the time to establish the connection, send the request, and receive the response. At batch size 1, that fixed cost is most of your total call time. At batch size 10, you pay the same fixed cost but process 10 inputs instead of 1, so your throughput is roughly 10 times higher. As batch size grows, that fixed cost gets spread across more inputs and throughput climbs.

Why throughput eventually stops climbing. Once the batch is large enough, the fixed overhead is fully spread out and the model’s compute time per input becomes the main cost. Throughput levels off. This is the plateau.

Why throughput can drop with very large batches. Very large batches introduce new costs: larger payloads take longer to send over the network, the server may need to split your batch into smaller internal chunks to process it, and memory pressure on the server side increases response time. The result is that throughput can actually decrease beyond a certain point, even though more data is being sent.

A few patterns worth noting in this kind of result:

Batch sizes 1 and 10 can have nearly the same total call time (both around 0.4 seconds in this example), yet batch 10 delivers about 10 times the throughput. This is the fixed-cost effect at its clearest - you are doing 10 times the work for no additional waiting time.

The provider’s maximum allowed batch size is often not the most efficient one. In this sample, batch 1,000 - the largest - produces the lowest throughput. The API accepting a batch size does not mean that size is the best choice.

The peak is typically a range, not a single number. In the sample above, batch sizes from about 200 to 300 all deliver throughput within 5-10% of the peak. This is useful in practice: if your batches are not always exactly the same size, anything in that range will perform similarly well.

How to run this test. Pick a fixed dataset of text inputs with a consistent average length (matching your real workload as closely as possible). Start with batch size 1 and work up through increasing sizes - for example, 1, 10, 50, 100, 200, 250, 300, 400, 500, up to your provider’s maximum. For each batch size, run at least three separate calls and average the results to reduce the effect of network variation. Include one unmeasured warm-up call at the start of each batch size to avoid counting any startup effects in your measurements.

Step 4: Test Each Model Separately

The batch size that gives the best throughput is not the same for every model. Smaller models compute embeddings faster per input, so they can handle larger batches before hitting the plateau. Larger models take more compute time per input, so the plateau arrives at a lower batch size, and the performance drop from oversized batches tends to be steeper.

When I ran the same batch size test across three models of different sizes - small (around 30M parameters), medium (around 125M), and large (around 500M or more) - the curve shape was similar for all three, but the peak location and the absolute throughput values were different for each. Larger models peaked at a lower batch size and at lower throughput overall.

The takeaway is simple: if you are working with more than one embedding model, run the batch size test for each one independently. Do not carry over the optimal batch size from a small model and assume it applies to a larger one.

To keep the comparison clean, hold everything else constant - the same input texts, the same infrastructure, the same measurement approach. Change only the model. This way, any difference in results can be attributed to the model rather than noise from other variables.

Step 5: Test With Different Input Lengths

Most performance tests use a fixed text length, but real data rarely has perfectly uniform length. The token length of your inputs affects throughput in ways that are worth understanding before you run your tests, and it affects providers differently depending on whether they use input count limits or token budget limits.

A practical way to test this is to prepare separate datasets at a few different average token lengths - for example, a short-text set (around 50 tokens per input, like product titles or search queries), a medium-text set (around 150 to 250 tokens, like short paragraphs), and a long-text set (around 400 to 500 tokens, like full document sections). Run the same batch size test on each.

With short texts, you can fit more inputs within a token budget per call, but the fixed API overhead becomes a larger proportion of your total time per input. With long texts, your effective batch size shrinks if you are working within a token budget. The optimal batch size by input count may be quite different across these three workloads.

One specific edge case worth testing: what happens when an input exceeds the model’s maximum token length? Some APIs truncate the input silently and return an embedding anyway. Others return an error. Others may return something that looks like a valid embedding but reflects only part of the input. The behavior varies by provider and model, and it is worth verifying directly for your setup rather than assuming. If your real data has any inputs that might be unusually long, this test can prevent silent failures in production.

Step 6: Test at Real Scale

Small tests tell you how the API behaves call by call. They do not tell you what happens when you run a job across millions of inputs over several hours. Scale introduces a different set of problems, and many of them only become visible once you are running at that size.

I will not share specific numbers from my own large-scale experiments here, as that work was done in a professional context. But the problems I ran into are common enough to be worth sharing.

The bottleneck often is not the API. When running a long embedding job end to end, the slowest part is not always the embedding API call. Time spent reading inputs from the source system, writing results back to storage, and handling coordination between parallel workers can all add up. Measure the full pipeline, not just the API call times, to find where the real time goes.

Write volume adds up at scale. Embedding vectors are not small. At 768 dimensions stored as 32-bit floats, each embedding is exactly 3 KB. Writing 1 million embeddings means writing just under 3 GB of data. If you are writing to a database or storage system that has transaction or write limits, those limits can become a bottleneck or cause failures mid-job. Check the limits of your write destination before starting a large run, not after hitting an error.

Concurrent API calls and token rate limits interact. Running multiple parallel workers to call the embedding API at the same time can speed things up significantly. But if the provider has a rate limit measured in tokens per minute, the token usage from all your parallel workers adds up together. It is easy to hit a rate limit mid-job when running in parallel even if no single worker would hit it on its own. Test your parallel configuration against the provider’s rate limits before committing to it at full scale.

Connection pool limits surface late. If your parallel workers each hold a database connection for writing results, the total number of concurrent connections can exceed your connection pool limit. This type of failure often does not show up until you are partway through a large job. Test with enough scale to expose it before running the full job.

Build resume capability before you need it. A job processing a million inputs that fails at 80% completion should be able to pick up where it left off, not start over. This is straightforward to build before you start a large run and very difficult to add after you have already failed and need to rerun. Track which inputs have been successfully processed and skip them on restart.

Step 7: Run Multiple Iterations and Look at Consistency

A single measurement tells you what happened once. API performance varies based on server load on the provider’s side, network conditions between you and the provider, and the time of day the call is made. A result from one run may not reflect what you will see reliably in practice.

Running each batch size configuration at least three times and looking at the spread of results gives a more reliable picture. If the three results are close together, you can be more confident in the average. If one run is significantly different from the others, that variance is itself useful information - it tells you that the performance at that configuration is less predictable, which matters for planning.

In my own tests, mid-range batch sizes tended to show more variation between runs than very small or very large ones. The practical lesson: when planning how much headroom to build into a performance estimate for a production job, do not base it only on the best case from a small number of test runs. Look at the range of results and plan for something closer to the typical case, not the best one.

Putting It Together: A Testing Checklist

Here is a checklist that pulls together the steps above. Use it as a starting point and adjust it to your situation.

Before you start testing:

• Find the current documented limits for your embedding API (input count or token budget).

• Determine whether you need static batch sizes (input count limit) or dynamic batch sizing (token budget).

• Set up local token counting so you can estimate batch sizes without API calls (e.g., tiktoken for OpenAI-compatible APIs).

• Instrument your code to separately measure payload preparation time, API wait time, and response parsing time.

• Prepare representative test datasets at short, medium, and long average token lengths.

Running the batch size test:

• Test batch sizes from 1 up to the provider’s maximum, using regular intervals.

• Run at least three iterations for each batch size.

• Use one unmeasured warm-up call at the start of each batch size series.

• Change only one variable at a time across test runs.

• Repeat the test for each embedding model you plan to use.

Edge cases to verify:

• What happens when an input exceeds the model’s maximum token length?

• What happens when you approach or exceed the provider’s rate limit?

• How does performance vary when input lengths within a single batch are uneven?

Scaling to production volume:

• Test the full pipeline end to end, not just the API calls

• Check write volume and storage limits for your target system at scale

• Test parallel worker configuration against the provider’s rate limits

• Verify connection pool limits under your expected concurrency level

• Build and test checkpoint and resume functionality before running at full scale

• Run at an intermediate scale (for example, 1% or 10% of your target volume) before committing to the full job

Closing Thought

The biggest performance gain in embedding batch workloads does not require new hardware, a faster model, or a different provider. It comes from batching correctly.

Two API calls that take the same time can have very different throughput — one sending a single input, the other sending a well-sized batch. The larger batch processes many more inputs for the same cost. At the scale embedding pipelines typically run, that difference translates directly into hours of processing time and real infrastructure cost.

The numbers will look different for every model, workload, and provider. The method for finding them does not change.

If you have run similar experiments or found different patterns, I would be curious to hear about it in the comments.

All experiments were run from my personal machine calling external APIs. Any numbers shown are illustrative samples only - they demonstrate the shape of results, not actual measurements from any specific provider. These are personal observations and do not represent my employer or any official benchmark.

2025 has been my most active year as a technical speaker in my 20-year IBM career. I traveled to seven cities across four continents, delivered 16 conference sessions totaling 25.5 hours, and reached over 1,000 in-person attendees - plus many more through webinars and internal sessions.

By the Numbers

6 conferences | 7 cities | 4 continents | 25.5 hours | 1,000+ in-person attendees

My Framework for Preparing Technical Talks

Here’s the structure I use:

One Thread, One Theme. Every slide supports a single theme. No distractions.

Own Your Content. I create all my slides from scratch. No borrowed decks. This means I know exactly what each slide says and can talk about it naturally.

Revise Until It’s Clean. For each slide, I ask: What is the key message here? I keep editing until every extra word and image is gone.

AI as My Design Partner. Last few years and much part of this year, I used ChatGPT for brainstorming and designing slide visuals. Now, my favorite tool for planning conference slides is Claude. For image editing, I use Canva.

Start with the Title. I open every talk by explaining each word in the title and telling the audience what they will learn. This sets clear expectations from the start.

Live Demos Only. No pre-recorded demos. I build and run demos myself, live on stage. This engages the audience. Also, when I built a demo, I did’t need to memorize - I just explained what I built.

Keep the Audience Engaged. I ask questions, add light humor, and repeat key points throughout the talk.

What Changed

At the start of this year, conference talks felt uncomfortable. With each session, it got easier. Now I look forward to larger audiences - the bigger, the better. I enjoy making eye contact, reading the room, answering questions mid-talk. The best moment is when I explain something complex and see it click. That’s the joy.

There is not one algorithm that’s going to solve all your problems. Think of it like wanting to unscrew something. You have a toolbox with different sized screwdriver bits, and you have to find the right one for the screw in front of you. Machine learning works the same way. You have to see what kind of data you are dealing with and the learning task at hand, and then find the right tool, the right algorithm, the right AI technique.

Some data sits neatly in rows and columns. Other data is a mess of text, images, and files. Each needs different tools to unlock its value.

In this article, I’ll walk you through the main types of machine learning models and learning tasks. This is not meant to be exhaustive. I want to cover the more representative models so you have a solid foundation.

First Things First: Structured vs. Unstructured Data

Before we dive into the model types, we need to understand the two broad categories of data we deal with: structured data and unstructured data.

Structured data is the kind of data stored in relational schemas, it’s not free form. It comes with a structured format. You have a table with a list of columns, and each column has some attribute values. Maybe an employee table with employee name, join date, department, and salary. Everything is fixed with columns and rows. You can make a lot of assumptions about the data structure, and you have much more predictable characteristics because it’s a fixed structure.

I find this analogy useful: think of going to a library where all the books are organized on shelves. If you’re looking for a specific book, you ask the librarian, and they just look up and tell you exactly which shelf you can find it on. Everything is very neatly organized.

Unstructured data, on the other hand, is like a very messy situation, maybe people who just returned all the books have left them haphazardly in a pile. You don’t know where you can find anything. Unstructured data includes things like free-form text, images, audio, and video.

How Machine Learning Models Learn

Before we look at specific model types, you need to understand the two fundamental ways models learn from data: supervised and unsupervised learning.

Supervised Learning

In supervised learning, you provide the model with examples that include the correct answers. You need two types of content in your data.

Features: A set of attributes that describe something. For example, if you want to predict customer churn, you might collect which products customers are using, how many licenses they have, how many times they called support.

Labels/Targets: The outcome, what actually happened. When the time came for renewal, did this customer renew or not?

You collect input and output combinations for many examples, then give this to a learning algorithm. The algorithm figures out how to map inputs to outputs.

A good way to think about it: remember equations like y = constant + a₁x₁ + a₂x₂?

The machine learning algorithm learns this kind of mapping from your data.

Once trained, for future events where you only know the inputs, you plug them into the model and it predicts the output. This process is called scoring or inferencing.

Self-Supervised / Unsupervised Learning

In self-supervised learning, you don’t provide the correct answers. You give the model data and let it find patterns or groupings on its own. You only have features, no labels.

Clustering is a common self-supervised technique. Algorithms like k-means clustering group similar data points together without being told what the groups should be. For example, you might have customer data and want to discover natural segments. The algorithm analyzes the features and groups similar customers together. Marketing teams often use this to identify customer personas they didn’t know existed.

Predictive Models

Predictive models use supervised learning to give you predictions about future outcomes. Classification, regression, and time series forecasting all require labeled training data.

Classification

For structured data, one kind of prediction you can make is the category or class of something. A classic and familiar example: you have an email, and you want to classify it into one of two categories, valid email so it can get to your inbox, or spam. That’s what Google has done with their spam folder and inbox. They have a spam classifier that looks at each email and a set of characteristics (the features), and based on this, classifies each email as spam or valid (the label).

Another example is fraud detection. You go to your grocery store and swipe your credit card to pay. Behind the scene, it goes to the credit card authorization system, which quickly checks whether this transaction is valid or fraudulent.

Why is this called classification? Because this model is making some conclusion about the input, a structured row of data, and saying that this record belongs to one of a finite set of classes. It’s either valid or fraudulent. You can have multiple classes, but it’s always a finite set of possible outcomes. That’s called classification.

Regression

The other kind of predictive model you can build with structured data is a regression model. A common example is predicting house prices. You want to predict what a house will sell for based on how many bedrooms it has, the square footage, and other features.

You can train a machine learning model with houses that were sold in a neighborhood, their features and the price at which each was sold. You give all these to a learning algorithm, and then you have a learned regression model. Next time a house that has yet to be sold comes along, you just collect its description and attributes, give it to this model, and the model gives you a numerical value.

Unlike classification, where you predict from a finite set of values, here your model is predicting a number that could vary widely. The number is not coming from a fixed set of values. This kind of model is called a regression model.

Time Series Forecasting

Another type of predictive model is time series forecasting. You might want to predict stock prices based on events and factors, or weather forecasting. There’s seasonality, there are variables and factors you’re collecting over time, what happened, what are some of the things that occurred. Based on this, you’re predicting future values. Time plays a role in these models, that’s what makes them different from standard regression.

Predictive Models for Unstructured Data

For unstructured data, similar predictive models exist, but they work with different types of input.

Image classification: You might want to classify pictures, give a machine learning model images and have it learn to predict whether a picture is of a cat, a dog, or some other animal.

Object detection: A self-driving car is constantly taking pictures of what’s in front of it and around it, identifying objects and their locations. It’s not sufficient to just know that there’s an object in front of the car, it also needs to know the location of that object.

Text classification and sentiment analysis: Maybe you have product reviews and you want to predict if the customer is happy or not. You can analyze their text comment and classify whether they’re a happy customer or not.

Generative Models

This is the category that’s been very popular in recent years, the large language models (LLMs), such as ChatGPT, and all these generative AI models. This kind of model is not just predicting based on an input. It takes a prompt and then creates content for you.

Text generation: You can give a prompt and all of a sudden get a long, AI-generated text. It can generate articles, very human-like responses, and more.

Language translation: You speak in one language, and it gets translated into another. Services like Google Translate use neural machine translation models for this.

Code generation: You can generate code using AI. IBM Project Bob, Claude, Gemini, ChatGPT, there’s a lot of code generation happening with these tools.

Image generation: These models can also generate images based on text prompts.

What Makes Large Language Models “Large”?

Two things make them large: massive training data from sources like Wikipedia, books, and web pages, and billions of parameters (the internal variables the model learns during training). GPT-3, for example, has 175 billion parameters. This scale allows LLMs to capture intricate patterns in language.

Embedding Models

Embedding models are a different kind of AI model. They don’t produce something immediately consumable like a prediction or generated text. Instead, they give you an intermediate representation called a vector, which is essentially a list of numbers that captures the meaning or characteristics of your data.

For example, each of us has certain background attributes, the field we work in, years in our career, our university education. If we want to learn which of us are closer to each other based on our technical or academic background, we could use some kind of embedding model to take a text description of who we are, and based on this, it gives a vector representation. Those of us who are similar are expected to have similar vector representations.

Similarly, you can have a list of documents or articles, vectorize them, and see which articles are similar to each other. Maybe a subset are food blogs and others are technical blogs, you expect articles in the food blogs to have similar vector representations. You can do the same with images.

With multimodal embedding, your content is a mix of text and image, and you can vectorize it to do semantic search, not just plain text-based matching, but vector-based semantic search.

Choosing the Right Model for Your Problem

When you’re approaching a problem, ask yourself three questions:

What kind of data do you have? Structured data (tables with rows and columns) works well with traditional machine learning. Unstructured data (text, images, audio, video) typically requires deep learning approaches.

Do you have labeled data? If you have historical data with known outcomes, you can use supervised learning (classification, regression). If not, consider unsupervised approaches like clustering or embedding models.

What kind of output do you need? This is often the deciding factor. Use the table below as a quick reference.

The temptation today is to reach for LLMs for everything. But LLMs are just one drawer in the toolbox. For structured, tabular data, traditional machine learning algorithms are often faster, simpler, and more accurate. The skill isn’t knowing the most powerful tool. It’s knowing which tool fits the problem in front of you.

Many companies don’t have a job title for the person who owns AI features end-to-end. They have ML engineers, data scientists, product managers. But the role that bridges all of them? That’s often unnamed.

AI Product Architect is a term I’m using to describe this role. It’s not a standard industry title. It’s how I make sense of a set of responsibilities that don’t fit neatly into existing job categories.

A software architect designs systems. An AI Product Architect, as I define it, does the same, but for AI features inside products. Same technical leadership. Different technical domain.

If you’re a senior engineer or software architect with years of experience shipping software, you don’t start over to move into this role. You add AI to what you already know: systems thinking, product delivery, technical leadership. The core skills transfer.

In my role as AI Architect at IBM Db2, I technically lead the development of AI features for the product. Db2 has architects for different areas such as compiler, security, and data movement. My responsibility is planning, designing, and implementing AI features in Db2 products.

That’s what an AI Product Architect does, at least in my experience: technically owns AI features end-to-end within a larger product.

What the Role Looks Like in Practice

In 2025, I was technically responsible for three major AI initiatives in Db2. I owned and architected the end-to-end implementation of each, from concept to release: planning, designing, implementing, testing, releasing, promoting, and adopting.

The three projects:

1. Native vector data type: introducing vector storage and similarity search functions inside Db2

2. Tooling integration: extending existing Db2 utilities to recognize and work with the new vector type

3. LLM API integration: connecting external language model APIs to Db2, making them callable via SQL functions

These weren’t research experiments. Each one shipped in a product release to enterprise customers running real workloads.

Each project followed the same pattern. Let me walk you through what that looks like.

Phase 1: Proposal and Validation

Every initiative starts with identifying a market opportunity. A problem worth solving. That problem may translate into multiple functions, new capabilities, new use cases for customers. But it starts with one question: what’s worth building?

Market Research

Analyst reports give you the landscape: Gartner Magic Quadrant, Forrester Wave, industry blogs from major conferences. Competitor announcements show you where the market is moving.

But reports tell you what’s popular, not what’s valuable. That requires talking to customers.

When we proposed vector support for Db2, I studied how AI practitioners were using vector types and similarity search: what problems these capabilities solved, what use cases they enabled.

Customer Validation

I spoke with customer representatives, internal data scientists, and machine learning practitioners. Many of them work directly with real customers. I also reached out to actual customers already building these kinds of use cases.

The goal isn’t to confirm your idea is good. It’s to find out where it’s wrong. What are customers already doing? What’s broken? Would they pay for this?

This back-and-forth sharpens the idea. You validate assumptions, crystallize the concept into specific problem categories, and identify where real demand exists.

Quick Prototyping

Before presenting to executives, when possible, I build low-fidelity prototypes using existing features. Not performant. Not production-ready. But tangible.

For the vector proposal, I implemented vector storage and similarity search using existing Db2 data types and user-defined functions. It was slow, but it demonstrated the experience and capability we were proposing.

A working prototype changes the conversation. Stakeholders stop debating whether it’s possible and start debating whether it’s worth it.

Building the Proposal

The initial proposal is still high level: a suggested set of new features, the problems they solve, evidence of customer interest, and the industry outlook.

I never go straight to executives. First, I validate with peer architects, including some more senior than me, and make them co-authors on the proposal. We go back and forth, refine the thinking, pressure-test the idea. Then we present to executives together.

Phase 2: Solution Externals

Once the proposal has stakeholder support, the next step is writing solution externals.

In software engineering terms, this is a software requirements specification. But it reads like a user manual, written before any code exists. You define how real users will interact with the proposed features: the interface, the syntax, the experience.

The goal is precision without code. By the time engineering starts, everyone (architects, developers, stakeholders) should have the same understanding of what the feature will do.

For each feature, I document:

• What it does: a clear description of the function or capability

• How it looks: syntax, parameters, inputs, outputs

• Processing: what the function does with those inputs to produce the output

• Restrictions: what this feature will not cover in this phase

• User scenarios: sample use cases with realistic values

For example, when I wrote the solution external for the vector functions in Db2, I documented each function. For vector distance calculations, that meant defining parameters, expected inputs, outputs, processing logic, and restrictions. Everything was documented textually and visually, without writing a single line of implementation code. The visual above shows what this looks like in practice.

This solution external becomes the foundation for the actual user documentation later.

This document saves months of rework. Ambiguity in requirements becomes bugs in production. The solution external eliminates ambiguity before a single line of code is written.

At this point, the feature exists on paper. Everyone agrees on what we’re building.

Now comes the hard part: making it real.

Next post: Design and Implementation.

In 2014, I was at IBM Littleton Lab in Massachusetts for a training on Watson. I was learning how to build a Q&A system for IBM customer support. I had spent nine years building Java applications. I knew nothing about AI beyond a graduate course on data mining and machine learning. Watching Watson generate answers, I was amazed. The technology was just starting out, but I could see where things were heading.

That week was my first exposure to AI. Two years later, I made the full transition.

My fear was stepping into territory with no clear path to production. But I had always tried new technology early, even in my Java career. I knew that if I stayed comfortable, I would be fine for a few years. Eventually, my growth would stop. I was not willing to let that happen.

The Java Years

For 11 years, my world was enterprise Java. I built web applications and web services, deployed them to production, and troubleshot server failures. I developed a search engine from scratch. I set up a wiki for a large organization. My days were writing code, testing, deploying, and fixing what broke.

I was not just an individual contributor. I led a team of Java developers, system administrators, and DBAs. I was hands-on with code and responsible for technical leadership and architecture. Doing both shaped how I think about building systems and leading people.

It was good work. I was good at it. Management was happy, and I received top ratings.

But at some point, the growth slowed down. The work became routine: add features to the front end, crawl more data sources into the search engine, repeat. I was not feeling challenged. I was not innovating. And I knew it.

The Turning Point

By 2016, “Data Science” and “Cognitive Computing” were impossible to ignore. AI had carried a negative reputation from its earlier failures, but deep learning was changing that. Watson had won Jeopardy. Kaggle competitions were producing impressive results. Neural networks were going through a transformation with modern architectures.

I did not understand most of it. But I could see AI was beginning to matter. It was the earliest stage of adoption. Most companies were hesitant to explore AI. Success stories were rare.

Then, at the end of September 2016, before a three-week vacation, my manager Chad Marston pinged me. He said, “Shaikh, we want to begin working in Cognitive Computing. I want you to lead that team.”

I would be building a team of 8 to 10 developers, and none of us had any background in AI. We all came from Java and web development.

I went on vacation, but I kept thinking about it. When I came back, I was already exploring what I could learn and where I could get hands-on with this technology.

Chad sent me to the IBM Watson Developer Conference in San Francisco later that year. I attended hands-on labs where I played with Watson APIs. I saw demos. I listened to talks from Ginni Rometty, IBM’s CEO, and other experts. I had seen Watson before, but this was different. I saw more of what AI could do in practice. I came back with confidence and a clearer picture of what we could build.

The Hard Part

Leading 8 to 10 people while learning myself was not easy. Online learning platforms had plenty of courses, still overwhelming. Where do you start? What is worth your time? What will matter when you try to apply it?

The first thing I did was set up regular check-ins with my mentors at IBM: Jean-Francois Puget, Jorge Castañón, and Seth Dobrin. I got their guidance, filtered courses based on their recommendations, put together a learning path, and shared it with the team.

Even with structure, the learning curve was steep. The concept I struggled with longest was evaluation metrics. Accuracy was easy. Precision and recall were harder, and harder still to explain to management. The harder problem was mapping model metrics to business metrics. A model can perform well on test data, but how do you know it is helping the business?

I have to admit: our first project failed. We were asked to predict customer satisfaction scores from support interactions. The model was accurate in training and testing. We deployed it. But we had no way to monitor business success. We failed quickly and learned from it.

Many questions from team members, I did not know the answers. I did not make things up. I admitted I did not know, then went and learned and came back. That honesty became part of how we operated.

The learning was hard for the whole team. We set up a weekly session where we watched videos together and discussed what we learned. Often, one team member would pre-read the material and present it. This kept us progressing together.

What Helped Us Learn

I tried many courses across Coursera, edX, Udemy, and DataCamp. A few stood out:

Andrew Ng’s Machine Learning course gave me a foundation in core algorithms: decision trees, linear and logistic regression, support vector machines, and neural networks. It was taught in Octave, not Python, but the concepts were what mattered.

Python for Data Science from DataCamp gave me fluency in Python once I had the conceptual foundation.

Two books were essential: An Introduction to Statistical Learning for statistical foundations, and Python Machine Learning by Sebastian Raschka for bridging theory to implementation.

But the most important learning did not come from courses. It came from applying concepts to real projects, getting stuck, and consulting mentors. That cycle of building, struggling, and asking for help was the core of my growth.

Where I Am Now

My AI journey started in 2014 with that Watson project. In fall 2016, I made the full transition, initially working with Watson’s NLP capabilities: sentiment analysis, question answering, and similar APIs. I became the lead machine learning engineer for IBM Analytics and held that role for three years. In 2019, I moved into my current role: AI Architect for IBM Db2.

My work today goes beyond building models. It has two dimensions. The first is improving database operations using AI, for example, applying machine learning to improve memory estimation for queries. The second is bringing AI infrastructure into the database itself, so that database administrators can run AI workloads directly within Db2.

Building AI into a database is different from building standalone models. The model must be small, fast, and light on resources. No human monitors these models, so they need self-feedback mechanisms. These were constraints I never faced before.

One project shows what my role looks like now: bringing vector support to Db2. At the beginning of 2025, Db2 had no vector capabilities. I owned this project end to end: planning, designing, architecting, leading the team, and contributing to development. We implemented vector type support, vector similarity search, and LLM API integration callable via SQL. We released it, then I created educational materials, gave talks, wrote blog posts, and promoted the work.

That cycle, from zero to shipped product to education to promotion, is what makes this role rewarding. I am challenged every day. I am not just building models. I am thinking about how AI fits into an enterprise database platform and how to make it useful for production workloads. I am more satisfied now than I ever was in my Java years.

What I Would Tell Someone Making This Transition

Find a mentor early. If you want to become an AI architect, find someone in that role. Get their guidance on what to learn and what to skip. This will save you months.

Your experience is not a liability. I did not abandon my 11 years of Java. I built on it. Production systems, working with teams, managing projects: all of that transferred. The AI skills were an addition, not a replacement.

Your learning approach should change over time. Early on, I took courses end to end. That was useful for building foundations. But after the fundamentals, I changed my approach. Now I start with the project, then learn only what I need to apply. In 2021, I took an entire specialization on cloud deployment of AI. I never applied most of it, and those skills are gone. Project-driven learning is more effective once you have the foundations.

Model accuracy is not the end goal. I used to think a highly accurate model was all that mattered. I was wrong. What matters is identifying the right business metric and tying it to your model’s performance. A model can look great in testing but have no impact if you have not made that connection.

User adoption is the part nobody warns you about. Even after you get the model and metrics right, users fall back to old routines. Deploying AI is not just a technical problem. It requires rolling it out slowly and getting buy-in. Skip this, and your model sits unused.

Questions to Ask Yourself

1. Who do you know who is already an AI architect? Write down three names. This week, reach out to one of them with a single question about how they got there. If you do not have someone in mind and think I could help, reach out to me.

2. What skills from your current role would transfer to an AI architect role? List three.

3. What have you learned in the past two years that you never applied? Going forward, learn less that way.

4. What problem in your current work could benefit from AI? Name the problem and the outcome you want.

5. If you became an AI architect today, what is the first project you would want to work on?

Your Turn

If you are a senior engineer feeling that growth has stopped, the path I took is available to you. You do not need to go back to school. You do not need to start over as a junior. You need a structured path, the right guidance, and willingness to be uncomfortable for a while.

If this post was useful, follow me. I will publish more on lessons from my transition, with steps you can apply.

If you are considering a move to an AI architect role, I would like to hear from you. What are your struggles? What is holding you back? Share in the comments. Your questions will shape what I write next.

One thing troubled me these past days: I couldn’t get llama.cpp to install on macOS. I had set it up successfully on Linux using Python 3.13. The same steps failed on my Mac. They kept failing

llama.cpp is a tool written in C++ that allows you to run large language models on consumer hardware.. It supports various model types and provides GPU acceleration on Apple Silicon through Metal. For developers and IT professionals, this means you can run AI models locally without relying on cloud services or paying for API calls.

I searched everywhere, ChatGPT, Claude, and Google. Nothing worked. I went back to the official llama.cpp documentation. There I found the answer: llama.cpp doesn’t work well with Python 3.13 on macOS.

After more testing, I arrived at the right steps for macOS. I run an M2 Mac, but this should work on other Apple Silicon chips, thoughI have not tested those variants.

I wanted to verify everything worked. I downloaded two models from Hugging Face in GGUF format: a text generation model and a text embedding model. I created two minimalistic Python scripts, one for text generation, another for embedding generation.

Both scripts ran fine. The embedding script generated embeddings for input text. The text generation script responded correctly to prompts.

To be certain, I documented all steps. Then I created a fresh directory and repeated everything. It worked again.

Here are my steps, the models I used, and the test scripts. I hope this helps those who want to set up language models locally on macOS for prototyping or development.

Complete Setup Process

Step 1: Environment Setup

# Install Python 3.12 using uv (if not already installed)
uv python install 3.12

# Check Python version
python3.12 --version

# Create virtual environment with Python 3.12
uv venv --python=$(which python3.12)

# Upgrade pip
uv pip install --upgrade pip

# Install required packages with Metal support
uv pip install \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal \
  llama-cpp-python \
  “langchain>=0.2” “langchain-community>=0.2”

Step 2: Download Models

# Download Nomic embedding model
wget https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q8_0.gguf

# Download Qwen text generation model
wget -O qwen2.5-3b-instruct-q4_k_m.gguf \
  https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf

Step 3: Create Test Scripts

Text Generation (langchain_qwen_generation.py):

from langchain_community.llms import LlamaCpp

llm = LlamaCpp(
    model_path=”qwen2.5-3b-instruct-q4_k_m.gguf”,
    n_ctx=2048,
    temperature=0.7,
    max_tokens=100,
    verbose=False
)

response = llm.invoke(”What is the capital of France?”)
print(response)

Embedding Generation (langchain_nomic_embeddings.py):

from langchain_community.embeddings import LlamaCppEmbeddings

embeddings = LlamaCppEmbeddings(
    model_path=”nomic-embed-text-v1.5.Q8_0.gguf”,
    n_ctx=512,
    verbose=False
)

query = embeddings.embed_query(”search_query: What is AI?”)
print(f”Dimensions: {len(query)}”)
print(f”First 5: {query[:5]}”)

Step 4: Run Test Scripts

# Test text generation
uv run langchain_qwen_generation.py

# Test embeddings
uv run langchain_nomic_embeddings.py

What I Learned

For now, use Python 3.12, not 3.13, on macOS. The installation command includes Metal support for Apple Silicon. Both test scripts verify that everything works.

This setup lets you run language models locally without cloud costs. Good for prototyping and development.

This part focuses on building the Search API for Agentic RAG using FastAPI, designed to orchestrate an AI agent powered by LangGraph. The agent coordinates multiple tools to retrieve the best possible answers, forming the core of an agentic RAG pipeline. I also unified this Search API with my previous Ingestion API under a single FastAPI gateway - creating a clean, production-ready interface.

You’ll find all the code and setup instructions in the GitHub repo:
🔗 https://github.com/shaikhq/agentic-rag-db2

To go with the 2nd installment of the tutorial series, I’ve recorded two short videos:

1. Implementation Walkthrough – A guided tour of the repo and how the APIs are structured

2. Live Demo – A hands-on demonstration of calling the APIs using simple curl commands

Both videos are included below.

This builds directly on the first part of the series, where I focused on turning the document ingestion logic into a standalone API. If you missed that, you can find it in the GitHub repo as well.

First Video: Deploy Agentic RAG to Production, Part 2 - Build the Search API End Point - Code Walkthrough

Second Video: Agentic RAG Search and Gateway APIs Demo

In the following video, I run a demo of the search and the gateway APIs using simple curl commands across three scenarios:

1. Ingesting new knowledge into the vector store using the ingestion API

2. Querying the Search API to retrieve the best possible answer through the LangGraph agent

3. Clearing the vector store using the cleanup API, giving the system a clean slate

These APIs form the foundation for building an Agentic RAG application on top of the ingestion and retrieval pipeline.

If you have missed the first installment of my deploying Agentic RAG to production tutorial series, here’s the link:

Deploying an AI Agent in Production: FastAPI Data Ingestion

You've built AI agent prototypes. They work in notebooks. But how do you deploy them in production?

I faced this exact challenge after building several AI Agent prototypes over the past few months.

The First Critical Step

Convert workflows from notebooks into APIs.

The above video shows you how I tackled this challenge with a real agentic RAG system built using the Db2 LangChain connector.

What You'll See in Action:

• GitHub repo setup on macOS

• FastAPI deployment process

• Live command line testing

• Notebook logic converted to production endpoint

Get the Code: Agentic RAG Document Ingestion Code + macOS setup

The Technical Architecture

The agentic RAG workflow has two components:

Document Ingestion Pipeline

• Downloads documents from URLs

• Extracts and cleans content

• Vectorizes the data

• Inserts vectors into a Db2 table

Agent RAG Workflow (coming next)

• Orchestrates the agent workflow using LangGraph

What's Coming Next

This is the first in my "AI Agents in Production" series. Each video will build toward a complete production deployment of this example AI Agent.

When I began building applications with large language models, LangChain was my starting point. It offered everything I needed to assemble a working system: tools for loading and splitting documents, embedding models for vector search, retrievers to find relevant content, and prompt templates to shape responses.

Using these tools, I built a basic retrieval-augmented generation (RAG) workflow. The steps were straightforward and sequential: receive a question, embed it, search a vector store, fill the context into a prompt, and pass it to an LLM. This kind of linear pipeline—with fixed steps and no branching—matched LangChain's design. For that use case, it worked well.

But as I explored more complex workflows, LangChain began to show its limits. I wanted to evaluate the quality of the retrieved content before sending it to the LLM. If the results were weak, I wanted to revise the question or try a different search method. In some cases, I wanted to retry a step or include a human in the loop. These were no longer simple chains of tasks. They were decisions, conditions, and loops—features that don't come naturally in a linear framework.

At that point, I needed more than a toolkit. I needed a way to orchestrate logic, not just execute steps. That's when I turned to LangGraph.

LangChain and LangGraph: Complementary Roles

LangChain and LangGraph serve different purposes, but they work together.

LangChain is both a tooling ecosystem and a workflow library. It provides integrations with language models, vector stores, embedding tools, document loaders, prompt templates, and more. It also supports basic orchestration, letting you compose linear workflows and agent loops using its Chain, Runnable, and AgentExecutor abstractions.

LangGraph builds on top of this foundation. It does not replace LangChain—it reuses and extends it. All the tools and model integrations offered by LangChain remain available. What LangGraph adds is a graph-based execution model, inspired by state machines, which enables complex control flows. You can define branches, retry conditions, state persistence, and error handling—all within a structured, declarative graph.

In LangChain alone, you can implement similar behaviors, but you must manage logic manually using Python. As your workflow grows, this can become unwieldy. LangGraph helps you organize that complexity using nodes (which represent steps) and edges (which represent conditions and transitions). This makes your workflow easier to reason about and maintain.

Linear vs. Adaptive Workflows

Here's a basic example to show the difference:

LangChain (linear):

Question → Embed → Search → Prompt → LLM → Answer

LangGraph (adaptive):

In one of my recent workflows, I needed to verify whether the retrieved content was relevant. If it wasn't, I wanted to reformulate the query and try again, possibly with a different strategy. With LangGraph, I could express this flow directly within the graph. The conditions, retries, and alternate paths were part of the structure—not hidden inside glue code.

Choosing the Right Tool

🤔 Key Decision: Do you need branching, retries, or conditional logic?

Use LangChain’s workflow when:

• Your workflow is linear and predictable.

• You don't need branching logic, retries, or persistent state.

• You're building a simple prototype or want to test tools quickly.

Use LangGraph when:

• Your workflow depends on decisions, loops, or adaptive behavior.

• You need to retry steps, rewrite inputs, or fall back to alternate strategies.

• You want structure, clarity, and modularity as your logic becomes more complex.

It's important to understand that LangGraph can be used with or without LangChain. While LangGraph can be used standalone, it also integrates seamlessly with any LangChain components. When used together, LangChain gives you the components; LangGraph helps you wire them together in richer ways.

Final Thoughts

LangChain helped me get started. It let me focus on building working pipelines quickly and gave me confidence in integrating LLMs with useful tools. But as my workflows grew more intelligent—needing logic, state, and decisions—I needed more structure. That's where LangGraph became essential.

You don't need LangGraph for every project. But when your workflow is more than just a sequence of steps—when it needs to think, adapt, and respond—LangGraph gives you the framework to do it cleanly and with control.

• Structures your workflow as a graph of nodes and transitions for greater transparency and control

Although LangGraph can run standalone, most developers use it alongside LangChain to take advantage of its rich ecosystem. LangGraph doesn't replace LangChain—it extends it, especially when workflows demand smarter behavior.

Two simple questions are driving this:

➊ What are the main pieces needed to build a production-grade MCP server?
➋ What open-source tools can I use locally (no cloud setup) to build each piece in a simple and testable way?

What I’ve figured out so far:

Below are the main parts of the system and the tools I started exploring:

① Accept and route incoming HTTP requests

Use a lightweight web framework like 𝐅𝐚𝐬𝐭𝐀𝐏𝐈 (used internally by 𝐅𝐚𝐬𝐭𝐌𝐂𝐏) to expose the MCP server over Streamable HTTP or SSE.

② Parse and validate the JSON request payload

Use 𝐏𝐲𝐝𝐚𝐧𝐭𝐢𝐜 (via FastMCP) to convert JSON input into clean, type-checked Python objects.

③ Verify the identity of the caller

Use 𝐏𝐲𝐉𝐖𝐓 to decode and validate JSON Web Tokens (JWTs) that prove who the client is.

④ Check what the caller is allowed to do

Use 𝐅𝐚𝐬𝐭𝐀𝐏𝐈'𝐬 Depends system to apply role-based access control for each request.

⑤ Enforce request limits per user

Use 𝐬𝐥𝐨𝐰𝐚𝐩𝐢 to add rate-limiting middleware so users can’t overload the server.

⑥ Validate tool input before execution

Let 𝐏𝐲𝐝𝐚𝐧𝐭𝐢𝐜 (automatically used by FastMCP) validate input arguments before calling the tool.

⑦ Look up the requested tool by name

Use FastMCP’s @mcp.tool() decorator to register callable tools.
(You can also use @mcp.resource() for shared resources and @mcp.prom() for prompt templates.)

⑧ Run the tool logic

FastMCP handles calling the right tool function, passing input, and handling any exceptions.

⑨ Format the tool output into a standard response

Use 𝐉𝐒𝐎𝐍-𝐑𝐏𝐂, which FastMCP handles under the hood, to return either the result or a structured error.

⑩ Send the response back to the client

Use built-in HTTP support or Server-Sent Events (SSE) in FastAPI/FastMCP to return the response.

I haven’t put everything together yet, but I’ve started.
I’ll keep sharing what I learn as I go—hoping to eventually have a full, working MCP server built from scratch using these tools.

This is the first in a short series of posts. In this one, I focus on setting up the MCP Gateway. I have not yet connected the gateway to any MCP servers—that will be covered in a future post.

Background

What Is the Model Context Protocol?

The Model Context Protocol (MCP) defines a standard way for AI agents to invoke external tools, APIs, and prompt templates. Instead of implementing custom integration logic for each action, an AI agent can issue a generic request to an MCP server, which exposes a registry of available capabilities. The protocol separates the agent’s reasoning from the mechanics of how tools are discovered, invoked, and secured. This design enables modular, scalable integration between agents and real-world systems.

Why MCP Gateway?

Building an MCP server is the easier part. Integrating it cleanly with AI agents? Not so much.

It starts simple—an AI agent calling one MCP server for a database, then another for search, maybe a third for file storage.

But as the number of MCP servers and tools grows, things get messy:

• Servers use different logins and authentication flows

• There are too many endpoints to manage and secure

• AI-generated requests lack guardrails and observability

• Integration logic clutters the AI agent’s codebase

While building MCP servers for my own projects, I started to see how the above issues could pile up quickly. That’s when I began exploring ContextForge MCP Gateway, an open-source middleware from IBM.

The MCP Gateway sits between your AI agent (MCP client) and your tool-specific MCP servers. It provides:

• Centralized login across servers

• A single endpoint to access multiple tools

• Built-in observability (logs and a web UI)

macOS Setup Instructions

The following steps describe how I set up and ran the MCP Gateway locally on macOS (M2 Apple Chip).

Prerequisites

Before beginning, I ensured that I had the following installed:

• macOS with Python 3.12 (the MCP Gateway requires Python 3.10 or higher)

Step 1: Install uv

uv is a fast, Rust-based tool for managing Python packages and environments.

To install it, run:

curl -LsSf https://astral.sh/uv/install.sh | sh

To make sure it's on your shell's PATH, add the following line to your shell configuration (e.g., .zshrc or .bashrc), or run it directly in your terminal:

export PATH="$HOME/.local/bin:$PATH"

To verify the installation, run:

which uv
uv --version

Step 2: Create a Project and Install the Gateway

To create a project directory and set up a local Python environment, run:

mkdir mcpgateway
cd mcpgateway
uv venv --python $(which python3.12)
source .venv/bin/activate

To install the MCP Gateway inside the environment, run:

uv pip install mcp-contextforge-gateway

Step 3: Start the MCP Gateway Server

To set the password for the default admin user, run:

export BASIC_AUTH_PASSWORD=pass

To set the JWT secret key for signing tokens, run:

export JWT_SECRET_KEY=my-test-key

To start the server, run:

mcpgateway --host 0.0.0.0 --port 4444 &

This command starts the server in the background, listening on all network interfaces at port 4444.

To access the admin UI, open a browser and go to:

http://0.0.0.0:4444/admin

To log in, use the following credentials:

Username: admin
Password: pass

This loads the administrative interface for the gateway.

Step 4: Generate a Bearer Token

To authenticate future requests, create a bearer token:

export MCPGATEWAY_BEARER_TOKEN=$(python3 -m mcpgateway.utils.create_jwt_token \
  --username admin --exp 10080 --secret my-test-key)

This token is valid for seven days and is used to authorize calls to the gateway.

Step 5: Test the Gateway

To confirm that the server was running and accepting authenticated requests, use the following:

curl -s -H "Authorization: Bearer $MCPGATEWAY_BEARER_TOKEN" \
  http://127.0.0.1:4444/version | jq

This returned version metadata in JSON format.

This concludes the local macOS setup. The steps closely follow the instructions published in the MCP Gateway GitHub repository, with slight expansions and clarifications added for readability.

Next Steps

Now that the MCP Gateway is set up and running locally, the next step is to start using it by registering one or more MCP servers behind it.

The official ContextForge MCP Gateway repository includes a dedicated section that shows how to do this in a local setup:

End‑to‑end demo (register a local MCP server)

In a future post, I plan to follow that path and walk through how to connect MCP servers to the gateway and route tool requests through it.

• Sandy Besson – Applied AI Research Engineer at IBM Research

• Chris Chileles – CEO at Zig

• Boaz Ashkenazi – CEO at Augmented AI Labs

The discussion focused on the practical challenges of building AI agents for enterprise use—not theoretical architectures or future visions, but the realities of deployment, safety, measurement, and cost. Topics included everything from managing expectations and ROI, to protocol choices like MCP, to what they’d build differently now.

Below is a distilled summary of the key insights they shared.

1. Demos Don’t Reflect Reality

• Clients love polished demos—but want strict guardrails in production, especially for customer-facing agents.

• Leaders often expect fast ROI. But value takes time, preparation, and honest risk evaluation.

• Systems that work in testing often become brittle or expensive at scale.

2. Success Requires Trade-Offs

• You can’t maximize speed, accuracy, and cost all at once.

• One panelist mentioned of the Pareto frontier framework to guide teams through meaningful trade-offs.

• In legal, finance, and other high-risk areas, human oversight is essential. A simple rule:
  If you wouldn’t trust an intern to do it alone, don’t let the AI.

3. Measuring Quality Isn’t Simple

• Standard metrics (accuracy, F1 score) help, but don’t tell the full story.

• Human feedback, logging, and telemetry are key to assessing response quality.

• Business teams should define what “good” means—not just engineers.

4. Handle Data with Care

• Sensitive data is driving many teams back to on-prem infrastructure.

• Limit what the model sees and what it can output. Apply the least-access principle.

• In many cases, small, focused ML models outperform general-purpose LLMs—especially when reliability and privacy matter.

5. Design for Safety and Modularity

• Guardrails are needed both before and after the model runs.

• Don’t treat models as all-knowing. Consider them as intern—with boundaries.

• Use synthetic data to model edge cases without exposing sensitive records.

6. Stay Flexible with Tools and Protocols

• New agent communication protocols (like MCP, A2A, ACP) are emerging, but no dominant standard exists yet.

• Choose simple, open tools with strong community backing.

• Avoid overbuilding. Many orchestration tools built last year are already obsolete.

7. SaaS Isn’t Dead—But It’s Changing

• SaaS still matters, especially for small to mid-sized teams.

• But usage-based AI workloads don’t fit neatly into per-user pricing models.

• In regulated industries, adoption is slower than it seems. Risk, compliance, and culture slow change.

8. Build for What’s Next

• Many custom tools built today may become unnecessary tomorrow as models improve.

• Before building, ask: “Can we wait six months?”

• Favor modularity and future adaptability over quick fixes or heavy infrastructure.

Closing Thought

The discussion remained focused on practical issues—what it takes to build and deploy AI agents responsibly in complex enterprise settings. For those working on agent frameworks, enterprise AI platforms, or real-world deployments, it offers several useful takeaways.

Today, I watched Andrej Karpathy’s keynote, “Software is Changing (Again),” at the AI Startup School hosted by a16z.
Karpathy is a leading AI researcher and educator, known for his work at OpenAI, Tesla, and Stanford.

His talk wasn’t a technical deep dive. It offered a broad, thoughtful look at how software development is shifting in the age of large language models (LLMs). For me, it helped clarify where things are heading—and what skills and habits we need to adapt.

What follows is my summary of the talk, including a few interpretations and examples I added to better understand the talk.

Who Was This Talk For?

Karpathy was speaking to startup founders, engineers, and students preparing to build AI-first tools. But his message applies to anyone involved in software development today.

The Three Eras of Software

Karpathy describes the evolution of software in three distinct paradigms. Knowing the strengths and tradeoffs of each helps us decide which one to apply—and when.

These paradigms didn’t replace each other overnight. Each emerged in response to new capabilities, and all three are still relevant. The first has shaped several decades of software history; the second has taken hold more recently as machine learning matured; and the third is unfolding now with the rise of large language models.

Software 1.0 — Human-Written Code

Developers write logic line by line in a programming language. The system does exactly what it's told.

Software 2.0 — Learned Programs

Instead of writing program logic directly, developers train models—often neural networks—on data. The model becomes the program.
Even so, this approach still requires coding: for building the model, preparing data, training, and evaluation.

Software 3.0 — Prompt-Based Development

Now, with LLMs, we describe what we want in natural language. The model generates the code or behavior.
This shifts software design away from implementation details and toward goal-oriented interaction.

Karpathy’s message is that we need fluency across all three paradigms. Each remains useful, depending on context.

From Code to Prompts: What Changes (and What Doesn’t)

While Software 3.0 offers dramatic productivity gains, Karpathy stresses the importance of discipline and control.

He recommends working with LLMs one step at a time:

• Generate a small block of code with LLM

• Review and verify it manually

• Then proceed to the next step

This loop—generate, verify, repeat—ensures that humans remain in charge. LLMs assist, but they don’t replace judgment or accountability.

Why Real-World Integration Still Matters

Karpathy emphasized that generating code is often the easiest part. But turning that code into a working, reliable product is much harder.

For example:

• You might prompt an LLM to generate a prototype iOS app in a few hours.

• But integrating that app—sign-up flows, payment systems, backend connections—can take days or weeks.

Prompting gets you started. Building something production-ready still takes engineering effort and care.

Partial Autonomy: A Better Path for Agents

Karpathy also touched on AI agents—autonomous systems built on LLMs. His advice? Don’t aim for full autonomy just yet.

Instead, build partial-autonomous agents:

• Keep humans in the loop

• Design systems for transparency and override

• Apply guardrails in higher-risk scenarios

The result is safer and more usable systems. Autonomy should be earned incrementally, not assumed by default.

What LLMs Are—and Aren’t—Good At

Strengths:

• Vast encyclopedic knowledge across domains

• Handling large context windows in a single session

Limitations:

• Hallucinating plausible but incorrect answers

• Lacking internal truth-checking mechanisms

• Forgetting everything between sessions unless augmented

These limitations are not bugs. They’re foundational constraints—and they shape how LLMs should be used responsibly.

Final Takeaway

Karpathy’s message was clear:

“Software is changing again. Developers must evolve with it.”

That evolution means:

• Learning all three paradigms—Software 1.0, 2.0, and 3.0

• Using LLMs with precision and skepticism

• Staying grounded in what it takes to build real, integrated systems

This isn’t just a tooling shift. It’s a mindset shift. And it’s already underway.

Recently, I built my first Agentic RAG workflow using LangGraph. I shared the code and setup steps in this GitHub repo.

Today, I'm publishing a full video walkthrough of this Agentic RAG implementation.

This video is intended for those who are already familiar with basic RAG workflows but haven’t yet explored Agentic RAG. My goal is to help you get started with your first Agentic RAG project.

Agentic RAG vs. Vanilla RAG: Key Differences

Before diving into the code, I want to explain some foundational concepts that helped me understand Agentic RAG and how it differs from vanilla RAG workflows.

Vanilla RAG Workflow:

• A sequence of steps.

• User submits a query to an LLM app (e.g., chatbot).

• App retrieves relevant chunks from a vector store.

• App sends an augmented prompt (original query + retrieved context) to the LLM.

• LLM returns a response.

Limitations:

• The process is static.

• If the original query is poorly worded, or if retrieval results are weak, there's no recovery mechanism.

• The LLM is only involved at the end.

Agentic RAG Workflow:

• Involves the LLM throughout the workflow.

• After the initial retrieval, the LLM assesses the quality of the retrieved chunks.

• If chunks are not highly relevant to the query, the LLM rewrites the query and retries retrieval.

• This loop continues until relevant content is found.

• Then, the LLM generates the final answer using the relevant chunks

This dynamic, decision-making process powered by the LLM is what makes the workflow “agentic.”

My Implementation: Improvements and Simplifications

I started with an LangChain tutorial on Agentic RAG with LangGraph:

Agentic RAG with LangGraph: https://langchain-ai.github.io/langgraph/tutorials/rag/langgraph_agentic_rag/

Then, I made a few key changes in the original tutorial:

1. Local Embeddings with Llama.cpp
   I replaced OpenAI’s embedding generation with a local embedding model using Llama.cpp and the Granite text embedding model. You can switch to any Hugging Face-compatible model in Llama.cpp format.

2. Cleaner Web Document Retrieval
   The original tutorial used WebBaseLoader, but I found it noisy (extra newlines, artifacts). I switched to a different library that gives cleaner web content. While I didn’t explore all WebBaseLoader config options, switching to another library was quicker and more effective for me.

3. Sentence-Aware Chunking
   The original used character-based chunking, which can split in the middle of a sentence. I replaced this with sentence-based chunking for better coherence.

4. LLM Optimization
   I reused the LLM instance throughout the workflow instead of re-instantiating it multiple times.

Repo and Setup Instructions

You’ll find the complete project and the macOS setup instructions here:
🔗 GitHub repo: https://github.com/shaikhq/agentic-rag-basic

Workflow Overview

The RAG pipeline is implemented in a notebook: agent.ipynb.

I use my blog post titled “A Step-by-Step Guide to Building a Linear Regression Model in IBM Db2” as the sample knowledge base for the RAG pipeline.

Chunking

• Chunks: 200 words max, 50-word overlap.

• Rationale: Overlap helps maintain context coherence when LLM stitches chunks together.

Vector Store and Embedding

• Local Granite embedding model via Llama.cpp.

• In-memory vector store created.

• A retriever tool is built on top of this store.

LLM for Answer Generation

• I used Watsonx.ai’s Mistral-large model.

• Can be replaced with another provider or a local model.

Pipeline Functions and Prompts

I defined several key components:

1. Query or Respond Decision
   Determines whether to retrieve more content, rewrite the query, or end the workflow.

2. Relevance Grading
   Prompt instructs LLM to score retrieved chunks as relevant or not, using a yes/no format.

3. Query Rewriting
   Reformulates the user query to improve retrieval quality.

4. Answer Generation
   Prompt guides the LLM to answer concisely (max 3 sentences). If it doesn’t know the answer, it says so.

These are the building blocks of the Agentic RAG pipeline.

Visualizing the Workflow

I created a LangGraph visual representation of the workflow.

Steps in the graph:

1. generate_query_or_respond

2. retrieve

3. rewrite_question

4. generate_answer

Conditional logic controls transitions between these steps based on relevance grading.

Testing the RAG pipeline: Questions and Responses

Query 1:
“How to calculate summary statistics in Db2?”
→ Workflow runs → Final answer generated by LLM.

Query 2:
“How to build a linear regression model with IBM Db2?”
→ Workflow runs → Final answer generated by LLM.

Conclusion

That concludes my walkthrough and demo.

Thanks for reading/watching—and have fun building your own Agentic RAG pipelines using LangGraph!