I came across a Vercel blog post on how they implemented their homegrown software factory for the AI SDK. It clarified how people have started putting the software factory into practice. What I found wasn’t a way to remove the human. It was a way to stop routing every single change through one human’s attention.

Let me put together a definition of software factory in my own words.

Software Factory is a multi-agent system deployed across the software engineering process, from the very first step (creating issues, bug reports, or feature requests) all the way to shipping the code. In other words, the requirements and fixes that go into the next iteration of software. And it applies to new software too, not just existing code.

In a traditional software engineering process, the release of a new feature or a fix goes through many steps, such as:

• If it’s a new feature request, putting together a design.

• Initial implementation, which needs to be evaluated, improved, and validated against the initial requirement.

• Quality checking (QA).

• Performance testing, and if it meets the bar, it goes out to production.

• Feature documentation on how the feature works.

Every one of these steps used to take human efforts. That’s the part that changes.

All these steps, before agentic coding became a thing, were done by human engineers. Now that we have scaled up the production of software artifacts with AI coding agents, we have more code and more requirements coming up at a speed that is hard for human engineers to handle end to end by hand.

The idea is that for each software engineering step, whether building a new feature or fixing a bug, there is a specialized agent. At Vercel, that means a separate agent for each step in the pipeline, not one agent doing everything. Each agent gets its own prompts, context, and evals. It is refined for its task and measured on its performance.

Then the human engineers assume the role of reviewers, not doers, looking at what the army of agents is producing. They are the final gatekeepers to ensure that what gets delivered meets the quality bar.

This is how Vercel implemented it for the AI SDK. The AI SDK is a big open-source project, over 20 million npm downloads a week, and the repo has more than 26,000 stars. Keeping a codebase like that healthy means watching four things move at the same time: the model providers, the UI frameworks, the sandboxes the agents run in, and the harnesses. After a few years of growth, the repo was getting 100+ new issues every month. When Anthropic’s Opus 4.6 model came out, the pull requests hit an inflection point. By late June, they had over 1,000 open issues and almost 800 pull requests sitting there.

And this is the part that matters: you cannot fix that backlog by hiring more maintainers or asking the ones you have to work harder. Because generating code is cheap now, the pile only gets bigger. So instead of trying to scale themselves, they built a software factory. Four weeks in, it authors between 25 and 35% of the PRs they merge and closes 70 to 80% of their issues.

They started with an agent that classifies incoming issues as a bug, a feature request, or a documentation update. Once that agent achieved a high level of accuracy, they moved on to the next step.

For example, if it’s a bug report, another agent sets up an isolated environment and works on reproducing the bug. Another agent fixes it, and another reviews the PR. There are also agents for feature analysis, feature implementation, documentation updates, and backports to older versions. Each agent hands off to the next, and some of these runs go in parallel. Each agent is evaluated and improved in isolation.

It’s not one monolithic agent doing everything from beginning to end. If some agent is not reaching the bar, it can be isolated and improved on its own through better prompts, better context, or new eval cases.

The repo is open source, so anyone can open an issue or leave a comment, and the factory has to treat all of it as untrusted input.

How I would put this into practice

I would give an agent a responsibility only once it achieves a certain, measured level of performance on that task. So I went looking for how others are thinking about this. I found two posts that say the same thing. Port suggests having a human approve every step at first, and then taking the human out of the steps you trust, one at a time. Augment Code found that their teams needed time to look at what an agent was producing and correct it before trusting it with longer tasks. Both are saying the same thing I am saying. You do not hand over everything at once. You hand over one step at a time.

I think of Software Factory like teamwork, where a team lead distributes tasks among the developers in the team. If some of the developers are falling a little behind, the team lead spends more time with them. In the same way, if some tasks are not done well by an agent, the human can spend more time reviewing that part. The team lead is responsible for overseeing the work produced by team members in isolation and in coordination.

It’s a human in the loop, not a human removed from the loop.

So here is where I landed. A software factory is not a way to take the human out of software engineering. It is a way to move the human to the end of the line, where the judgment actually matters. The agent does the work. The human decides what ships.

If you are building or thinking about Software Factory, the real question is not which agents to build first. It is which steps you are willing to hand over, and which you want to keep your eyes on. Start with the ones you can measure, and hand them over one at a time.

I have only read a few posts on Software Factory so far. As I come across more, I will keep sharing.

References

• Vercel, “Building a software factory for AI SDK”: https://vercel.com/blog/building-a-software-factory-for-ai-sdk

• Port, “AI software factory”: https://www.port.io/blog/ai-software-factory

• Augment Code, “Beyond AI coding agents: how we built Augment’s software factory”: https://www.augmentcode.com/blog/beyond-ai-coding-agents-how-we-built-augments-software-factory

The numbers in the Vercel section are theirs: open issues, pull requests, npm downloads, stars, and the share of merged PRs and closed issues. The teamwork analogy, the reviewer framing, and everything under “How I would put this into practice” are my own thinking.

This method is for nonfiction. It is for the papers, chapters, and manuals that you read because you need to understand a new concept or a skill. It is not for stories. You do not want to “skim” a novel and ruin the ending. But when you are learning a difficult concept, reading front to back is a trap.

People compare learning a tough subject to solving a puzzle. The comparison misses a hard part. With a puzzle, every piece is already on the table. You just figure out which one goes where. With a difficult concept, the pieces aren’t handed to you. You have to decide which ones matter, and then work out how they fit.

Reading front to back assumes all the pieces matter, in the order the author stacked them. That is how you end up with a bookshelf of half-read chapters and a head full of the first few pages or paragraphs of other people’s arguments.

The way I read now looks like mapmaking. You don’t understand a city by walking every street in number order. You guess what the territory is, you fly over it to see its shape, you hike only the parts relevant to where you’re going, and then you draw the map from memory so it’s yours.

Here are the four moves. I use them on papers, chapters, manuals, and anything dense enough to be scary.

Step 1: Guess the territory

Before reading anything, look at the title and guess what the material is about. Write the guess down. A few lines are enough.

Then write the questions you’d want answered if your guess is right. These first questions are anchors: they catch the first wave of information while your understanding is still shallow. As you read, you refine them, and those refined questions are where the real learning happens.

A wrong guess is still better than no guess. You corrected your model, and you learned exactly where it broke. That’s faster than drifting through a chapter with no model at all.

Step 2: Fly over the landscape

Now preview the whole thing, quickly. Don’t read. Skim. Grab the landmarks: title, abstract, introduction, conclusion. Pictures, captions, and section headings. You’re looking for the shape of the piece and the parts that matter for what you need.

Think of it as looking at a city from altitude. You don’t study individual houses. You see where the main roads run and which neighborhoods you actually need to visit.

This is also when I draw my mindmap. A blank sheet of paper, turned sideways so it’s wider than it is tall. One question at the center, like “What is this paper about?” As I skim, I write keywords and short phrases around it. By the end of the preview, I have a skeleton of the material in my own words.

Step 3: Hike the peaks

From the satellite view, you know which sections deserve your time. Go straight to them. Read with a specific question, or better, a project you’re actually working on. Once you find the answer you’re looking for, highlight it and stop. The parts of the material that aren’t useful to you right now are simply off your route.

I used to read with no purpose at all, hoping it would stick. My schedule is too tight for that now. Reading everything is the slowest way to learn. Reading what answers your question is how you finish things.

When a section fights back, don’t reread it harder. Try these instead:

• Copy the difficult lines word for word. Writing them out forces your brain to actually see the structure.

• Redraw figures in different colors. The act of drawing pushes your eye through every detail.

• Color-code diagrams. Use red for parts you don’t understand, and write your questions right next to them.

Step 4: Draw the map from memory

The final move is the one many skip, and the one that tests you. Put the material away. Close the tab. Now write the summary from memory, in your own words.

If you can’t write it, you haven’t learned it. The parts you can’t reconstruct are the gaps in your map.

I write my summaries with three pens. Blue for the summary, red for questions, purple for new ideas. When a question or an idea pops up, I switch pens, capture it, and switch back. I don’t chase answers while writing. I just mark the spot on the map so I can come back to it later without breaking my flow.

Reading is selection

In a knowledge economy, your career runs on learning difficult things quickly. Time is short, so the whole game is selection: choosing the material that deserves your attention, then making sure it stays with you.

Guess the territory. Fly over the landscape. Hike the peaks you need. Draw the map from memory so the understanding belongs to you.

Try it on one thing this week. A paper, a chapter, or a manual you’ve been avoiding. Then reply and tell me which move felt different.

Recently I was revising slides for a patent disclosure. I had a list of changes that would have taken hours to make manually. I prompted the LLM with what I needed, went through a few iterations, and finished before my coffee got cold.

The point is not that I am clever. The point is that LLMs have become a commodity. We all use them for dozens of things every day.

But here is what I actually want to tell you.

When you have a business problem. When you need to get deep analytics from your data. When someone asks you to build something that will run in production and make decisions that matter. Is an LLM always the right tool?

The answer is no. And treating an LLM like a Swiss Army knife, something you reach for by default no matter the task, is the fastest way to build something that breaks.

The IKEA toolbox

About 12 years ago I walked into an IKEA in Canada and bought a toolbox. It had a hammer, a set of screwdrivers, pliers, the usual. One box, many tools.

Here is the thing about that toolbox. You do not use the hammer for everything. If you need to unscrew something and you grab the hammer, it is not going to do the job. It is going to push it in harder. You need a screwdriver.

That is how a lot of people are using LLMs right now. They have a problem, they know LLMs are powerful, so they reach for the LLM. Sometimes it works. Sometimes it kind of works but takes ten times the effort it should. And sometimes it fails in ways they do not even notice because the output sounds confident enough.

An LLM is one tool. A powerful one. But it is not the only tool, and it is not always the right one for the task in front of you.

The real skill is knowing what set of tools you have access to, and which one is likely to be the best fit. Then making a judgment call and starting there.

What is a model, anyway?

Most of the AI models in use today are created from data. You collect data. You run it through a learning algorithm, of which there are many. The algorithm, paired with the data, produces a model. That model becomes a reusable tool. You feed it new data and it gives you output.

If the training data changes, you retrain or refresh the model. Simple enough.

The models I want to walk through today fall into two broad camps. The dividing line is what kind of data you are dealing with.

The two data worlds

The data around us can be split into two categories.

Structured data. This is the kind you store in a database like Db2. It has a fixed schema. Rows and columns. You know exactly what each column holds: employee name, employee age, department, hire date. It can also live in a CSV file or a spreadsheet. But the shape is defined and consistent.

Unstructured data. PDFs, emails, web articles, videos, images. No fixed schema. You can store a PDF in a database column, sure. But you are just storing it. You cannot query into it. You cannot ask the database to find the paragraph that answers a specific question. The data has no shape the database can reason about.

Different data, different models.

For structured data, there are well established model families. Classification. Regression. Clustering. These have been around for decades.

For unstructured data, we lean on pre-trained models. These are the ones big tech companies have spent millions of dollars training on massive volumes of internet data. Some live behind paywalls, like the models from Anthropic or OpenAI. Many others are available on Hugging Face. You can download them, run them locally, build on top of them.

And because these pre-trained language models are so powerful and freely available, they are starting to blur some lines. We are seeing applications of these models even for structured data problems. You will see an example of that in a minute.

The structured data toolbox

If your data lives in rows and columns, three model families cover most of the problems you will face.

Classification. You have data coming in, and you want to put it in the right bucket. Your email inbox does this every day. Incoming emails get classified as spam or not spam. A customer list gets split into tiers. High risk of churn versus low risk. Classification works when there is a predefined set of categories and your job is to assign each row to the right one.

Regression. You want to predict a number. House prices. Stock prices. How much a customer is likely to spend on their next visit. You have a set of features about each record and you want to predict a numerical value.

Clustering. What if you have the features but no labels? Nobody has told you what the categories are. You just want to know if there are natural groupings in the data. Maybe you have a stack of books and you are deciding how to organize them. By topic? By publication year? By technical versus non-technical? There is no single correct answer.

Classification and regression are both supervised learning. You give the model inputs and the known outputs, and it learns to produce more accurate outputs over time. Clustering is unsupervised. You give it only the inputs and ask it to find the patterns.

An example regression model

Let me make this concrete.

A few weeks ago I built a demo for a talk. I had a table of 48,000 customer records. For each customer, I had some attributes: age, gender, a few other details. And for a large subset of those customers, I also had their spending amount.

The goal was simple. Train a model that could predict how much a customer would spend, given only their attributes.

I did the entire thing inside Db2. Create a table. Call six stored procedures. That is it. The model trained, evaluated itself against a held-out portion of the data, and gave me an accuracy of about 92%. It was off by roughly $10 in either direction.

Watch this demo of a regression model I built using SQL.

When keywords are not enough

Let me shift to unstructured data, because this is where things get interesting.

Imagine you are back in university. You walk into an exam. The professor hands you a ten-page article, maybe 50 paragraphs total, and a list of questions. It is an open-book exam and you have tight time. What do you do?

You read each question. You pick out some keywords. You hit Control-F and search the document for those keywords. You find the matching passages, read them, and write your answer.

That is how text retrieval has worked for decades. Keyword search. It is precise. It can do exact matching, synonym expansion, all of that. But it has a fundamental limitation: it sees parts, not the whole. It is looking at individual words and phrases. It does not understand what a paragraph is about overall.

Now imagine the questions use words that do not appear in any of the paragraphs. Yet the answers are definitely in there. Keyword search fails completely. You would have to read every paragraph, make sense of each one, and find the answer yourself. That takes real thinking.

This is where embedding models come in.

How embeddings see the whole paragraph

An embedding model is a language model that has been trained on a massive volume of text. Its job is different from an LLM. Instead of generating a response, it converts text into numbers. A numerical fingerprint.

What it does is take an entire paragraph, as an example, look at it as a whole, and compress all the information it can into a fixed set of numbers. That numerical fingerprint captures the overall meaning of the paragraph. The topic, the theme, the concept. Not the individual words. The global sense of it.

Now take your exam question. Feed it to the same embedding model. It maps the question into the same numerical space. Then you do some vector math. In that vector space, which of the paragraph vectors is closest to your question vector? That is likely where your answer lives.

Think of the embedding model as an interpreter. When I travel and do not speak the local language, I use Google Translate. It takes what I say and projects it into a space the other person can understand. The embedding model does the same thing. It projects paragraphs and questions into a shared numerical space and looks for matches.

Here is the tradeoff. Keyword search sees parts but not the whole. Embeddings see the whole but not the parts. The real power comes when you combine both. Do a keyword search for precision, do a vector search for semantic understanding, and merge the results. That is what modern search applications do.

Product recommendations with vectors and SQL

This blurring I mentioned earlier, where pre-trained language models start solving structured data problems. Here is an example.

Say you have a database table of shoes. Each row has attributes: material, color, style, a few other details. You want to build a recommendation engine. When a customer looks at a particular shoe, show them similar ones.

The traditional approach would take a data science team months. They would need to design a specialized model, figure out the right formula, tune it, test it.

But now we have access to pre-trained language models. You can take all the attributes of a shoe, combine them into a single text description, and send it to an embedding model. The model gives you back a vector for that shoe. Do that for every shoe in your catalog. Store the vectors alongside the relational data. Now, when someone looks at a shoe, you search for other shoes whose vectors are closest to it.

The beautiful part is you can also apply relational filters before you search. Only show me shoes available in a specific store. Only show me shoes in a particular size. Filter first, then search the vectors. That is the advantage of doing this inside a relational database.

Which model when?

You do not need to memorize a taxonomy. You need a simple decision process.

If your question is about individual rows and columns and you want to find a category: start with classification.

If you want to predict a numerical value from a row: start with regression.

If you have rows of data and you want to find natural groupings, with no predefined labels: start with clustering.

If you are dealing with text or images and you want to find similarity matches: try embeddings.

If you have text and you want to generate content: that is where text generation models, the LLMs we all know, come in.

These are starting points. Not rigid rules. As you dig into a problem, you might find that combining techniques works better. But this framework gives you a place to begin.

The skill is knowing the toolbox

LLMs are one tool. A powerful tool. I use them every day, and you do too.

But the skill is not about knowing one tool really well. The skill is understanding the full set of tools you have, and making a judgment call about which one fits the problem in front of you.

The next time someone tells you to solve a problem with an LLM, ask yourself: am I holding a hammer when I need a screwdriver?

Start with the data. Look at what kind of answer you need. Then pick the model family that matches.

We connected over the call, exchanged warm salams, and he got straight to it. His first real question was about understanding customers. But I could tell he was asking something else underneath. He wanted validation for a role he had already chosen. The conversation that followed changed what he was looking for.

The Setup

Afnan is 20 years old, finishing his second year, doing an internship at OPG. He is working through a double degree in computer science and business. He has shadowed product managers, researched the field, and decided this is where he wants to go.

He asked me several questions. How do I understand customer needs? What separates effective product managers from average ones? Where should he position himself in the next few years?

But when I asked him why he wanted product management, his answer was telling.

He said he liked solving technical problems. He liked finding solutions when someone had a need. He liked the cross-combination across different teams and seeing the difference he could make.

I listened. And then I told him something he did not expect.

The Pivot

I should be clear: I am not a product manager.

My background is in engineering, research, and strategy. However, I work closely with product management teams every day. I sit between the code and the customer. I see the business strategy, and I see the technical implementation. I answered Afnan’s questions based on that intersection (what I’ve observed from effective PMs and what I know from my own experience working directly with customers).

I started by asking him a question: “Do you want to stay technical, non-technical, or a combination?”

He said combination. Product management felt like the sweet spot.

Then I asked again. “So you enjoy solving technical problems. Working with customers. Helping them find solutions. Is that right?”

He said yes.

“That would not probably be too much on the product manager’s role, though.”

Product management, as I have seen it, lives mostly on the business side. Working with customers, industry analysts, strategy. The business ownership of a product. It is not that you cannot bring technical work into the role, but the core of it is not solving technical problems directly. It is defining what to build, why to build it, and for whom.

If you really enjoy working with customers in solving a technical problem, then most likely a consulting role would be more appropriate. Or there are forward deployment engineers, client engineering teams (different companies call them differently), but the common thread is the same. You are hands-on with customers, solving technical problems.

I was not trying to talk him out of product management. I was trying to get him to ask a better question.

Find the alignment between your aspiration (what you actually enjoy doing) and the role. Do not fix a particular role title first and then try to make yourself fit. See if that role is going to get you where you want to go.

The Framework

Then we got to the real question. The one he came to ask.

“How do you understand the customer needs?”

The first layer is obvious. You need to connect with customers. Talk to them. Understand their use cases. Anyone can tell you this, and you can figure it out yourself. You need to talk to people.

But the nuanced part is that oftentimes you will not hear directly from the customers what they need. You have to put yourself in the customer’s shoes. You have to think beyond what they are asking.

So how do you figure this out?

One thing is to target a specific persona or use case category. You cannot target everyone. I learned this the hard way.

My role is focused on building capabilities inside Db2 that will help customers use Db2 as a database for their AI applications. When I talk to customers, many of them are not yet building AI applications. They are thinking about it. Some of them are still not putting active thought into it. Maybe it is somewhere in their future.

So you have to look beyond what customers are asking. You look at the industry broadly. Go outside your customer base and see what the needs are. And you need to be more hands-on.

I am also a AI practitioner. Those use cases I want to support in Db2 with my team, I build them myself. I look at the industry patterns. For example, if you look at the implementation of AI use cases (I do not have exact numbers), but roughly 80% of the use cases are not getting into production. Only about 20% make it.

So what pushes things to production? Think of the completeness of the AI stack. That is the question that drives what I build.

Then you have to give a vision to the customer. Show them what you are building, what you are considering building. Get them excited. Give them those possibilities.

Show them the path. Hold their hand. Take them through the journey.

It is a combination of educating them, showing them new angles, and then building a solution that works for them. It is a journey with your customers.

The Strategy

Then I told him something more specific. Something I made a conscious decision about a few years ago.

In the customer base for Db2, there are database administrators. There are business leaders. There are application developers building applications with Db2. Even within application developers, there is a subcategory: AI application developers.

I made a decision that I was not going to target every single persona. Because then my message would be cluttered. It would not land.

I picked one. AI application developers.

Then I went deeper. I looked at their use case patterns. What are they trying to build? What is stopping them? How can we support this?

I started creating tutorials. First, a lot of beginner-friendly tutorials. I still do those. Then I focused on more advanced tutorials. What it takes to push things to production. The pain points. The gaps.

Then I reverse-engineered from there. To support those kinds of use cases and production deployments, what do we need to bring into the database? What capabilities are missing?

Once you build this, you have a vision. Then when you talk to customers, they recognize that this is coming. Even if they are not actively building it yet, they see the path to get to the finish line. To production.

You have to think the entire journey.

The Advice

By the end of the conversation, Afnan was reconsidering things. Not in a bad way. He was thinking.

I told him: if your ultimate goal is to contribute technically, maybe spend some years early in your career focused on hands-on building. Less travel, less non-technical work. You learn more, you are more hands-on, you build a solid technical edge. Then you can always add product management or move toward it later.

A strong technical base is going to pay you off long term.

He said: “I’m going to kind of deepen my understanding. Maybe I have the wrong alignment on what product management is and the consulting side. I’m just going to put some thought into everything you told me.”

Then he asked if we could talk again in a few weeks.

That is all I could ask for. Not a fixed answer. A better question to sit with.

If you are early in your career, ask yourself what work you actually enjoy. Not what title sounds good. Find the alignment first. Everything else follows from there.

My writing process has three steps.

Step 1: Finding an idea. I pick an idea that is fresh in my mind, something I worked on or learned within the last day or two. I don’t go too far back. Reconstructing my memory takes time that I don’t have. The freshness gives me extra excitement because the details are still vivid and I want to share them right away. If I wait too long, I forget the details and the emotional attachment weakens.

For example, yesterday I published a post after finishing a chapter from AI-Powered Search. Last week I published a post on calling an LLM from Db2 using SQL, written the same day as the tutorial on a Git repo. Both were fresh in my mind.

If there are multiple candidate ideas, I trust my intuition and go with the one I feel a strong pull for.

Step 2: Developing the idea. I have two methods for coming up with the first draft: dictation and writing by hand. Which one I use depends on how heavy the idea is.

For lighter content, I use dictation. This includes announcing a tutorial, reposting someone else’s post with my own brief thoughts, or sharing a lighter non-technical topic like a productivity tip. In all these cases, the actual content is already developed and published, either in a Git repo or by someone else, or it’s light. Dictation saves time, and it is still my own voice, my own language, my own tone. It becomes a conversation.

For deeper technical content, I brainstorm quite a bit. For example, last week I published a post on multi-vector representation. That is a complex topic and something new I came across. I ask myself: what is the concept I want to convey? I drew a mind map of the concepts, then I wrote the post by hand. I don’t type it on the computer. I take a screenshot of my handwritten notes, give it to AI, and ask it to transcribe them into a digital note. Then I work with it digitally for editing.

Step 3: Editing. I keep drafting and editing as two separate steps. I do not edit while I am writing the first draft. My goal is a messy first draft with all the grammatical mistakes, spelling errors, and missing ideas. I do not pause to edit.

Once I have the draft in digital form, I go through the editing phase in multiple passes.

1. I ask Claude to check for grammatical mistakes, fix them, and put every fix in bold. For each fix, I ask it to add an explanation in italic, following the fix, so I can review all the fixes along with its explanation. I review each grammar fix and the explanation, and either accept or reject.

2. Then a redundancy check. I ask Claude to identify the redundancies in my draft and put a comment next to each one in italics. If I agree, I accept the cuts.

3. Then a clutter check, following William Zinsser’s editing principles from On Writing Well. Same strategy: keep the changes in bold, show the explanation in italics.

4. Finally, I ask it to identify and fix any ambiguity. For example, if there is a pronoun and it is not clear what it points to, I ask it to be more specific.

After these passes, I have a final draft. I read it out loud from beginning to end. If any part doesn’t sound like my voice, i.e., the AI editing has taken away my tone, style, or language, I speak it out loud to the AI and say: use my version, what I just dictated, instead of that line in the draft.

Then I read it again from beginning to end. If it sounds like my own voice, I add line breaks so it is readable on LinkedIn, paste it in, and read it one more time to make sure nothing was lost or broken during copying. If it all looks good, I click post.

After building this hybrid search app, I initially used an AI coding agent to generate synthetic test queries and evaluate the results. I was not happy with that approach. Using an LLM to generate test queries, and then using the LLM to judge the results, felt like grading my own homework. So, I built this evaluation workbench.

In information retrieval, the standard practice is to use queries from real human users to evaluate search quality, and to have those same users judge the relevancy of the results.

In my evaluation workbench, I can construct a set of test queries, and collect relevancy judgments from human users.

Here’s how it works:

1. I click a button to start a new test query set and give it a name.

2. I type a query into the test query set.

3. The workbench runs searches against the lexical index and the vector index, pulling the top 10 results from each.

4. It combines both lists, removes duplicates, and discards the relevancy rankings from the individual indices.

5. I go through that combined list of results.

6. I give my own relevancy judgment on each result.

7. I use a three-point relevancy scale: Highly Relevant, Relevant, or Not Relevant.

At the end of this process, for each test query I have 15 to 20 result documents, and for each document I have a relevancy judgment. I repeat this process for whatever number of queries I want in a test set. Initially I am aiming for 15 to 20 test queries per set.

Once I have a test set, I go to the Evaluate tab, pick it from a dropdown, and run the evaluation.

Using this evaluation framework, I can now create my own yardstick for evaluating the performance of my app, not using synthetic queries coming from an LLM and the relevancy judgment decided by an LLM.

The accompanying video walks you through the workbench UI, creating a sample test query, and assessing the relevancy of the result documents.

If you are building AI search (RAG, vector search, hybrid search), how are you evaluating the quality of your retriever? Are you using synthetic queries from an LLM, or something else?

Then life happened. Advanced courses. A career. A move into AI architecture. The fundamentals didn’t go anywhere; they got covered over by everything I learned after them. Twenty-five years is a long time to let a foundation sit unused.

Most days, that doesn’t matter. I’m thinking about embeddings, retrieval pipelines, system architecture. I almost never think about the byte. But I’m currently designing a RAG solution, and as I worked through the design, I ran into a fact I had been quietly assuming away: not all the source data is in English. Some of it sits in databases built for other languages, encoded in ways that aren’t UTF-8 and aren’t always obvious from the outside. That raised a set of questions I couldn’t answer cleanly: how is the text actually encoded in the source database? What does it look like at the byte level? What happens when something downstream assumes one thing and the source provides another?

So today I went back. I opened a chat with an AI and asked the simplest questions I could think of, one after another, until the rust came off. How does a computer store the letter A? Where do bytes enter the story? How does it know where one string ends and the next begins? The concepts I had filed away in 2000 came back, slowly, just as clear as the first time.

This post is that refresher.

What you’ll get

If you took CS courses once and haven’t touched this material in years, I want this post to do one thing: rebuild your mental model of how English text lives in a computer. From the keystroke to the disk. By the end, you’ll see the bytes under your strings, and you’ll notice when an assumption you’ve made for years quietly stops holding.

The scope is narrow. English only. Not Unicode. Not multi-byte encodings. The moment text from other languages enters the picture, a lot of these assumptions break, but that’s a separate conversation. This post is the foundation everything else sits on.

1. Characters are numbers

A computer doesn’t know what “A” is. It only stores numbers. So someone had to decide: when you press A, what number gets saved?

That decision lives in a table called ASCII:

Press A. The system translates that to 65. The computer stores 65.

The table isn’t stored alongside the data. It’s an agreement between whoever wrote the bytes and whoever reads them. The byte on disk is only a number. The byte becomes “A” only because the program reading it agrees to interpret it that way.

2. Bytes: the container

A byte is 8 bits. A bit is a 0 or a 1. Eight switches in a row give you 2⁸ = 256 values (0 through 255).

Here’s 65 in bits:

0 1 0 0 0 0 0 1

Each position is a power of 2: 128, 64, 32, 16, 8, 4, 2, 1. Add the 1-bits: 64 + 1 = 65. That’s the letter A as it actually exists on disk.

ASCII was originally a 7-bit code, defining 128 characters: enough for English letters, digits, punctuation, and a handful of control codes. When 8-bit bytes became the standard, the upper 128 values opened up. Different countries filled those slots with their own characters. Japanese and Chinese have far more than 256 characters in total, so those languages can’t fit in a single byte at all. They have to use multiple bytes per character.

For English, one byte per character is enough. Every English-only system quietly leans on that assumption.

3. Everything is bytes

Bytes have no type. Photos, audio, integers, structs, ML model weights, all of it is just sequences of bytes. The byte itself is dumb. It holds a number from 0 to 255 and nothing else.

What changes from one kind of data to another is two things: how many bytes you use, and what rulebook you read them with.

• One English letter: 1 byte

• A 32-bit integer: 4 bytes

• A model weight tensor: gigabytes

Across all of those, the bytes look identical. The program reading them brings the meaning. Same bits, different agreement, different interpretation.

This is why production bugs happen. A file opens with the wrong parser. A packet is misaligned. A text column gets handed binary data. The bytes were fine. The agreement broke.

4. From one character to a string

If “A” is one byte, “Hello” is five bytes in a row:

H     e     l     l     o
72    101   108   108   111

Five numbers. That’s the whole string. The computer sees a row of bytes and treats them as text because the program told it to.

Spaces, tabs, and newlines are characters too. They have ASCII numbers and take up bytes, exactly like letters do.

"Hello World" is 11 bytes, not 10. The space counts. "Hello\n" is 6 bytes, not 5. The newline counts. They're characters, with byte values like everything else.

We treat these as structure. Underneath, they’re content.

5. Where does a string end?

Two strings sit next to each other on disk: “Hello” and “World.”

72 101 108 108 111 87 111 114 108 100

How does the computer know “Hello” ends after the fifth byte? There’s no fence. No separator. Just numbers in a row.

Computer science settled on two answers.

Strategy 1: Length-prefixed

Store the length first, then the content:

[5] 72 101 108 108 111   [5] 87 111 114 108 100
 ↑                        ↑
 "next 5 bytes            "next 5 bytes
  are one string"          are one string"

The reading program, called the reader, sees 5, counts off 5 bytes, knows the string is done. Any byte value, including 0, can sit inside the string.

This is what modern systems use. Databases. Network protocols. The string types built into modern programming languages.

Strategy 2: Null-terminated

Pick a special byte to mean “end.” Traditionally, the byte 0, the null byte.

72 101 108 108 111 [0]   87 111 114 108 100 [0]
                    ↑                        ↑
                  "stop"                   "stop"

The reader scans until it hits 0. This is how C represents strings.

The serious limitation: you cannot store the byte 0 inside the string. The moment the reader sees 0, it thinks the string ended. Text never contains a byte 0, so this works for strings. Binary files like images and archives are full of zeros, so it doesn’t.

That’s why modern systems went length-prefixed. You spend a couple of bytes on the length. In exchange, any byte value can live in the string, and you don’t have to scan to find the end.

6. Bytes don't tell you what they are

In a length-prefixed string, the length byte and the content bytes look identical. The byte 5 (00000101) and a content byte holding the value 5 are the same eight bits. So how does the computer know byte 1 is a length and not a letter?

The computer doesn't know from the bytes. It knows from position.

The rule “byte 1 is a length” doesn’t live in the data. It lives in the code reading the data. The format spec, the database engine, the protocol definition told the program: “byte 1 is a length.” That’s the agreement.

Lose the agreement and you have noise. Open a JPEG as a Word document. The bytes are valid. Word is reading them under the wrong format.

This is why file formats are specifications. PNG starts with a specific 8-byte signature. Database row layouts are defined down to the byte. The bytes are always just bytes. The format is the contract.

Tokenizers are agreements about how to split byte sequences into tokens. AI model weight files are bytes interpreted under a fixed schema. When the agreement between writer and reader breaks, you get bugs that don't show up in unit tests, because the bytes themselves are valid.

7. What I took away

English is the easy case by design, not by accident. ASCII was built for English in 1963, and a single byte is more than enough to hold any ASCII character. So in English, one character is one byte. Counting characters and counting bytes give the same answer. English-only systems never have to think about this.

That property stops holding the moment text from other languages enters the system. In other languages, characters often take more than one byte. Character boundaries stop being obvious. Assumptions about length, indexing, and splitting that worked for decades suddenly don’t.

Where this goes from here is harder: multi-byte encodings, Unicode, the point where bytes and characters stop being the same thing. That’s what I’m still working through.

If you’ve built a basic RAG pipeline, you’ve done this: chunk your documents, embed them with a model like text-embedding-3-small, store the vectors, and at query time embed the user’s question and pull back the top-k most similar chunks.

It works. Sort of. Then one day a user asks a reasonable question, and your pipeline returns chunks that are about the right topic but don’t answer the question.

Sometimes worse. The user asks “Is ibuprofen safe to take during late pregnancy?” The pipeline returns this chunk as the top match:

“Ibuprofen is widely used and considered safe for general pain relief.”

The chunk that actually answers the question sits two ranks below:

“Ibuprofen is not recommended after 20 weeks of pregnancy due to fetal kidney risks.”

Both chunks are about ibuprofen safety, so vector search ranks both highly. But only one answers the question the user asked. The pipeline can’t tell the difference, and confidently hands back the wrong chunk.

This post explains why that happens, and how re-rankers fix it.

I published a companion GitHub repo with a runnable notebook that demonstrates three common ways vector search fails (negation, missed constraints, topic-over-answer) and shows the re-ranker fixing them.

The Library Analogy

A giant library. Every book has many sections, each holding facts, claims, definitions, maybe answers to questions someone will ask one day.

You need to make it searchable. So you hire a librarian to summarize every section on a small card. The catch: she writes the cards before any reader walks in, with no idea what readers will ask.

What’s the best she can do? Write a high-level summary. “This section discusses the causes of type 2 diabetes.” “This section covers lithium-ion battery thermal runaway.”

These cards capture the topic. They don’t capture every fact, every constraint, every nuance. A one-line summary loses detail by definition.

That’s what an embedding model does. Feed it a chunk of text and you get back a vector: a list of numbers, usually between 384 and 1536 long, depending on the model. The vector is the digital summary card. It captures roughly what the chunk is about, written before any question is asked.

A reader walks in with a question. She writes that on a card too. The librarian compares the question card against every section card and hands back the closest matches.

That’s vector search. Embed the question, compute similarity against the pre-computed chunk vectors, return the top matches.

How Embedding Models Work, and Where They Break

An embedding model is a transformer encoder, a model that reads text and produces numbers that represent it. In vector search, the encoder runs twice: once on each chunk (ahead of time, to build the index) and once on each query (at search time). The two outputs (vectors) are then compared by a similarity score computed outside the model.

This architecture is called a bi-encoder. The model never sees the query and chunk together.

The bi-encoder design is what makes vector search fast. You pre-compute every chunk vector once, store them in an index, and reuse them for every future query. At query time, you embed the question once and search across millions of vectors in milliseconds.

The architecture has two compressions, both lossy:

Chunks get compressed before anyone asks a question. A chunk holds several facts, possibly answers to several questions, plus context and noise. All of it squashes into one vector: roughly the aggregate of everything in the chunk, in the model’s coordinate space (often called “semantic space” because chunks with similar meanings land near each other). No fact gets its own representation. The model doesn’t know which fact will matter for a future question, so it can’t favor any.

Questions get compressed without knowing which document they’ll meet. Same problem on the question side. Intent, constraints, framing: all reduced to one topical fingerprint. When you embed “What was Apple’s revenue in Q3 2024?”, the vector mostly represents “Apple revenue recent quarter.” “Q3 2024” barely registers.

You’re comparing two lossy fingerprints. Nothing in your pipeline ever looks at the raw question alongside the raw chunk.

Three Common Ways Vector Search Fails

These are three canonical failure modes the companion repo shows. Each maps to a query in the demo notebook.

Negation gets lost.

• Question: “Is ibuprofen recommended late in pregnancy?”

• Chunk A: “Ibuprofen is widely used for general pain relief.”

• Chunk B: “Ibuprofen is not recommended after 20 weeks of pregnancy due to fetal kidney risks.”

Both chunks share the same vocabulary about ibuprofen, so vector search ranks them similarly. But only Chunk B answers the question about late pregnancy; Chunk A answers a different question, about general pain relief. The model can’t tell the difference.

Constraints get washed out.

• Question: “What was Apple’s revenue in Q3 2024?”

• Chunk A: “Apple’s Q2 2023 revenue grew significantly, reflecting strong iPhone sales.”

• Chunk B: “Apple reported Q3 2024 revenue of $85.8 billion.”

Both chunks share the same vocabulary about Apple and quarterly revenue, so vector search ranks them similarly. But only Chunk B answers the question about Q3 2024; Chunk A answers a different question, about Q2 2023. The exact quarter barely registers in either vector.

Topic match beats answer match.

• Question: “How do I reset my password?”

• Chunk A: “Password security is critical. Use strong passwords, enable 2FA, never share passwords...”

• Chunk B: “To reset your password, click ‘Forgot Password’ on the login page...”

Both chunks share the same vocabulary about passwords, so vector search ranks them similarly. But only Chunk B answers the question about resetting a password; Chunk A answers a different question, about password security best practices.

Every failure traces back to one source: vector search ranks chunks by topical overlap, but topical overlap doesn’t tell you which question each chunk actually answers. The fix is in the architecture.

The Re-Ranker

The architectural fix is to bring the question and the chunk inside the same model. Instead of comparing two vectors outside the model, a re-ranker reads the question and a candidate chunk together in one forward pass, and scores how well the chunk answers that question. It compares them word by word.

This architecture is called a cross-encoder, because the query and passage attend across each other inside the model. A re-ranker is the API around it: you send a query and a list of candidates, and the service pairs the query with each candidate internally, runs the cross-encoder on every pair, and returns the sorted list.

The cross-encoder sees which chunk answers the asked question, even when several chunks share the same vocabulary: late-pregnancy safety vs. general use, Q3 2024 vs. Q2 2023, password-reset instructions vs. password best practices. It returns a relevance score for every pair: higher means more relevant.

If you were searching by hand, you’d do the same thing. You wouldn’t write summary cards in advance. You’d grab a stack of candidate books, sit down with the question in front of you, and read each passage with the question in mind. Then you’d rank them.

Why joint processing works

The cross-encoder’s joint processing, reading the query and passage in one forward pass, works because of the transformer’s attention mechanism. At every layer, each token weights its relationship to every other token in the input. With the question and chunk in the same input, the model can match “Q3 2024” in the question against “Q3 2024” in one chunk and “Q2 2023” in another, and tell which one the question is asking about.

A bi-encoder runs two separate forward passes. The model has no chance to compare words across them. The comparison happens at the end, between two finished vectors, after the word-level details have been compressed away.

Joint processing means more information flowing through the same computation.

How does the model know which words are query and which are passage (answer chunk)?

If both texts share one input, how does the model tell query from passage?

Two special tokens. [CLS] marks the start of the input. [SEP] marks the boundary between query and passage. The combined input looks like this:

[CLS] is ibuprofen safe during late pregnancy ? [SEP] ibuprofen is not recommended after 20 weeks . [SEP]

The model processes both texts together, but always knows which words came from the query and which came from the passage.

Input and output

A re-ranker typically takes 10 to 100 candidate passages per query. It returns a relevance score per passage. Some models return values between 0 and 1; others return unbounded numbers, often in the -10 to +10 range, where higher means more relevant. Most also return a sorted index list so you know which score belongs to which passage.

You sort by score and keep the top N for your LLM.

One constraint: most cross-encoders cap input at 512 tokens for the combined query plus passage. If your chunks are long, the end gets truncated, which quietly degrades quality. Keep chunks under about 400 tokens, or pick a longer-context re-ranker like bge-reranker-v2-m3, which supports inputs up to 8192 tokens (though it’s fine-tuned at 1024, so very long inputs see diminishing returns).

Bi-encoder and cross-encoder, side by side

The bi-encoder and the cross-encoder are built on the same transformer encoder, used differently. The bi-encoder runs it twice and compares vectors outside the model; the cross-encoder runs it once and judges relevance inside.

The bi-encoder gives up precision to search at scale. The cross-encoder gives up scale to score with precision. Each makes up for the other’s weakness, which is why production RAG pipelines use both, in that order.

In your pipeline, the re-ranker call sits between vector search and the LLM:

Why Not Use Re-Rankers for Everything?

A bi-encoder lets you pre-compute every chunk’s vector once and index it. At query time, you embed the question once and search across millions of vectors in milliseconds.

A re-ranker can’t be pre-computed. Every (query, passage) pair runs through the full model at query time. Re-ranking a million chunks per query would take hours.

So you combine them:

1. Retrieval with a bi-encoder. Pull 50 to 100 candidates from the full corpus. Cast a wide net. The goal is recall: make sure the right chunks sit somewhere in those 100. (Recall measures how often the right answer appears in your results at all; precision measures how often the top result is right.)

2. Re-ranking with a cross-encoder. Score each of the 50 to 100 candidates against the question. Sort. Keep the top 5 or 10 for the LLM.

Speed at the recall stage, precision at the ranking stage. The two-stage approach mirrors the human librarian: narrow with rough signals, then read closely to pick the winners.

Why Not Just Let the LLM Filter for Relevance?

A reasonable question: why add a re-ranker at all? Why not send the top 50 chunks from vector search straight to the LLM and ask it to use only the relevant ones?

You can. It loses on accuracy, cost, and latency.

Accuracy. LLMs are bad at picking the most relevant chunks when many candidates sit in the same context window. The “Lost in the Middle” paper (Liu et al., 2023) showed that across GPT-3.5 and several other models, accuracy is highest when the relevant chunk sits at the very start or end of the context, and drops 20 to 30 percentage points when it sits in the middle. Follow-up work has confirmed the same pattern on GPT-4, Claude, and Gemini, even as their context windows grew past 100K tokens. A cross-encoder scores each (query, chunk) pair on its own. Position in a list never enters the math.

Cost. Sending 50 chunks (about 25,000 tokens) to a frontier LLM on every query runs roughly 10x the input-token cost of re-ranking first and sending only the top 5 (about 2,500 tokens). At 10,000 queries a day, the gap adds up.

Latency. Two stages are usually faster than one. A re-rank pass takes 100 to 300ms. Sending 25,000 tokens to a frontier LLM typically takes several seconds for the first token; 2,500 tokens takes one or two. Re-ranker plus a smaller LLM call ends up noticeably faster end to end.

A re-ranker is small, fast, and trained on millions of (query, doc, relevance) pairs to do one job. An LLM is trained to write fluent answers. Asking it to also act as a precise relevance classifier inside the same call moves the work to a slower, costlier, less accurate component.

The exception is small candidate sets. With only 3 to 5 chunks, position bias is mild and re-ranking adds little. Skip it. Past about 10 candidates, the gap opens fast.

When a Re-Ranker Isn’t Worth It

Re-rankers have their cost. Before you add one, check whether the cost is justified:

• Tight latency budgets. Re-ranking adds 100 to 500ms per query. If your end-to-end budget is under a second, that hurts.

• Mostly exact-keyword lookups. For product SKUs, error codes, and literal strings, BM25 (a keyword algorithm that scores documents by term frequency and rarity, used by most search engines before embeddings) often beats vector search on its own.

• High query volume. API re-rankers charge per call. At millions of queries per day, costs add up. Self-host an open-source re-ranker if volume is high.

If none of these apply, and your current pipeline returns “close but not quite” results, a re-ranker is worth exploring.

Re-Ranker Options

A short list of re-rankers in active use. They fall into two buckets: hosted APIs (call an endpoint, pay per query) and open-weights models (download and run yourself).

API-based:

• IBM watsonx.ai Rerank. Hosts several cross-encoder models.

• Cohere Rerank. Current models include rerank-v3.5 (4096-token context) and rerank-v4.0.

• Jina Reranker. Available as API or self-hosted.

• Voyage Rerank. API-based; current models include rerank-2 and rerank-2.5.

Open weights, self-host:

• BAAI/bge-reranker-v2-m3. Multilingual; supports inputs up to 8192 tokens.

• mixedbread-ai/mxbai-rerank-large-v2. Supports inputs up to 32K tokens.

• cross-encoder/ms-marco-MiniLM-L-6-v2. 6-layer MiniLM cross-encoder trained on MS MARCO; commonly used as a starter model.

Summary

Vector search compares fingerprints, not specifics. It compresses every chunk and every question into their vectors, then matches them by topical overlap. That works for “what is this about?” It fails when negations, exact constraints, or specific entities decide the answer.

A re-ranker reads the question and the chunk together, word by word, and scores how well one answers the other. Vector search narrows the corpus to about 50 candidates; the re-ranker picks the winners. Rough filtering, then careful reading.

The payoff is concrete: a correct chunk ranked 30th by embedding similarity gets promoted to position 1 or 2 by the re-ranker. That’s the precision gap the cross-encoder recovers.

If your RAG pipeline returns “close but not quite” answers, try a re-ranker next.

Try This

• Clone the companion repo: github.com/shaikhq/rag-reranker-demo. Run the notebook end to end, see the three failure modes, see them fixed.

• Swap in your own PDF and rerun. The same lift usually shows up on real data.

• Wire a re-ranker (watsonx.ai, Cohere, or a self-hosted BGE model) into your pipeline behind a feature flag.

• Measure top-1 and top-5 accuracy on a small eval set, with and without re-ranking.

• Tune the candidate count (50 vs 100 vs 200) and watch the precision-versus-latency curve.

I walked out of that meeting with a long list of gaps and one thought in my head: how am I going to pull all of this together?

I had just presented my plan for adding vector storage and search to Db2 at the Architecture Review Board, the committee of senior technical leaders who must approve every major new project before a line of code is written. I had walked in confident. I walked out humbled. The committee had found problem after problem in my plan that I had not considered. The list kept growing. Each item was a real use case that a customer would hit, and my plan had nothing to say about any of them.

I had spent nineteen years building software at IBM. I thought I knew what the job required.

I was wrong. Not about the technology. About what it actually takes to own a major AI project from concept to delivery inside a large, mature software product.

My Two Recent AI Projects at Db2

In the past two years I built and delivered two major AI projects at Db2.

The first was vector storage and similarity search, built directly into the database engine. This lets AI applications store collections of vector data inside the database and search through it to find similar items quickly. It is a core capability for modern AI use cases like recommendation engines and semantic search.

The second was connecting large language model APIs to Db2. This project lets application developers call these AI models directly from inside the database using standard SQL commands. Application developers no longer need to write separate code outside the database to do this.

These projects ran from the initial proposal to the final delivery under my technical ownership. More than twenty developers across multiple teams. Tight deadlines. Five Db2 platforms and many more deployment configurations. Db2 is a product used by companies for over thirty years, so a mistake is not just a bug report. It breaks the daily workflow of teams that have depended on it for decades.

The first project shipped to customers last year. The second has completed development and is scheduled to ship in the next few months.

These two projects taught me more about technical leadership than the previous nineteen years combined. Here are the five lessons I learned from leading these projects.

Lesson 1: A new feature must work with everything the product already has and does.

When we started building vector support, I thought the job was simple. Add something new. A new data type for storing vectors. A new way to search those stored vectors. Clean, contained work. Like building a new room in a house without touching the rest.

Users have built numerous business workflows around Db2’s existing features. There are tools for copying data out of the database to a file. There are tools for moving data from one database to another. There are SQL commands for every common task a user needs to do.

When we added the vector data type, users did not care about our internal design. They cared about their work. They expected the new vector type to work with all the same tools they already used every day with other data types. If the copy tool worked with varchar data type, they expected it to work with vectors too.

We were not just adding something new. We were taking on the responsibility of fitting into everything that already existed, without breaking the ecosystem. I had not understood that responsibility before I walked into the Architecture Review Board.

That is where the list of gaps I described in the opening came from. Each gap the committee found was a real use case that a customer would hit, and my plan had nothing to say about any of them.

I went straight to the senior architects and most experienced developers. We went through the list together, gap by gap. I asked for their guidance. We rewrote the plan several times until it was solid. A new shared plan, built together with the senior developers in my squad.

The method that came from those sessions: find the existing feature in Db2 most similar to the one you are adding. Search the entire codebase. Look at every place in the code that handles that existing feature, one by one, and decide whether your new feature needs to do the same thing. This gives you a map before you write a line of code. In a product built over many years, the people who built it will always find problems you missed. Bring them in before you think you are ready.

Lesson 2: A good design does not coordinate teams. A leader does.

The second project, integrating large language model APIs into Db2, made this lesson unavoidable. The feature required work from three component teams in sequence.

To add one of the new functions, we first needed new entries added to the database catalog, a central store that holds information about every object in the database. The catalog team had to finish their work before the query compiler team could start theirs. The query compiler team had to finish before the runtime team could finish theirs. Each team was waiting on the team before them. Like links in a chain.

If the catalog team finished one week late, the compiler team was stuck for a week doing nothing useful. If the compiler team was late, the runtime team was stuck too. The whole project fell behind.

I did not track these connections carefully at the start of the project. I assumed each team would manage their own work and speak up when stuck. They did not. Halfway through the project I could see that work was piling up in the wrong order. Teams were waiting on each other but not saying so out loud. Things were quietly falling apart.

I stopped the usual work and took charge of tracking every dependency myself. I asked each team one direct question: what do you need from another team before you can finish your work? I asked this at every standup meeting. If any work was not helping clear a dependency, I stopped it and pointed the team at what actually needed to happen next.

When a dependency could not be fixed in time, we found a workaround. The runtime team needed specific data from the catalog team, but the catalog work was still two weeks away. We hardcoded a temporary number as a stand-in. The runtime team used that number to keep building, completing 80 percent of their work without waiting. Once the catalog work was done, we swapped the hardcoded number for the real value. The team kept moving. The deadline held.

As a technical leader, you have to find the dependencies between teams, identify them clearly, write them down, and make sure every team can see what they are waiting for, and what other teams are waiting for from them. Do this early, or the project will slow down for reasons that are invisible until it is too late.

Lesson 3: A squad is not formed by putting people on the same project. It is built.

I formed my squad by pulling developers from different teams inside Db2. Some were experts in the runtime layer. Some worked on the catalog. Some worked on the query compiler. Every one of them was skilled and experienced in their own component.

What I did not expect was the resistance: not to the project itself, but to working outside their own component. Developers who spend years on one part of a codebase become very attached to that component. When asked to work in a different part, some resist. Not because they are difficult people, but because it is unfamiliar and they do not want to take responsibility for components owned by another team.

In a project where every layer of the product needs to work together, that reluctance causes real problems.

I had to build a new team culture through action, not words. I asked a runtime developer to write test cases for the query compiler section. I asked a compiler developer to write code for the catalog section. Every time someone stepped outside their usual area and helped another team, I called it out and thanked them. I wanted that behavior to feel normal, not unusual.

There was pushback. One situation stands out. The runtime team was waiting on a piece of work from the query compiler team. But the compiler team was in the middle of something that, if interrupted, would slow the whole project. Simply stopping their work was not the answer.

So we went deeper. We identified the specific code paths shared between the runtime and query compiler components, the intersection where both teams had a stake. Then we brought a runtime developer into that shared territory. They worked on the overlapping piece directly, consulting the compiler developer as needed and including them in the pull request review. Nobody had to abandon their current work. The dependency got cleared by finding where the two domains met and working from that common ground.

That became a pattern. Whenever two components had a dependency, I looked for the intersection, the code or interface they shared, and found a way to bring both developers to that point together. It was slower than simply assigning tasks. It was faster than waiting for one team to finish before the other could start.

I also made a point to listen. When someone had a concern about working outside their component, I heard them out and worked with them to find a way to contribute that felt manageable. People work better toward a shared goal when they feel heard.

Slowly the pushback faded. The squad started acting like a team. Once that happened, the speed of the work increased noticeably.

You cannot create a real team just by putting people on the same project. You build one by giving people a shared goal, listening to them, and reminding them, over and over, of what they are building together, until working as one team feels more natural than working apart.

Lesson 4: Testing runs alongside development. Not after it.

We wrote the code. We ran tests on our own development machines. Everything passed. We delivered the code to the release branch. I thought we were ready.

What I did not know was that Db2 has a central automated testing system. Think of it as a shared dashboard that runs thousands of tests on every piece of delivered code across all platforms and all deployment setups, automatically and continuously. Anyone can look at it and see which tests are passing and which are failing.

I had never looked at it. I did not fully understand what it showed.

Our QA lead, Darrin Woodard, showed it to us one day. The first time I saw the results for our delivered code, it was bad. Test failures everywhere. Red marks across every platform and every setup we needed to support.

Db2 runs on five different OS platforms: standard Linux, Linux on Power, Linux on Z, AIX, and Windows. Our code had to work correctly on all five. Db2 also runs in different deployment setups. Some installations spread across multiple nodes, others optimized for different types of workloads such as analytical or transactional. Every setup type can expose different bugs.

I had not thought carefully about any of this when planning the project. I underestimated how many test cases we needed to write. I underestimated how long it would take to investigate each failure and fix it.

The Db2 Chief Architect, Mike Springgay, regularly reminded me of this throughout the project. He consistently brought up the importance of estimating the QA effort separately from the development effort. Eventually that lesson landed. I began to treat QA planning as its own work stream, not an afterthought.

After seeing the QA dashboard for the first time, I requested the QA lead to join our regular standup meetings. Twice a week, we opened the dashboard together and looked at every new test failure for the features we had delivered for vector support. Before the meeting ended, every failure had a name next to it: the developer in my squad responsible for investigating and fixing it.

Week by week, the failures went down. More green, less red. Then, before our internal delivery date, everything was green.

Writing code that works on your own development machine is only half the job. The other half is proving it works on every platform, in every deployment setup, in every situation a user might encounter. When it did land, it changed how I planned every subsequent release. QA was no longer something that happened after the work. It was part of the work.

Looking back, Lesson 1 and Lesson 4 are two sides of the same coin. Lesson 1 brought ecosystem awareness to the design phase. Lesson 4 confirmed that the implementation actually honored that ecosystem. One is about seeing clearly before you build. The other is about proving it after you do.

Lesson 5: Writing code made me a better technical leader.

For a long time I believed that leading a technical project meant staying at the level of architecture and direction, and that keeping my distance from the code made me more effective.

It did not. It kept me comfortable.

I was running design meetings. Writing technical proposals. Talking with directors, architects, and customers.

But all of it kept me away from writing code for the actual product features we were shipping. The developers on my team were writing code every day. I was managing, reviewing designs, making decisions, and representing the project to people outside the team. I understood the system at a high level: what it did, how the pieces fit together. I did not understand it at the level where the real problems live: inside the code.

I gave myself a coding assignment. Real product code: the kind that gets reviewed, approved, merged into the main codebase, and shipped to customers.

I started with a relatively simpler function: the vector dimension count function. This calculates the number of dimensions in an input vector. I had built an earlier version of this as a user-defined function, a separate script outside the database. Now I needed to build it properly as a native part of Db2 itself.

I wrote the SQL grammar rules that let users call the function. I wrote the code that actually runs the calculation. I built it the same way every developer on my team builds their features: design it, write it, test it, find the bugs, fix them, write more tests, repeat.

Implementing this first function took me nearly four weeks. I was not working on it full time. I was still doing everything else too. And I was learning this part of the codebase, much of it for the first time. I wrote notes about everything as I progressed: what I tried, what worked, what did not, what I learned. I kept a running log in the Git issue tracker. Screenshots. Code comments. A record of how the feature was built from nothing.

Then I sent the code for review. To a senior developer on my team, someone whose work I had been reviewing and approving for months.

He came back with more than fifteen things that needed to change.

I had thought the code was good. It ran. The tests passed on my machine. But “it runs” and “it is ready for shipping” are very different bars. A developer who has worked in this codebase for years sees things a newer person cannot see. I read every comment. I made every change. I sent it back. More comments. More changes. Several rounds. It was hard, not because the feedback was unfair, but because it was right.

Everything I wrote down during the first feature paid off on the second. The steps were similar. I had a working template for developing new features. Four weeks became two. On the third feature, more complex than the first two, I blocked time on my calendar and focused. I opened a pull request with complete, working code in just two days.

Not because the reviewers stopped looking carefully. Because the code I was writing was getting better.

I was no longer an outsider, a non-developer technical leader telling others what to build. I was one of them. One of the developers in the front line, co-creating the feature code. When I sat in design discussions, my questions were more specific, more tied to the actual code, more connected to how things would actually work. The team’s relationship with me changed too, not because of my job title, but because they had looked at my code line by line, told me what to fix, and watched me fix it the same way I asked them to fix their own work.

The day I stopped thinking of myself only as the leader and started being a developer again. That was the day I became a better leader.

What I Know Now

On any given day across these projects I was the architect, the project manager, the developer, the QA reviewer, the stakeholder manager, and the face of the project to customers and senior leadership. Sometimes all in the same day. That is what technical ownership of an AI project at this level actually looks like. It is not one job. It is many jobs, held together by a clear goal and the discipline to keep moving toward it.

These projects gave me something harder to build than technical knowledge: real confidence, strong working relationships across many teams, and the clarity to take on what comes next. I can now lead AI projects at a scale and complexity I could not have handled two years ago. There is more to build. I plan to build it.

When I walked out of that Architecture Review Board meeting with a long list of gaps, I did not have what I have now. I had nineteen years of experience and a plan that was not ready.

What I have now came from two years of doing the work, getting things wrong, and rebuilding my understanding one lesson at a time. It came from tracking dependencies no one else was tracking. From writing code that got sent back with fifteen things to fix. From saying the same thing to the same team dozens of times until it finally landed.

These are not lessons I read in a book or learned in a course. They are the kind that only come from doing the work, from standing in the middle of something hard and finding your way through it.

I shared them because I wish someone had shared them with me.

Last week, I hit a bug in code I was writing for Db2. The same category of problem that once kept me up until mid morning. I shared the error with an AI coding agent. Root cause identified. Fix applied. Done. Under two minutes.That is not a small improvement. That is a different kind of work.

Last Saturday I shared both of those stories with 35 university students from India, most of them computer science majors, preparing for careers in AI engineering. Then I showed them exactly what sits between those two moments, the tools, the habits, and the mindset shifts that made the difference.

In the session we covered three things. First, how I currently hold a full-time role as AI Architect at IBM Db2 and complete a PhD in AI at the same time, using AI tools to run both tracks in parallel. Second, three prompting habits that immediately change what you get back from any AI tool, including how I use AI when reading something new. Third, two free tools you can open today: Claude and NotebookLM.

The session also opened up questions from the audience that made it richer. Someone in the audience asked how a student studying neuroscience, not computer science, can start using AI. We talked about how AI makes you a sharper learner, not just a faster coder. And we closed with three specific tasks you can act on this weekend.

If you are an engineering student preparing for a career in AI, this session was made for you.

My first day at IBM was in 2005. I was a student intern, excited, prepared technically, and completely lost by the end of the day. Not lost in the code. Lost in everything around it. I joined full time shortly after graduating and the feeling did not go away. If anything it got sharper.

This is the post I wish had existed before I walked through those doors.

Where this came from

Before IBM I had done internships at smaller organizations in New Brunswick, a government office and a few small IT companies. Good experiences. But nothing that prepared me for what IBM actually was. A few friends had interned there before me but nobody told me what it was like on the inside. The only thing I thought I needed to prepare was my technical skills.

I was wrong.

Day one I was lost. Not after a few weeks of settling in. Day one. The scale of it, the complexity of it, the sheer number of things I did not understand hit me immediately. The people, the team dynamics, the processes, the unspoken expectations. Smart, experienced professionals all around me who seemed to know exactly how everything worked. I did not know how to navigate any of it. I did not know what was expected of me beyond the technical work. I did not know how to read the room, how to ask the right questions, or how to exist in that environment without feeling like I did not belong there.

The technology was never the problem. I could handle that.

What I couldn’t handle was everything else.

I struggled to communicate, really communicate. Listening in meetings without tuning out. Writing clearly under pressure. Speaking with any confidence in a room full of people who seemed to know exactly where they stood. Reading the room. Navigating across personalities. Knowing when to push and when to wait.

I didn’t know how to exist in that environment. I didn’t believe I belonged there.

I didn’t know the path forward. I just knew I was lost.

Three people changed that.

My supervisor Tahsin Rouf, my first IBM manager Kent Klaas, and my second manager Ed Van Gennip. Their support in my early years at IBM was not incidental. It was foundational. The trajectory I have been able to build over the past 20 years, everything I have learned, taught, and contributed, sits on the base they helped me build. Tahsin saw something in me at exactly the right moment and offered guidance when I had no idea how to ask for it. Kent supported me consistently and introduced me to mind mapping, a thinking tool I still use today for planning and problem solving. Ed believed in me deeply and gave me the kind of continuous feedback, support, and direction that most early-career people never get. I want to name them here because people like this rarely get named. They should.

And I joined Toastmasters.

That decision opened something I did not expect. Not just better communication. A deep curiosity about the foundational skills nobody had ever taught me. How to learn better. How to think more critically. How to listen deeply, and build the kind of discipline that actually compounds over a career. I started taking IBM’s internal courses. I started building personal frameworks. I started testing things on myself.

Then came the moment that pulled everything together.

About a year or two into my time at IBM, I volunteered to run Think Friday. A weekly internal knowledge-sharing session for our team. I wanted to do it well, not just show up and lead it. So before I ran my first session I read a book called “Not Another Meeting” by Frances A. Micale. That book gave me something specific: a method. How to prepare with a clear outcome in mind, how to structure the discussion, how to manage debate productively, how to take notes that were actually useful, and how to close with clear action items organized by topic and next step.

I ran Think Friday that way for two to three years.

Looking back, those sessions taught me more than I realized at the time. How to bring a group of people toward a common objective. How to hold a productive discussion without letting it drift. How to capture what actually matters and make it useful afterward. How to lead a room, not just occupy it.

That weekly meeting was a turning point. Not because it was high-stakes. Because it was consistent. I got the repetitions. And the confidence that came from those repetitions was real in a way that no single course or credential ever produced.

What was missing

Looking back, I was technically prepared and foundationally underprepared. Both things were true at the same time.

The technical skills got me hired. They kept me moving in the first few weeks. But the moment I needed to work across a team, recommend a clear solution, listen in a meeting without missing half of what was actually being said, or simply believe I had a right to be in the room, the technical skills went quiet. They had nothing to say.

Nobody had ever taught me those things. Not in school. Not in any course. Not in any internship prep session I had ever sat through.

I had to figure them out the hard way, mostly alone, mostly by making mistakes I did not understand until much later. The difference was that I eventually found people who believed in me, and I found a way to practice. Most students never find either.

The framework, step by step

After two decades of learning this myself and then teaching it to students, I have found the foundational skills fall into a small number of areas that show up again and again. These are not soft skills in the vague, resume-filler sense. They are specific and learnable.

1. Communication that actually works in a corporate environment. Not presentation polish. The ability to write a clear email, summarize a meeting, give feedback without creating defensiveness, and ask a question without sounding lost. This also includes thinking tools. Kent Klaas showed me mind mapping early in my career as a way to organize thinking before writing or speaking. It changed how I approach any problem that feels too big to tackle in a straight line. These skills are different from anything you practiced in school and they matter from day one.

2. Learning how to learn on the job. School gives you structured problems with known answers. Work gives you ambiguous problems with moving targets. Building a personal system for learning new things fast, the way I built one by reading a single practical book before running my first meeting, is a skill in itself.

3. Navigating the environment, not just the work. Understanding how decisions get made. Reading team dynamics. Knowing when to speak up and when to listen. This is not political in the negative sense. It is situational awareness, and it determines whether your technical work ever lands.

4. Believing you belong there. This one sounds soft. It is not. Self-doubt is not a feeling to push through. It is a pattern that shapes every decision you make, who you ask for help, whether you volunteer for harder problems, how you handle feedback. The students who address it directly move faster than the ones who try to outwork it.

5. Building discipline that compounds. The habits you build in your first year set the baseline for everything that follows. Not productivity hacks. The small consistent things: how you prepare for meetings, how you follow up, how you manage your own energy, how you stay curious when the work gets repetitive. Running Think Friday every week for two to three years was not glamorous. It was the repetition that built the skill.

The two assets most students skip

Following this kind of framework produces two things that matter more than any project portfolio.

The first is self-awareness about where you actually are. Not where you think you are, not where you hope you are. A clear read on what is working and what is not, grounded in real situations from your actual job.

The second is a personal operating system. A small set of principles and habits you have tested on yourself and trust. Not borrowed from a book. Yours. Mine started with a practical meeting guide and a weekly Friday session. It grew from there.

Why this works

Corporate environments, and AI teams in particular, move fast and reward people who can operate with ambiguity, communicate clearly under pressure, and keep building without waiting to be told what to do next.

The students I have seen thrive earliest are not always the most technically brilliant. They are the ones who understood early that the job was bigger than the code. They put time into the foundational skills deliberately, not because someone told them to, but because they could see the gap and decided to close it.

That gap is real. It is common. And unlike a lot of things in your career, it is entirely within your control to work on.

After 20 years at IBM, mentoring and helping more than 100 students across undergrad, masters, and PhD programs, I still see the same gap show up in the same way. Not because students are underprepared. Because nobody told them it existed.

That is exactly why I am sharing this with you now.

Students reach out to me regularly. They share their portfolios, ask for feedback, want to know why they are not hearing back from applications.

I look at their work. And almost every time, I see the same pattern.

A RAG pipeline. An AI agent. A classification model. Clean code, decent README, maybe a demo video. And then it stops there.

Building something real takes effort, and that effort is worth acknowledging. But there is a next step most students have not been shown, and that gap is what is holding them back.

That is what this post is about.

The question that separates candidates

When I review a portfolio as an AI architect, I am not only asking can you build this? I am also asking do you know if what you built is any good?

Those are not the same question. And in my experience, most portfolios only answer the first one.

Think about what the job actually involves. You will not be handed tutorials to replicate. You will be handed a system that partially works and asked to improve it. Or you will build something from scratch and be responsible for demonstrating it meets a quality standard before it ships.

The question your portfolio needs to answer is: can you take something that partially works and make it measurably better?

What this looks like in practice

Let us use RAG as the example. It is the most common AI portfolio project I see right now, and a good one to learn from.

You built a RAG pipeline. It retrieves documents and generates answers. You tested it with a few queries and the responses looked reasonable.

But do you actually know how well it is working?

There is an open source evaluation framework called RAGAS, published at EACL 2024, built specifically to evaluate RAG systems. It gives you measurable scores on dimensions that matter in production:

Faithfulness measures whether the model’s answer is grounded in the retrieved documents, or whether it is generating claims the documents never supported. This is how you detect and quantify hallucinations.

Answer Relevancy measures whether the response actually addresses what the user asked.

Context Precision and Recall measures whether your retriever is surfacing the right document chunks in the first place.

You run your system through RAGAS and get a faithfulness score of 0.61. That means roughly 4 in 10 claims your system generates are not supported by the retrieved documents. For most production use cases, that is not acceptable.

[What most portfolios show vs. what hiring managers are also looking for]

So you investigate. You find your chunk size is too large and the retriever is returning loosely related content alongside relevant content. You reduce the chunk size. Faithfulness moves to 0.74. You introduce a re-ranker to improve which chunks reach the model. Now it is 0.84.

You document every step. The baseline score, what you changed, why you made that change, and what improved as a result.

That documented process, not just the final system, is what demonstrates engineering thinking.

This applies beyond RAG

If you are building a classification model, it is worth asking: do you know your precision, recall, and F1 score, and do you understand which one matters most for your specific use case?

If it is a regression model, can you explain what your RMSE is telling you about your system’s real world behaviour, beyond the formula itself?

These are not advanced topics. They are foundational to working with models professionally. And developing fluency with them early will serve you well throughout your career.

There is a principle worth holding onto: you cannot confidently claim something works if you have no definition of what working means.

I came across this idea early in my career through a book called Exploring Requirements: Quality Before Design by Donald Gause and Gerald Weinberg. I read it years ago and it shaped how I think about building systems. Their core argument is straightforward: define quality before you design, not after. The book was written in 1989 about software engineering. The principle applies just as directly to AI systems today.

What production ready actually means

"Production ready" gets mentioned constantly in job descriptions. Here is what it actually means in practice.

Reliability means your system handles unexpected conditions without breaking. What happens when the retriever returns nothing useful? When an API call times out? A production system fails gracefully and recovers.

Observability means you can see what is happening inside your system after deployment. Are you logging inputs, outputs, and quality scores in a way that lets you diagnose problems when they occur?

Cost means your system stays economically viable at scale. How many LLM calls does it make per query? Does that remain reasonable if usage doubles or triples?

Edge cases are the inputs your system handles poorly. Every system has them. The difference between a student project and a production ready system is often just whether those edge cases were found and documented before users encountered them.

You do not need to have solved all of this as a student. No one expects that. But if your README shows you have thought about these dimensions, identified the limitations of your own system, and have a reasoned view on what you would address first, that communicates something important about how you think. It signals that you are ready to operate on a real engineering team.

That kind of thinking is what I have seen make a genuine difference when students move from portfolio to interview to role.

One thing to do this week

Take any project in your portfolio. Add one section to the README called Evaluation and Known Limitations. Answer these three questions:

## Evaluation and Known Limitations

**1. What metrics did you use, and what did the numbers show?**
e.g. "Faithfulness: 0.84 after two iterations. Context Recall: 0.71, identified as the next area to improve."

**2. Where does the system underperform, and under what conditions?**
e.g. "Retrieval degrades on queries shorter than 5 words. Fails gracefully but returns low-confidence responses."

**3. What would you address first to improve it, and why?**
e.g. "Improve Context Recall by experimenting with hybrid retrieval. Currently retriever is keyword-only."

The engineers who stand out are not always the ones who built the most projects. They are the ones who can examine a system honestly, articulate where it falls short, and think clearly about how to make it better.

That is the skill worth developing. And it is one you can start demonstrating today.

If you are preparing for an AI role and want feedback on your portfolio, feel free to reach out or leave a comment below. And if you have worked with evaluation frameworks or metrics in your own projects, I would be glad to hear what you found useful.

If you are an engineering student preparing for an AI internship and not sure what to build for your portfolio, this post is for you.

I want to share an approach that turned a single open-source tutorial into three publishable assets over a couple of weeks. It is not a shortcut. It is a deliberate process, and it taught me more than starting from scratch would have.

Where this came from

About a year ago, after a decade of working in AI and ML, I decided to learn agentic RAG properly. Not just read about it. Build something real with it.

I found a well-structured tutorial in the LangChain repo. It builds an agentic RAG system that loads web content, chunks it, indexes it for semantic search, and uses a LangGraph agent to decide when to retrieve versus when to answer directly. Clean, well-organized, a good foundation.

My first instinct was not to modify it. It was to understand the tutorial code completely.

So I ran it as written. Worked through every component. Traced each step in the workflow. Asked myself why each design choice was made, not just what it did. Why that retrieval strategy. How the agent decides when retrieved documents are relevant enough to generate an answer, and when to rewrite the query and try again. Where the boundaries between components are drawn and why.

That process took time. But it was the only reason I could later see where the gaps were. You cannot see what is missing in something you have only skimmed.

What the tutorial was missing

Once I had that foundation, the gaps became clear.

The tutorial used an in-memory vector store, fine for a prototype but not for production. It loaded content from public blog posts, useful for a demo but not connected to any real enterprise use case. There was no evaluation of retrieval quality, no baseline numbers to measure anything against, and no way to run any of it outside a Jupyter notebook.

Those were my extension targets.

I replaced the in-memory vector store with IBM Db2 as the vector database. Swapped the public blog content with a technical article I had written myself, so I could evaluate the outputs against answers I already knew. Added local embedding generation using a Granite model so the entire pipeline could run offline. Then decomposed the notebook into three production microservices: a data ingestion API, a search and generation API, and a unified gateway that runs both.

Having built RAG pipelines with my team, I can tell you that decomposition step is where most of the real engineering decisions surface. Error handling, input validation, service boundaries, modularity. None of that appears in a notebook.

My full repo is here: shaikhq/agentic-rag-db2

The framework, step by step

Here is the process in the order I followed it. It applies to any AI tutorial you want to extend, not just RAG.

1. Run the original first, without touching it. Understand every component and every design choice before you form an opinion about what to change. This is the step many students skip. They skim the tutorial, get the general idea, and immediately start modifying things. That is why their extensions feel shallow. Deep understanding is what lets you see the gaps.

2. Record baseline numbers before changing anything. Retrieval accuracy, latency, response quality, whatever is measurable for the use case. You need a starting point. Without a baseline, any improvement you make later is invisible. This is also one of the most common gaps in tutorials themselves, and closing it alone is a meaningful contribution.

3. Swap the data source with something real to you. Replace the tutorial’s sample data with your own content, your own domain, something you can evaluate honestly. I used a technical article I had written. The pipeline’s responses immediately became more meaningful to read because I already knew what a correct answer looked like.

4. Close one gap with a targeted improvement. Better retrieval logic, a smarter chunking strategy, an evaluation step the original skipped, a production-grade vector store. Pick one improvement at a time, implement it, and measure the difference against your baseline from step 2. That comparison is the substance of your portfolio, not the code alone.

5. Convert the notebook to APIs. This is the step that separates a prototype from something deployable. I split the notebook code into a data ingestion API and a search and generation API, then combined them behind a gateway. It forces decisions about error handling, input validation, and service boundaries that notebook code never surfaces. Most students never get there, which is exactly why it is worth doing.

6. Write the README while the work is fresh. Not a usage guide. A record of what you found in the original tutorial, what you changed, and what the numbers showed before and after. Include a workflow diagram. This is the document an interviewer will ask you to walk them through. Writing it also forces you to articulate what you actually did and why, which is harder than it sounds.

The two assets that most students skip

Once the code is done, produce two more assets from the same work before you close the project. These are the ones that actually get you noticed.

Write a post about what you extended and what you measured. Not a tutorial explaining the technology. Your own account of the extensions you made, what surprised you, and what the numbers showed. A hiring manager can tell the difference between a post written from experience and one assembled from documentation. This kind of writing is harder to produce from AI alone, which is exactly what makes it credible.

Record a short video walking through the repo. Many students applying for internships will not do this. I published two walkthroughs covering the above sample work: Deploying Agentic RAG to Production, Part 1: FastAPI Data Ingestion and Deploying Agentic RAG to Production, Part 2.

A hiring manager who watches three minutes of you explaining your own work has a clearer picture of how you think than one who reads a README alone.

One tutorial. A couple of weeks of deliberate work. Three assets: an extended repo, a written reflection, and a video walkthrough. Each one builds on the same foundation and tells a consistent story about how you think and work.

Why this works

The reason this approach produces better portfolio projects than building from scratch is not that it is easier. It is that it mirrors how engineering work actually happens. In any real job, you will inherit code, understand it, identify what is missing, and improve it. The tutorial gives you something to push against. The gaps give you something real to solve. The baseline numbers give you a way to show that the solution worked.

A portfolio project that shows you can do that is more convincing to a hiring manager than one that shows you can follow a course and build what the instructor built.

If you try this approach, I would be curious to hear which step felt hardest, finding a tutorial worth extending, deciding which gap to close first, or getting the post and video out after the code was done. Those are usually where people get stuck, and they are worth talking through.

When you build an application that uses embedding models - whether for semantic search, retrieval-augmented generation, or any other vector-based workflow - measuring performance is easy to put off. You get the feature working, the embeddings look right, and you move on.

Then someone asks: “How long will it take to vectorize our 50 million documents?”

If you have not done the performance work by that point, the question becomes a problem without a quick answer. I have been there. This post is about doing that work before you need the answer, building a repeatable performance testing approach rather than figuring it out under pressure.

One clarification on scope: this guide focuses specifically on batch embedding generation - sending multiple text inputs to an embedding API in a single call and measuring the throughput. If you are calling an embedding API one input at a time, some of this still applies, but the biggest gains described here come from batching. That context matters as you read.

I will walk through the performance testing approach I have been developing and refining through real experiments against cloud embedding APIs and a self-hosted vLLM endpoint. The work ranged from small single-request tests all the way to running embedding generation over datasets at million-row scale.

Why Embedding APIs Warrant Their Own Performance Testing Approach

Embedding APIs are used differently from text generation APIs, and that difference shapes how you should think about performance.

With a generation API, you typically process one request at a time - a user sends a prompt, the model responds. With an embedding API, you are almost always working in bulk. A product catalog with 10 million descriptions needs all 10 million vectorized. A document corpus gets re-indexed every night. The volume is large, and it often runs as a background job where throughput matters more than single-request response time.

This bulk processing nature means that small configuration choices - like batch size - can have large effects on the total time and cost to complete a job. For example, running with a poorly chosen batch size can cut your throughput significantly compared to an optimized one, and that difference compounds across every run, every day, at every scale. It is not the only thing that matters, but it is one of the most controllable levers you have.

The performance testing approach covered here focuses on four questions:

1. What limits does your embedding API put on each call?

2. At what batch size does throughput peak for your model and data?

3. How does throughput drop when you go past that batch size?

4. What breaks when you run the same job at production scale?

Step 1: Know Your API’s Hard Limits Before You Test Anything

Before running a single test, check the documentation for your embedding provider and answer one question: how does this API limit what you can send in a single call?

There are two common patterns:

Input count limits. Some providers cap the number of text inputs you can include in one API call. Once you exceed that number, the request is rejected. Always verify the current limit in the provider’s documentation, as these can change.

Token budget limits. Other providers do not set an input count limit directly, but cap the total number of tokens across all inputs in a single call. For example, if the limit is 300,000 tokens and your texts average 450 tokens each, that budget translates to roughly 666 inputs per call. The limit is not stated as an input count, but it effectively becomes one based on your data.

These two approaches require different batching logic. With an input count limit, you can set a fixed maximum batch size. With a token budget, your effective batch size depends on the actual token length of your inputs, so you need to calculate it dynamically for each batch rather than hardcoding a number. Getting this wrong - for example, assuming all inputs are short when some are long - can cause unexpected failures partway through a large job.

All of the APIs I tested were REST APIs. The limits below reflect what the REST API accepts per call. These figures come from publicly available documentation at the time of writing. Always check the provider's current docs before using them, as limits can change.

• WatsonX AI — Input count limit. Observed up to 1,000 inputs per call in my own testing. IBM Cloud API Docs

• OpenAI — Token budget. 300,000 tokens total across all inputs per call. OpenAI API Reference

• Azure OpenAI — Input count limit. Up to 2,048 inputs per array per call. Azure OpenAI docs

• AWS Bedrock (Titan) — Single input only. The InvokeModel REST API accepts one input string per call — batch arrays are not supported. AWS docs

• vLLM (self-hosted) — Depends on deployment. The server operator controls limits at startup via --max-num-batched-tokens and --max-num-seqs. In my own testing I was able to send up to 10,000 inputs in a single call. vLLM Engine Arguments

A note on counting tokens before you send them. If your provider uses a token budget, you need a way to estimate token counts on your side before constructing each batch. Calling the API itself to count tokens is expensive and defeats the purpose. A practical alternative is to use an open-source tokenizer locally. For OpenAI-compatible APIs, the tiktoken Python library can count tokens offline at no API cost. For other models, the transformers library from Hugging Face provides tokenizers that run locally. Neither requires a network call, and both run quickly enough to be part of your batching logic at scale.

Step 2: Measure the Right Things - Break Down Where Time Goes

Before you start adjusting batch sizes, set up your measurement code to separate request time into three distinct parts. This is easy to skip, but it is where the most useful diagnostic information comes from.

Payload preparation is the time your code spends building the request - formatting your list of input texts into the JSON structure the API expects. For small batches, this is negligible. For large batches with many long texts, it can add up and become worth optimizing separately.

API wait is the time from when your request leaves your machine to when the response starts coming back. This includes network travel time in both directions, any time the request spends waiting in a queue on the provider’s side, and the time the model actually takes to compute the embeddings. This phase typically takes the most time, and it grows with batch size - though not in a straight line, as you will see in Step 3.

Response parsing is the time your code spends reading the response and extracting the embedding vectors. At larger batch sizes, responses get large. A batch of 1,000 embeddings at 768 dimensions, stored as 32-bit floats, contains about 3 MB of numerical data to read and process. Measuring this separately helps confirm whether response handling is ever a bottleneck.

When you break latency into these three parts, you can see where to focus. If API wait is the dominant cost, then batching strategy is your main lever. If payload prep is surprisingly slow, look at how you are constructing requests. If response parsing takes longer than expected, look at how you are reading and storing results.

Step 3: Find Where Throughput Peaks - and Where It Falls Off

This is the core of the performance testing work. The goal is to find the batch size where you get the most inputs processed per second, and to understand what happens on both sides of that point.

When you plot throughput against batch size, you get a curve with a predictable shape: it rises quickly at first, peaks somewhere in the middle, and then gradually declines. Three things drive that shape:

Why throughput rises as batch size increases. Every API call has a fixed cost you pay regardless of how many inputs are in it: the time to establish the connection, send the request, and receive the response. At batch size 1, that fixed cost is most of your total call time. At batch size 10, you pay the same fixed cost but process 10 inputs instead of 1, so your throughput is roughly 10 times higher. As batch size grows, that fixed cost gets spread across more inputs and throughput climbs.

Why throughput eventually stops climbing. Once the batch is large enough, the fixed overhead is fully spread out and the model’s compute time per input becomes the main cost. Throughput levels off. This is the plateau.

Why throughput can drop with very large batches. Very large batches introduce new costs: larger payloads take longer to send over the network, the server may need to split your batch into smaller internal chunks to process it, and memory pressure on the server side increases response time. The result is that throughput can actually decrease beyond a certain point, even though more data is being sent.

A few patterns worth noting in this kind of result:

Batch sizes 1 and 10 can have nearly the same total call time (both around 0.4 seconds in this example), yet batch 10 delivers about 10 times the throughput. This is the fixed-cost effect at its clearest - you are doing 10 times the work for no additional waiting time.

The provider’s maximum allowed batch size is often not the most efficient one. In this sample, batch 1,000 - the largest - produces the lowest throughput. The API accepting a batch size does not mean that size is the best choice.

The peak is typically a range, not a single number. In the sample above, batch sizes from about 200 to 300 all deliver throughput within 5-10% of the peak. This is useful in practice: if your batches are not always exactly the same size, anything in that range will perform similarly well.

How to run this test. Pick a fixed dataset of text inputs with a consistent average length (matching your real workload as closely as possible). Start with batch size 1 and work up through increasing sizes - for example, 1, 10, 50, 100, 200, 250, 300, 400, 500, up to your provider’s maximum. For each batch size, run at least three separate calls and average the results to reduce the effect of network variation. Include one unmeasured warm-up call at the start of each batch size to avoid counting any startup effects in your measurements.

Step 4: Test Each Model Separately

The batch size that gives the best throughput is not the same for every model. Smaller models compute embeddings faster per input, so they can handle larger batches before hitting the plateau. Larger models take more compute time per input, so the plateau arrives at a lower batch size, and the performance drop from oversized batches tends to be steeper.

When I ran the same batch size test across three models of different sizes - small (around 30M parameters), medium (around 125M), and large (around 500M or more) - the curve shape was similar for all three, but the peak location and the absolute throughput values were different for each. Larger models peaked at a lower batch size and at lower throughput overall.

The takeaway is simple: if you are working with more than one embedding model, run the batch size test for each one independently. Do not carry over the optimal batch size from a small model and assume it applies to a larger one.

To keep the comparison clean, hold everything else constant - the same input texts, the same infrastructure, the same measurement approach. Change only the model. This way, any difference in results can be attributed to the model rather than noise from other variables.

Step 5: Test With Different Input Lengths

Most performance tests use a fixed text length, but real data rarely has perfectly uniform length. The token length of your inputs affects throughput in ways that are worth understanding before you run your tests, and it affects providers differently depending on whether they use input count limits or token budget limits.

A practical way to test this is to prepare separate datasets at a few different average token lengths - for example, a short-text set (around 50 tokens per input, like product titles or search queries), a medium-text set (around 150 to 250 tokens, like short paragraphs), and a long-text set (around 400 to 500 tokens, like full document sections). Run the same batch size test on each.

With short texts, you can fit more inputs within a token budget per call, but the fixed API overhead becomes a larger proportion of your total time per input. With long texts, your effective batch size shrinks if you are working within a token budget. The optimal batch size by input count may be quite different across these three workloads.

One specific edge case worth testing: what happens when an input exceeds the model’s maximum token length? Some APIs truncate the input silently and return an embedding anyway. Others return an error. Others may return something that looks like a valid embedding but reflects only part of the input. The behavior varies by provider and model, and it is worth verifying directly for your setup rather than assuming. If your real data has any inputs that might be unusually long, this test can prevent silent failures in production.

Step 6: Test at Real Scale

Small tests tell you how the API behaves call by call. They do not tell you what happens when you run a job across millions of inputs over several hours. Scale introduces a different set of problems, and many of them only become visible once you are running at that size.

I will not share specific numbers from my own large-scale experiments here, as that work was done in a professional context. But the problems I ran into are common enough to be worth sharing.

The bottleneck often is not the API. When running a long embedding job end to end, the slowest part is not always the embedding API call. Time spent reading inputs from the source system, writing results back to storage, and handling coordination between parallel workers can all add up. Measure the full pipeline, not just the API call times, to find where the real time goes.

Write volume adds up at scale. Embedding vectors are not small. At 768 dimensions stored as 32-bit floats, each embedding is exactly 3 KB. Writing 1 million embeddings means writing just under 3 GB of data. If you are writing to a database or storage system that has transaction or write limits, those limits can become a bottleneck or cause failures mid-job. Check the limits of your write destination before starting a large run, not after hitting an error.

Concurrent API calls and token rate limits interact. Running multiple parallel workers to call the embedding API at the same time can speed things up significantly. But if the provider has a rate limit measured in tokens per minute, the token usage from all your parallel workers adds up together. It is easy to hit a rate limit mid-job when running in parallel even if no single worker would hit it on its own. Test your parallel configuration against the provider’s rate limits before committing to it at full scale.

Connection pool limits surface late. If your parallel workers each hold a database connection for writing results, the total number of concurrent connections can exceed your connection pool limit. This type of failure often does not show up until you are partway through a large job. Test with enough scale to expose it before running the full job.

Build resume capability before you need it. A job processing a million inputs that fails at 80% completion should be able to pick up where it left off, not start over. This is straightforward to build before you start a large run and very difficult to add after you have already failed and need to rerun. Track which inputs have been successfully processed and skip them on restart.

Step 7: Run Multiple Iterations and Look at Consistency

A single measurement tells you what happened once. API performance varies based on server load on the provider’s side, network conditions between you and the provider, and the time of day the call is made. A result from one run may not reflect what you will see reliably in practice.

Running each batch size configuration at least three times and looking at the spread of results gives a more reliable picture. If the three results are close together, you can be more confident in the average. If one run is significantly different from the others, that variance is itself useful information - it tells you that the performance at that configuration is less predictable, which matters for planning.

In my own tests, mid-range batch sizes tended to show more variation between runs than very small or very large ones. The practical lesson: when planning how much headroom to build into a performance estimate for a production job, do not base it only on the best case from a small number of test runs. Look at the range of results and plan for something closer to the typical case, not the best one.

Putting It Together: A Testing Checklist

Here is a checklist that pulls together the steps above. Use it as a starting point and adjust it to your situation.

Before you start testing:

• Find the current documented limits for your embedding API (input count or token budget).

• Determine whether you need static batch sizes (input count limit) or dynamic batch sizing (token budget).

• Set up local token counting so you can estimate batch sizes without API calls (e.g., tiktoken for OpenAI-compatible APIs).

• Instrument your code to separately measure payload preparation time, API wait time, and response parsing time.

• Prepare representative test datasets at short, medium, and long average token lengths.

Running the batch size test:

• Test batch sizes from 1 up to the provider’s maximum, using regular intervals.

• Run at least three iterations for each batch size.

• Use one unmeasured warm-up call at the start of each batch size series.

• Change only one variable at a time across test runs.

• Repeat the test for each embedding model you plan to use.

Edge cases to verify:

• What happens when an input exceeds the model’s maximum token length?

• What happens when you approach or exceed the provider’s rate limit?

• How does performance vary when input lengths within a single batch are uneven?

Scaling to production volume:

• Test the full pipeline end to end, not just the API calls

• Check write volume and storage limits for your target system at scale

• Test parallel worker configuration against the provider’s rate limits

• Verify connection pool limits under your expected concurrency level

• Build and test checkpoint and resume functionality before running at full scale

• Run at an intermediate scale (for example, 1% or 10% of your target volume) before committing to the full job

Closing Thought

The biggest performance gain in embedding batch workloads does not require new hardware, a faster model, or a different provider. It comes from batching correctly.

Two API calls that take the same time can have very different throughput — one sending a single input, the other sending a well-sized batch. The larger batch processes many more inputs for the same cost. At the scale embedding pipelines typically run, that difference translates directly into hours of processing time and real infrastructure cost.

The numbers will look different for every model, workload, and provider. The method for finding them does not change.

If you have run similar experiments or found different patterns, I would be curious to hear about it in the comments.

All experiments were run from my personal machine calling external APIs. Any numbers shown are illustrative samples only - they demonstrate the shape of results, not actual measurements from any specific provider. These are personal observations and do not represent my employer or any official benchmark.

2025 has been my most active year as a technical speaker in my 20-year IBM career. I traveled to seven cities across four continents, delivered 16 conference sessions totaling 25.5 hours, and reached over 1,000 in-person attendees - plus many more through webinars and internal sessions.

By the Numbers

6 conferences | 7 cities | 4 continents | 25.5 hours | 1,000+ in-person attendees

My Framework for Preparing Technical Talks

Here’s the structure I use:

One Thread, One Theme. Every slide supports a single theme. No distractions.

Own Your Content. I create all my slides from scratch. No borrowed decks. This means I know exactly what each slide says and can talk about it naturally.

Revise Until It’s Clean. For each slide, I ask: What is the key message here? I keep editing until every extra word and image is gone.

AI as My Design Partner. Last few years and much part of this year, I used ChatGPT for brainstorming and designing slide visuals. Now, my favorite tool for planning conference slides is Claude. For image editing, I use Canva.

Start with the Title. I open every talk by explaining each word in the title and telling the audience what they will learn. This sets clear expectations from the start.

Live Demos Only. No pre-recorded demos. I build and run demos myself, live on stage. This engages the audience. Also, when I built a demo, I did’t need to memorize - I just explained what I built.

Keep the Audience Engaged. I ask questions, add light humor, and repeat key points throughout the talk.

What Changed

At the start of this year, conference talks felt uncomfortable. With each session, it got easier. Now I look forward to larger audiences - the bigger, the better. I enjoy making eye contact, reading the room, answering questions mid-talk. The best moment is when I explain something complex and see it click. That’s the joy.

There is not one algorithm that’s going to solve all your problems. Think of it like wanting to unscrew something. You have a toolbox with different sized screwdriver bits, and you have to find the right one for the screw in front of you. Machine learning works the same way. You have to see what kind of data you are dealing with and the learning task at hand, and then find the right tool, the right algorithm, the right AI technique.

Some data sits neatly in rows and columns. Other data is a mess of text, images, and files. Each needs different tools to unlock its value.

In this article, I’ll walk you through the main types of machine learning models and learning tasks. This is not meant to be exhaustive. I want to cover the more representative models so you have a solid foundation.

First Things First: Structured vs. Unstructured Data

Before we dive into the model types, we need to understand the two broad categories of data we deal with: structured data and unstructured data.

Structured data is the kind of data stored in relational schemas, it’s not free form. It comes with a structured format. You have a table with a list of columns, and each column has some attribute values. Maybe an employee table with employee name, join date, department, and salary. Everything is fixed with columns and rows. You can make a lot of assumptions about the data structure, and you have much more predictable characteristics because it’s a fixed structure.

I find this analogy useful: think of going to a library where all the books are organized on shelves. If you’re looking for a specific book, you ask the librarian, and they just look up and tell you exactly which shelf you can find it on. Everything is very neatly organized.

Unstructured data, on the other hand, is like a very messy situation, maybe people who just returned all the books have left them haphazardly in a pile. You don’t know where you can find anything. Unstructured data includes things like free-form text, images, audio, and video.

How Machine Learning Models Learn

Before we look at specific model types, you need to understand the two fundamental ways models learn from data: supervised and unsupervised learning.

Supervised Learning

In supervised learning, you provide the model with examples that include the correct answers. You need two types of content in your data.

Features: A set of attributes that describe something. For example, if you want to predict customer churn, you might collect which products customers are using, how many licenses they have, how many times they called support.

Labels/Targets: The outcome, what actually happened. When the time came for renewal, did this customer renew or not?

You collect input and output combinations for many examples, then give this to a learning algorithm. The algorithm figures out how to map inputs to outputs.

A good way to think about it: remember equations like y = constant + a₁x₁ + a₂x₂?

The machine learning algorithm learns this kind of mapping from your data.

Once trained, for future events where you only know the inputs, you plug them into the model and it predicts the output. This process is called scoring or inferencing.

Self-Supervised / Unsupervised Learning

In self-supervised learning, you don’t provide the correct answers. You give the model data and let it find patterns or groupings on its own. You only have features, no labels.

Clustering is a common self-supervised technique. Algorithms like k-means clustering group similar data points together without being told what the groups should be. For example, you might have customer data and want to discover natural segments. The algorithm analyzes the features and groups similar customers together. Marketing teams often use this to identify customer personas they didn’t know existed.

Predictive Models

Predictive models use supervised learning to give you predictions about future outcomes. Classification, regression, and time series forecasting all require labeled training data.

Classification

For structured data, one kind of prediction you can make is the category or class of something. A classic and familiar example: you have an email, and you want to classify it into one of two categories, valid email so it can get to your inbox, or spam. That’s what Google has done with their spam folder and inbox. They have a spam classifier that looks at each email and a set of characteristics (the features), and based on this, classifies each email as spam or valid (the label).

Another example is fraud detection. You go to your grocery store and swipe your credit card to pay. Behind the scene, it goes to the credit card authorization system, which quickly checks whether this transaction is valid or fraudulent.

Why is this called classification? Because this model is making some conclusion about the input, a structured row of data, and saying that this record belongs to one of a finite set of classes. It’s either valid or fraudulent. You can have multiple classes, but it’s always a finite set of possible outcomes. That’s called classification.

Regression

The other kind of predictive model you can build with structured data is a regression model. A common example is predicting house prices. You want to predict what a house will sell for based on how many bedrooms it has, the square footage, and other features.

You can train a machine learning model with houses that were sold in a neighborhood, their features and the price at which each was sold. You give all these to a learning algorithm, and then you have a learned regression model. Next time a house that has yet to be sold comes along, you just collect its description and attributes, give it to this model, and the model gives you a numerical value.

Unlike classification, where you predict from a finite set of values, here your model is predicting a number that could vary widely. The number is not coming from a fixed set of values. This kind of model is called a regression model.

Time Series Forecasting

Another type of predictive model is time series forecasting. You might want to predict stock prices based on events and factors, or weather forecasting. There’s seasonality, there are variables and factors you’re collecting over time, what happened, what are some of the things that occurred. Based on this, you’re predicting future values. Time plays a role in these models, that’s what makes them different from standard regression.

Predictive Models for Unstructured Data

For unstructured data, similar predictive models exist, but they work with different types of input.

Image classification: You might want to classify pictures, give a machine learning model images and have it learn to predict whether a picture is of a cat, a dog, or some other animal.

Object detection: A self-driving car is constantly taking pictures of what’s in front of it and around it, identifying objects and their locations. It’s not sufficient to just know that there’s an object in front of the car, it also needs to know the location of that object.

Text classification and sentiment analysis: Maybe you have product reviews and you want to predict if the customer is happy or not. You can analyze their text comment and classify whether they’re a happy customer or not.

Generative Models

This is the category that’s been very popular in recent years, the large language models (LLMs), such as ChatGPT, and all these generative AI models. This kind of model is not just predicting based on an input. It takes a prompt and then creates content for you.

Text generation: You can give a prompt and all of a sudden get a long, AI-generated text. It can generate articles, very human-like responses, and more.

Language translation: You speak in one language, and it gets translated into another. Services like Google Translate use neural machine translation models for this.

Code generation: You can generate code using AI. IBM Project Bob, Claude, Gemini, ChatGPT, there’s a lot of code generation happening with these tools.

Image generation: These models can also generate images based on text prompts.

What Makes Large Language Models “Large”?

Two things make them large: massive training data from sources like Wikipedia, books, and web pages, and billions of parameters (the internal variables the model learns during training). GPT-3, for example, has 175 billion parameters. This scale allows LLMs to capture intricate patterns in language.

Embedding Models

Embedding models are a different kind of AI model. They don’t produce something immediately consumable like a prediction or generated text. Instead, they give you an intermediate representation called a vector, which is essentially a list of numbers that captures the meaning or characteristics of your data.

For example, each of us has certain background attributes, the field we work in, years in our career, our university education. If we want to learn which of us are closer to each other based on our technical or academic background, we could use some kind of embedding model to take a text description of who we are, and based on this, it gives a vector representation. Those of us who are similar are expected to have similar vector representations.

Similarly, you can have a list of documents or articles, vectorize them, and see which articles are similar to each other. Maybe a subset are food blogs and others are technical blogs, you expect articles in the food blogs to have similar vector representations. You can do the same with images.

With multimodal embedding, your content is a mix of text and image, and you can vectorize it to do semantic search, not just plain text-based matching, but vector-based semantic search.

Choosing the Right Model for Your Problem

When you’re approaching a problem, ask yourself three questions:

What kind of data do you have? Structured data (tables with rows and columns) works well with traditional machine learning. Unstructured data (text, images, audio, video) typically requires deep learning approaches.

Do you have labeled data? If you have historical data with known outcomes, you can use supervised learning (classification, regression). If not, consider unsupervised approaches like clustering or embedding models.

What kind of output do you need? This is often the deciding factor. Use the table below as a quick reference.

The temptation today is to reach for LLMs for everything. But LLMs are just one drawer in the toolbox. For structured, tabular data, traditional machine learning algorithms are often faster, simpler, and more accurate. The skill isn’t knowing the most powerful tool. It’s knowing which tool fits the problem in front of you.

Many companies don’t have a job title for the person who owns AI features end-to-end. They have ML engineers, data scientists, product managers. But the role that bridges all of them? That’s often unnamed.

AI Product Architect is a term I’m using to describe this role. It’s not a standard industry title. It’s how I make sense of a set of responsibilities that don’t fit neatly into existing job categories.

A software architect designs systems. An AI Product Architect, as I define it, does the same, but for AI features inside products. Same technical leadership. Different technical domain.

If you’re a senior engineer or software architect with years of experience shipping software, you don’t start over to move into this role. You add AI to what you already know: systems thinking, product delivery, technical leadership. The core skills transfer.

In my role as AI Architect at IBM Db2, I technically lead the development of AI features for the product. Db2 has architects for different areas such as compiler, security, and data movement. My responsibility is planning, designing, and implementing AI features in Db2 products.

That’s what an AI Product Architect does, at least in my experience: technically owns AI features end-to-end within a larger product.

What the Role Looks Like in Practice

In 2025, I was technically responsible for three major AI initiatives in Db2. I owned and architected the end-to-end implementation of each, from concept to release: planning, designing, implementing, testing, releasing, promoting, and adopting.

The three projects:

1. Native vector data type: introducing vector storage and similarity search functions inside Db2

2. Tooling integration: extending existing Db2 utilities to recognize and work with the new vector type

3. LLM API integration: connecting external language model APIs to Db2, making them callable via SQL functions

These weren’t research experiments. Each one shipped in a product release to enterprise customers running real workloads.

Each project followed the same pattern. Let me walk you through what that looks like.

Phase 1: Proposal and Validation

Every initiative starts with identifying a market opportunity. A problem worth solving. That problem may translate into multiple functions, new capabilities, new use cases for customers. But it starts with one question: what’s worth building?

Market Research

Analyst reports give you the landscape: Gartner Magic Quadrant, Forrester Wave, industry blogs from major conferences. Competitor announcements show you where the market is moving.

But reports tell you what’s popular, not what’s valuable. That requires talking to customers.

When we proposed vector support for Db2, I studied how AI practitioners were using vector types and similarity search: what problems these capabilities solved, what use cases they enabled.

Customer Validation

I spoke with customer representatives, internal data scientists, and machine learning practitioners. Many of them work directly with real customers. I also reached out to actual customers already building these kinds of use cases.

The goal isn’t to confirm your idea is good. It’s to find out where it’s wrong. What are customers already doing? What’s broken? Would they pay for this?

This back-and-forth sharpens the idea. You validate assumptions, crystallize the concept into specific problem categories, and identify where real demand exists.

Quick Prototyping

Before presenting to executives, when possible, I build low-fidelity prototypes using existing features. Not performant. Not production-ready. But tangible.

For the vector proposal, I implemented vector storage and similarity search using existing Db2 data types and user-defined functions. It was slow, but it demonstrated the experience and capability we were proposing.

A working prototype changes the conversation. Stakeholders stop debating whether it’s possible and start debating whether it’s worth it.

Building the Proposal

The initial proposal is still high level: a suggested set of new features, the problems they solve, evidence of customer interest, and the industry outlook.

I never go straight to executives. First, I validate with peer architects, including some more senior than me, and make them co-authors on the proposal. We go back and forth, refine the thinking, pressure-test the idea. Then we present to executives together.

Phase 2: Solution Externals

Once the proposal has stakeholder support, the next step is writing solution externals.

In software engineering terms, this is a software requirements specification. But it reads like a user manual, written before any code exists. You define how real users will interact with the proposed features: the interface, the syntax, the experience.

The goal is precision without code. By the time engineering starts, everyone (architects, developers, stakeholders) should have the same understanding of what the feature will do.

For each feature, I document:

• What it does: a clear description of the function or capability

• How it looks: syntax, parameters, inputs, outputs

• Processing: what the function does with those inputs to produce the output

• Restrictions: what this feature will not cover in this phase

• User scenarios: sample use cases with realistic values

For example, when I wrote the solution external for the vector functions in Db2, I documented each function. For vector distance calculations, that meant defining parameters, expected inputs, outputs, processing logic, and restrictions. Everything was documented textually and visually, without writing a single line of implementation code. The visual above shows what this looks like in practice.

This solution external becomes the foundation for the actual user documentation later.

This document saves months of rework. Ambiguity in requirements becomes bugs in production. The solution external eliminates ambiguity before a single line of code is written.

At this point, the feature exists on paper. Everyone agrees on what we’re building.

Now comes the hard part: making it real.

Next post: Design and Implementation.

In 2014, I was at IBM Littleton Lab in Massachusetts for a training on Watson. I was learning how to build a Q&A system for IBM customer support. I had spent nine years building Java applications. I knew nothing about AI beyond a graduate course on data mining and machine learning. Watching Watson generate answers, I was amazed. The technology was just starting out, but I could see where things were heading.

That week was my first exposure to AI. Two years later, I made the full transition.

My fear was stepping into territory with no clear path to production. But I had always tried new technology early, even in my Java career. I knew that if I stayed comfortable, I would be fine for a few years. Eventually, my growth would stop. I was not willing to let that happen.

The Java Years

For 11 years, my world was enterprise Java. I built web applications and web services, deployed them to production, and troubleshot server failures. I developed a search engine from scratch. I set up a wiki for a large organization. My days were writing code, testing, deploying, and fixing what broke.

I was not just an individual contributor. I led a team of Java developers, system administrators, and DBAs. I was hands-on with code and responsible for technical leadership and architecture. Doing both shaped how I think about building systems and leading people.

It was good work. I was good at it. Management was happy, and I received top ratings.

But at some point, the growth slowed down. The work became routine: add features to the front end, crawl more data sources into the search engine, repeat. I was not feeling challenged. I was not innovating. And I knew it.

The Turning Point

By 2016, “Data Science” and “Cognitive Computing” were impossible to ignore. AI had carried a negative reputation from its earlier failures, but deep learning was changing that. Watson had won Jeopardy. Kaggle competitions were producing impressive results. Neural networks were going through a transformation with modern architectures.

I did not understand most of it. But I could see AI was beginning to matter. It was the earliest stage of adoption. Most companies were hesitant to explore AI. Success stories were rare.

Then, at the end of September 2016, before a three-week vacation, my manager Chad Marston pinged me. He said, “Shaikh, we want to begin working in Cognitive Computing. I want you to lead that team.”

I would be building a team of 8 to 10 developers, and none of us had any background in AI. We all came from Java and web development.

I went on vacation, but I kept thinking about it. When I came back, I was already exploring what I could learn and where I could get hands-on with this technology.

Chad sent me to the IBM Watson Developer Conference in San Francisco later that year. I attended hands-on labs where I played with Watson APIs. I saw demos. I listened to talks from Ginni Rometty, IBM’s CEO, and other experts. I had seen Watson before, but this was different. I saw more of what AI could do in practice. I came back with confidence and a clearer picture of what we could build.

The Hard Part

Leading 8 to 10 people while learning myself was not easy. Online learning platforms had plenty of courses, still overwhelming. Where do you start? What is worth your time? What will matter when you try to apply it?

The first thing I did was set up regular check-ins with my mentors at IBM: Jean-Francois Puget, Jorge Castañón, and Seth Dobrin. I got their guidance, filtered courses based on their recommendations, put together a learning path, and shared it with the team.

Even with structure, the learning curve was steep. The concept I struggled with longest was evaluation metrics. Accuracy was easy. Precision and recall were harder, and harder still to explain to management. The harder problem was mapping model metrics to business metrics. A model can perform well on test data, but how do you know it is helping the business?

I have to admit: our first project failed. We were asked to predict customer satisfaction scores from support interactions. The model was accurate in training and testing. We deployed it. But we had no way to monitor business success. We failed quickly and learned from it.

Many questions from team members, I did not know the answers. I did not make things up. I admitted I did not know, then went and learned and came back. That honesty became part of how we operated.

The learning was hard for the whole team. We set up a weekly session where we watched videos together and discussed what we learned. Often, one team member would pre-read the material and present it. This kept us progressing together.

What Helped Us Learn

I tried many courses across Coursera, edX, Udemy, and DataCamp. A few stood out:

Andrew Ng’s Machine Learning course gave me a foundation in core algorithms: decision trees, linear and logistic regression, support vector machines, and neural networks. It was taught in Octave, not Python, but the concepts were what mattered.

Python for Data Science from DataCamp gave me fluency in Python once I had the conceptual foundation.

Two books were essential: An Introduction to Statistical Learning for statistical foundations, and Python Machine Learning by Sebastian Raschka for bridging theory to implementation.

But the most important learning did not come from courses. It came from applying concepts to real projects, getting stuck, and consulting mentors. That cycle of building, struggling, and asking for help was the core of my growth.

Where I Am Now

My AI journey started in 2014 with that Watson project. In fall 2016, I made the full transition, initially working with Watson’s NLP capabilities: sentiment analysis, question answering, and similar APIs. I became the lead machine learning engineer for IBM Analytics and held that role for three years. In 2019, I moved into my current role: AI Architect for IBM Db2.

My work today goes beyond building models. It has two dimensions. The first is improving database operations using AI, for example, applying machine learning to improve memory estimation for queries. The second is bringing AI infrastructure into the database itself, so that database administrators can run AI workloads directly within Db2.

Building AI into a database is different from building standalone models. The model must be small, fast, and light on resources. No human monitors these models, so they need self-feedback mechanisms. These were constraints I never faced before.

One project shows what my role looks like now: bringing vector support to Db2. At the beginning of 2025, Db2 had no vector capabilities. I owned this project end to end: planning, designing, architecting, leading the team, and contributing to development. We implemented vector type support, vector similarity search, and LLM API integration callable via SQL. We released it, then I created educational materials, gave talks, wrote blog posts, and promoted the work.

That cycle, from zero to shipped product to education to promotion, is what makes this role rewarding. I am challenged every day. I am not just building models. I am thinking about how AI fits into an enterprise database platform and how to make it useful for production workloads. I am more satisfied now than I ever was in my Java years.

What I Would Tell Someone Making This Transition

Find a mentor early. If you want to become an AI architect, find someone in that role. Get their guidance on what to learn and what to skip. This will save you months.

Your experience is not a liability. I did not abandon my 11 years of Java. I built on it. Production systems, working with teams, managing projects: all of that transferred. The AI skills were an addition, not a replacement.

Your learning approach should change over time. Early on, I took courses end to end. That was useful for building foundations. But after the fundamentals, I changed my approach. Now I start with the project, then learn only what I need to apply. In 2021, I took an entire specialization on cloud deployment of AI. I never applied most of it, and those skills are gone. Project-driven learning is more effective once you have the foundations.

Model accuracy is not the end goal. I used to think a highly accurate model was all that mattered. I was wrong. What matters is identifying the right business metric and tying it to your model’s performance. A model can look great in testing but have no impact if you have not made that connection.

User adoption is the part nobody warns you about. Even after you get the model and metrics right, users fall back to old routines. Deploying AI is not just a technical problem. It requires rolling it out slowly and getting buy-in. Skip this, and your model sits unused.

Questions to Ask Yourself

1. Who do you know who is already an AI architect? Write down three names. This week, reach out to one of them with a single question about how they got there. If you do not have someone in mind and think I could help, reach out to me.

2. What skills from your current role would transfer to an AI architect role? List three.

3. What have you learned in the past two years that you never applied? Going forward, learn less that way.

4. What problem in your current work could benefit from AI? Name the problem and the outcome you want.

5. If you became an AI architect today, what is the first project you would want to work on?

Your Turn

If you are a senior engineer feeling that growth has stopped, the path I took is available to you. You do not need to go back to school. You do not need to start over as a junior. You need a structured path, the right guidance, and willingness to be uncomfortable for a while.

If this post was useful, follow me. I will publish more on lessons from my transition, with steps you can apply.

If you are considering a move to an AI architect role, I would like to hear from you. What are your struggles? What is holding you back? Share in the comments. Your questions will shape what I write next.

One thing troubled me these past days: I couldn’t get llama.cpp to install on macOS. I had set it up successfully on Linux using Python 3.13. The same steps failed on my Mac. They kept failing

llama.cpp is a tool written in C++ that allows you to run large language models on consumer hardware.. It supports various model types and provides GPU acceleration on Apple Silicon through Metal. For developers and IT professionals, this means you can run AI models locally without relying on cloud services or paying for API calls.

I searched everywhere, ChatGPT, Claude, and Google. Nothing worked. I went back to the official llama.cpp documentation. There I found the answer: llama.cpp doesn’t work well with Python 3.13 on macOS.

After more testing, I arrived at the right steps for macOS. I run an M2 Mac, but this should work on other Apple Silicon chips, thoughI have not tested those variants.

I wanted to verify everything worked. I downloaded two models from Hugging Face in GGUF format: a text generation model and a text embedding model. I created two minimalistic Python scripts, one for text generation, another for embedding generation.

Both scripts ran fine. The embedding script generated embeddings for input text. The text generation script responded correctly to prompts.

To be certain, I documented all steps. Then I created a fresh directory and repeated everything. It worked again.

Here are my steps, the models I used, and the test scripts. I hope this helps those who want to set up language models locally on macOS for prototyping or development.

Complete Setup Process

Step 1: Environment Setup

# Install Python 3.12 using uv (if not already installed)
uv python install 3.12

# Check Python version
python3.12 --version

# Create virtual environment with Python 3.12
uv venv --python=$(which python3.12)

# Upgrade pip
uv pip install --upgrade pip

# Install required packages with Metal support
uv pip install \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/metal \
  llama-cpp-python \
  “langchain>=0.2” “langchain-community>=0.2”

Step 2: Download Models

# Download Nomic embedding model
wget https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF/resolve/main/nomic-embed-text-v1.5.Q8_0.gguf

# Download Qwen text generation model
wget -O qwen2.5-3b-instruct-q4_k_m.gguf \
  https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf

Step 3: Create Test Scripts

Text Generation (langchain_qwen_generation.py):

from langchain_community.llms import LlamaCpp

llm = LlamaCpp(
    model_path=”qwen2.5-3b-instruct-q4_k_m.gguf”,
    n_ctx=2048,
    temperature=0.7,
    max_tokens=100,
    verbose=False
)

response = llm.invoke(”What is the capital of France?”)
print(response)

Embedding Generation (langchain_nomic_embeddings.py):

from langchain_community.embeddings import LlamaCppEmbeddings

embeddings = LlamaCppEmbeddings(
    model_path=”nomic-embed-text-v1.5.Q8_0.gguf”,
    n_ctx=512,
    verbose=False
)

query = embeddings.embed_query(”search_query: What is AI?”)
print(f”Dimensions: {len(query)}”)
print(f”First 5: {query[:5]}”)

Step 4: Run Test Scripts

# Test text generation
uv run langchain_qwen_generation.py

# Test embeddings
uv run langchain_nomic_embeddings.py

What I Learned

For now, use Python 3.12, not 3.13, on macOS. The installation command includes Metal support for Apple Silicon. Both test scripts verify that everything works.

This setup lets you run language models locally without cloud costs. Good for prototyping and development.

This part focuses on building the Search API for Agentic RAG using FastAPI, designed to orchestrate an AI agent powered by LangGraph. The agent coordinates multiple tools to retrieve the best possible answers, forming the core of an agentic RAG pipeline. I also unified this Search API with my previous Ingestion API under a single FastAPI gateway - creating a clean, production-ready interface.

You’ll find all the code and setup instructions in the GitHub repo:
🔗 https://github.com/shaikhq/agentic-rag-db2

To go with the 2nd installment of the tutorial series, I’ve recorded two short videos:

1. Implementation Walkthrough – A guided tour of the repo and how the APIs are structured

2. Live Demo – A hands-on demonstration of calling the APIs using simple curl commands

Both videos are included below.

This builds directly on the first part of the series, where I focused on turning the document ingestion logic into a standalone API. If you missed that, you can find it in the GitHub repo as well.

First Video: Deploy Agentic RAG to Production, Part 2 - Build the Search API End Point - Code Walkthrough

Second Video: Agentic RAG Search and Gateway APIs Demo

In the following video, I run a demo of the search and the gateway APIs using simple curl commands across three scenarios:

1. Ingesting new knowledge into the vector store using the ingestion API

2. Querying the Search API to retrieve the best possible answer through the LangGraph agent

3. Clearing the vector store using the cleanup API, giving the system a clean slate

These APIs form the foundation for building an Agentic RAG application on top of the ingestion and retrieval pipeline.

If you have missed the first installment of my deploying Agentic RAG to production tutorial series, here’s the link:

Deploying an AI Agent in Production: FastAPI Data Ingestion