I finished creating an evaluation workbench for my AI search app yesterday.
After building this hybrid search app, I initially used an AI coding agent to generate synthetic test queries and evaluate the results. I was not happy with that approach. Using an LLM to generate test queries, and then using the LLM to judge the results, felt like grading my own homework. So, I built this evaluation workbench.
In information retrieval, the standard practice is to use queries from real human users to evaluate search quality, and to have those same users judge the relevancy of the results.
In my evaluation workbench, I can construct a set of test queries, and collect relevancy judgments from human users.
Here’s how it works:
1. I click a button to start a new test query set and give it a name.
2. I type a query into the test query set.
3. The workbench runs searches against the lexical index and the vector index, pulling the top 10 results from each.
4. It combines both lists, removes duplicates, and discards the relevancy rankings from the individual indices.
5. I go through that combined list of results.
6. I give my own relevancy judgment on each result.
7. I use a three-point relevancy scale: Highly Relevant, Relevant, or Not Relevant.
At the end of this process, for each test query I have 15 to 20 result documents, and for each document I have a relevancy judgment. I repeat this process for whatever number of queries I want in a test set. Initially I am aiming for 15 to 20 test queries per set.
Once I have a test set, I go to the Evaluate tab, pick it from a dropdown, and run the evaluation.
Using this evaluation framework, I can now create my own yardstick for evaluating the performance of my app, not using synthetic queries coming from an LLM and the relevancy judgment decided by an LLM.
The accompanying video walks you through the workbench UI, creating a sample test query, and assessing the relevancy of the result documents.
If you are building AI search (RAG, vector search, hybrid search), how are you evaluating the quality of your retriever? Are you using synthetic queries from an LLM, or something else?
