Arham started his journey with ML fundamentals. Linear regression, the tree models, then deep learning, CNNs, LSTMs, transformers, and RAG. Then he looked at where the field is going, and he told me:
“I spent like two years of my life completely learning each and every mathematical formulas behind every neural network... But coming to the future, what I could see is like people are just prompting out things. I was in the doubt.”
His question was simple. Everything is evolving so fast. What should a new grad focus on? Go deep on one topic, or have knowledge of every topic?
We recorded the full session. It runs 12:56.
My answer, to Arham and to any software engineering or computer science new grad who wants to build a career in AI: do not feel lost on your effort in learning core machine learning techniques. Those are not gone. I like to see this in three layers.
Layer 1: The AI/ML foundation
This are the learning algorithms, how machine learns, and how to evaluate them. I would strongly encourage you not to skip that foundation.
Even though you could easily generate a classification model or a regression model with a prompt, understand those foundations. Have a strong intuition of how different algorithms work, how boosting trees like XGBoost work. Have this mental model, this intuition, and know how to evaluate them.
The ML evaluation part is the one I find very relevant even with current AI evals. How we evaluate classification models. Confusion matrix, recall, precision. How we construct ground truth. You will find those fundamentals everywhere, even in all the AI evals we keep hearing about and in RAG evaluation. Some of these are coined using different terms, maybe a golden eval set. But you keep seeing the old discipline of how ML systems are evaluated. It is very much part of the latest AI system evaluation.
Layer 2: Application architecture patterns
Keep watching the application architecture patterns. RAG is one. Agentic systems are another. There will be more.
You may not need to be the expert on everything, but try to have a high-level understanding of each. Then go deep on the one relevant to your work. If you are working with RAG, try to see how to make RAG work, how to build a RAG system that works in production, and how to evaluate it. Try to understand all aspects of it.
Soon something else will come up, some other architecture pattern. You will see many of the things you learned are relatable.
Layer 3: Agentic coding
Use coding agents for building high quality software. It is not just coming up with a prompt and generating thousands of lines of code. It is bringing your knowledge of software engineering, then iteratively planning, generating code, and doing verification.
Code generation is now very cheap. But producing high quality code with a coding agent is really tough.
Practitioners are putting together processes for this. Loop engineering was introduced several months ago. There has been graph engineering. Software factory is another one. Keep using them and trying them. Learning and mastering this software engineering process, using AI coding agents effectively, is very important.
Whatever you are doing, do it in the best possible way. Quality is non-negotiable. There is no quick fix that skips the quality part.
Compared to those who are jumping straight to building AI applications without foundations in AI or machine learning and software engineering, you will be ahead of them. Do not skip those foundations.
Write it in your own words
I gave Arham one more piece of advice. Keep writing about the core ML concepts you have learned. Not with an LLM prompt. You can use it for polishing your draft. But share your own understanding in writing. Once you have this first pass writing, you can always edit and proofread using an LLM. Bring your own understanding and share this knowledge. That is what we need more of.
Where we started: is RAG still needed?
Our conversation actually started with RAG. Arham had seen a new model come into the market with one trillion parameters. He asked:
“Do you think RAG will still be required or we can just directly dump the whole documents to the LLM and it can be able to get the answers for us?”
Dumping a very large document to the LLM has some limitations. When you are asking a question, the entire document is not the answer. If your document has 500 pages, maybe some paragraph or some part of it is where the answer is. LLM still need to do some search and find which part of that large document has your answer.
The first problem is cost. If you are going to a commercial LLM running on the cloud, you will be paying for a lot of tokens unnecessarily. There are some caching capabilities as well, but there will still be cost.
The second is noise. You are giving the LLM a lot more information than what it needs. Say it is a book of ten chapters, and your answer is in one chapter, in some paragraphs. It is information overload. The LLM might be able to find the right context, but it is going to require some computation, and it will also get noise.
The cleaner approach is a pipeline that extracts the right context and gives it to the LLM. You are not giving the noise. You are giving the signal.
RAG is still one of the high value LLM use cases out there. LLMs have frozen knowledge. Whatever they learned during training is not updated. RAG is an architecture fix to bring in this knowledge, a mechanism to ground the LLM on a company’s own knowledge.
We are already seeing a lot of progress is improving the RAG workflow:
Hybrid search. Combining lexical search with vector search.
Multi-vector representations. A standard vectorization turns a paragraph of, say, 100 words or tokens into a single numeric representation. In that process you lose a lot of the low-level details. A multi-vector representation looks into all the tokens and keeps one vector for each token. It is more granular, and computationally more expensive, but it is a richer representation.
Rerankers. The reranker is stage two. The first stage is retrieval with vector search and keyword search, which gives you broader search results. You pass those to a reranker model, which does further post-processing and gives you a more accurate answer.
A lot of improvements are happening, and many companies will continue to use RAG.
Watch the session
The video at the top of this post is the full mentoring session with Arham. It covers everything here, plus the diagrams for hybrid search, multi-vector representations and rerankers.
Watch it on YouTube: