Ahmed built a multi-class classification model that predicts his sleep quality for a given night. Three class labels: good, okay, bad. Thirty-one days of data on his sleep. He also did feature engineering, and a few things in it show good thinking.
He recognized that 31 examples is a very small dataset. With a traditional 80/20 train-test split, you train on about 25 examples and test on six. That is not that good. So I asked him how many folds of cross-validation he used. That question opened up the rest of our session.
The mentoring session
This is a recorded mentoring session with Ahmed Barakat, a 3rd year student of AI at the Durham College in Ontario. We went through the evaluation of his model: separating features from the label, why a train/test split falls short at 31 examples, cross-validation step by step, simplifying to binary classification, the metrics to understand (precision, recall, F1 and the confusion matrix), and tracking experiments. We close on why I asked him not to rush to the next learning topic.
A good instinct: separating features from the label
He used the data from each day. The features come from his activities during the day, and the label, whether he had a good night of sleep or not, is an event that happened later.
I am not sure if he consciously made this decision, but it is a good practice to have this time series division. In the timeline of events, his sleep is the very last event that happened. He has a series of activities, so his features are happening earlier in the timeline.
One thing I have seen people do, and I made those mistakes earlier myself: say I am predicting my energy level during the day, and then I include some features from something I am doing at the very end of the day. That means I am predicting something based on features that may not be available at that time, or that may be happening at the same time.
Signal leakage
This is called signal leakage. Signal leakage means you cannot have the ML algorithm cheating, which means it has knowledge of the future when it is making the prediction. It is cheating.
It is like writing an exam when you have some of the exam questions in your hand.
So we need to have a strict time partition between the feature space and the labels, because the label is the outcome. The features are the input. The input needs to be completed before the outcome happens.
Whether he did it intentionally or just by chance, either way it is a good thing to remember: signal leakage.
Why a train/test split falls short with 31 examples
The other thing I liked is that Ahmed recognized he had a very small dataset, 31 examples.
He told me what that did to him. At first he wanted to do just seven days of examples, but it was not enough. There was not really much to go on, and he did not even have all three showing up. It was all good and then maybe one bad night of sleep. So he added more days data, and he thought around a month is a good number.
Cross-validation step by step
In cross-validation he is not training a single model. He is training many models. Each time he takes a slice of the entire dataset for model training, and uses another slice for model evaluation. Then in each training run he shifts the training partition and shifts the test partition.
An extreme scenario is leave-one-out. With 31 training examples he could train 31 models. He gives a number to each training example, one through 31, each with an ID. He takes the first 30 examples for model training. Using this trained model, he predicts the sleep quality for night 31. He records this, then shifts. Night 31, which was the single test example, now becomes part of the training set, and he picks another example from the previous training set and makes it the test. He keeps looping this 31 times. Each time there are exactly 30 training examples and one test example. Then he takes the average.
There are other cross-validation setups. He could also do 10-fold cross-validation, which means making 10 equal partitions. With 31 examples, each partition will have roughly three. Now he trains the model 10 times. Each time he takes nine partitions, combines them, and this becomes the training set. He trains the model and uses one of the 10 partitions for evaluation. He keeps repeating, and in each training run a different partition becomes the test. Then he has the scores, and he averages them.
He used three-fold cross-validation.
Understanding cross-validation matters because he makes the most of his examples for both training and evaluation, and gets a more robust and more accurate measure of model performance.
Compare algorithms with a fixed evaluation setup
Once the model evaluation steps are hardened, meaning the cross-validation, the next thing I asked him to try is a few different algorithms, such as XGBoost and logistic regression. He can try a bunch of algorithms. It is easy.
For each algorithm, he keeps the evaluation pipeline fixed, but switches the algorithms. He also keeps recording the model's performance: accuracy, precision, recall. In a three-class classification task there are numbers for each of these classes.
Simplify to binary classification
For the sake of simplicity, I asked him to simplify his classification task to binary classification: did he have a good night of sleep or a bad one? He can combine the okay class with either one. He can take the okay labels to be part of good.
He already created the dataset, so all I am suggesting is that he re-label all the nights where he had okay sleep. He could relabel them as good, or make them bad, either way. If he has very few bad, he can take the okay to be bad. It does not matter during learning. I just want to keep the classification task a bit simpler so that he can spend more time on model evaluation and learn those things.
So he keeps the dataset and the learning task relatively simple, which means he only focuses on binary classification. His sleep is good or bad.
Homework: precision, recall, F1, confusion matrix
Convert the current dataset into two classes, so all the sleeps are either good or bad.
Spend more time understanding accuracy, precision, recall. Compute those scores and understand them.
Look into cross-validation and the confusion matrix, and calculate all these numbers so that you understand them.
F1 is a good one too. Understand these performance metrics.
Hyperparameters and experiment tracking
Keeping this simpler setup, he tries other learning algorithms. For each algorithm, he starts with a simple hyperparameter configuration. For XGBoost, for example, he can start with fewer trees, maybe 100. Then he tries other variations: increasing the number of trees, tuning some of the hyperparameters, and seeing if the model's evaluation metrics jump or change. This is very important, so that he knows which tweaks are improving his model's performance.
More importantly, he keeps track of these experiments. For this particular model configuration and these hyperparameters, here is the accuracy he got. Then he made some changes, and here is the improvement he got. There are experiment tracking tools like MLflow. He can search, or ask Claude or another coding agent. It can come up with recommendations and actually set up his experiment tracker.
Otherwise, if he keeps changing a single notebook, he has no history to go back to. He cannot compare which hyperparameter configuration gave him which number. He has nothing to compare against. So he should not keep changing the code in the same notebook. He could have multiple notebooks, or there are better ways of tracking experiments.
If he gives the prompt to a coding agent, it will help him with this. He does not have to do many of these things by hand.
Master the basics first
There is no need to rush. I asked him master the model evaluation really well, because this will keep coming back to him often. We want a good foundation here before we move to other things.
I am not asking him to finish all of this before our next call. He should just make some progress and ping me, and then we’ll have our next chat.