Why memorizing isn’t learning, and how to spot a model that’s cheating.
In Module 5 of my Machine Learning Professional Certificate, I tackled a fundamental philosophical question: Is learning even feasible?
It turns out that without making assumptions, the answer is “no”. We explored the “No Free Lunch” theorem and the problem of induction, which tells us that just because the sun rose every day in the past, we can’t theoretically prove it will rise tomorrow. To make learning possible, we need three ingredients: a probabilistic setting, stationarity (the future resembles the past), and prior knowledge .
Here are the core concepts I mastered this week:
1. The Feasibility of Learning (PAC)
We started with a “toymaker” example: a binary classification problem with 3 inputs and 1 output . Even with 5 observed data points, there were still 8 candidate functions that could perfectly explain the past data but predict differently for the future.
This proved that we can never be 100% certain. Instead, we use the PAC (Probably Approximately Correct) framework: with enough data and a simple enough hypothesis, we can be probably confident (with high probability 1-δ) that our model is approximately correct (with small error ε).
2. Probabilistic Approaches
Since we can’t be certain, we rely on probability. I compared four key approaches:
- Bayesian Inference: Uses priors and likelihoods to update beliefs. It’s great for cases like medical diagnoses where disease prevalence (prior) is low.
- Laplace’s Rule of Succession: Adds “pseudo-counts” (one imaginary success and failure) so you never predict 0% or 100% certainty just because data is scarce.
- Frequentist: Relies solely on observed data (e.g., if 30/200 website visitors buy, the probability is 15%).
- Maximum Likelihood Estimation (MLE): Finds the specific parameter values that make the observed data “most probable” .
3. The Bias-Variance Trade-off
This was the central theme of the module. We are constantly balancing two opposing errors:
- High Bias (Underfitting): The model is too simple (like drawing a straight line through a curved dataset). It misses the signal entirely.
- High Variance (Overfitting): The model is too complex (like a wiggly polynomial) . It memorizes the noise in the training data and fails on new data.
- The Goal: Find the “Goldilocks” zone—the simplest model that captures the signal without learning the noise .
4. The Golden Rule of Data Splitting
To measure if our model is actually learning (generalizing) or just memorizing, we must split our data.
- Training Set (~60-80%): Used strictly to teach the model.
- Validation Set (~10-20%): Used to tune the model and select the best one.
- Test Set (~10-20%): The “final exam.” It is never used during training or selection. It is only touched once at the very end to estimate real-world performance.
Conclusion
Module 5 shifted my perspective from just “fitting data” to true generalization. The most complex model isn’t the best; the one that performs best on unseen data is. Next week, I’ll be applying these principles to actual model selection in Python!