From Bagging to Boosting—discovering the wisdom of the crowd in Machine Learning.
Last week in my Machine Learning Professional Certificate, we took the Decision Trees I learned about in Module 9 and gave them a massive upgrade.
While a single decision tree is incredibly intuitive, it has a fatal flaw: instability. If you change the training data just a little bit, you can end up with a completely different tree . To fix this, we move from individual trees to Ensemble Methods—combining multiple models to create one super-estimator.
Here is how we turn a single, fragile tree into a powerful, predictive forest.
1. The Depth Dilemma
Before building a forest, we have to understand the limits of a single tree. Choosing a tree’s maximum depth is the ultimate test of the bias-variance trade-off :
- Shallow Trees: Often underfit the data. They are fast and easy to interpret but miss subtle underlying patterns .
- Deep Trees: Often overfit the data. They model the training data perfectly (including the noise) but fail to generalize to new, unseen data .
2. Bagging (Bootstrap Aggregating)
To solve the high variance (overfitting) of deep trees, we use Bagging.
- Instead of training one tree on all the data, we use bootstrapping to draw multiple random samples (with replacement) from our dataset.
- We grow a separate, independent decision tree for every single sample.
- To make a final prediction, we simply aggregate the results: we take a majority vote for classification, or an average for regression . Averaging these models dramatically reduces variance .
3. Random Forests: Forced Diversity
Bagging is great, but it has a weakness: if there is one incredibly strong feature in your data, every single bootstrapped tree will use it at the top split, making all the trees highly correlated .
- Random Forests fix this by forcing the trees to be different.
- At every single splitting step, the algorithm is restricted to only looking at a random subset of features (often the square root of the total features) .
- By preventing the trees from always picking the “best” absolute predictor, we introduce a little bias, but we massively reduce variance because the trees are truly decorrelated .
4. Boosting: Learning from Mistakes
While Bagging and Random Forests build deep trees in parallel to reduce variance, Boosting builds shallow trees sequentially to reduce bias .
- It starts with a “weak learner”—a very shallow tree.
- After the first tree makes its predictions, the algorithm identifies the misclassified data points and increases their “weights” (importance) .
- The next tree is forced to prioritize fixing the mistakes of the previous tree .
- The final prediction is a weighted vote, where highly accurate trees get more say in the final outcome .
5. Feature Importance and Fairness
Finally, we looked at how to peek inside these ensembles to see what features are actually driving predictions.
- Permutation Importance: A clever technique where you randomly shuffle a single feature’s data and measure how much the model’s accuracy drops. A big drop means the feature was highly important!
- Fairness: We also discussed the ethical responsibility of ML. Models can easily learn and amplify historical societal biases if we aren’t careful with the data we feed them (like using zip codes, which can act as proxies for race or income) .
Conclusion
Module 10 proved that in Machine Learning, the wisdom of the crowd almost always beats the smartest individual. Whether we are forcing diversity through Random Forests or learning from mistakes via Boosting, ensemble methods are the secret to winning Kaggle competitions and building robust, real-world models.