Hi All!! Hope everyone had a great weekend!! I was busy last two days and could not work on the part 2 interview concepts asked in FAANG. However, I had few time to spare, I decided to no longer wait and procastinate my work. Previously, we looked in to concepts such as overfitting, dimensional reduction, boosting, what is random forest and how naive bayes is better than random forest, and finally how to handle nulls. You can find the post in Part1. This week, we will look into concepts such as decision trees, SVMs and random forest and their pros and cons, what is ACF & PACF, what is the bias-variance trade off, and how does XGboost handle bais-variance trade off. Without further ado, let's go through the concepts:
1. What is Decision trees, random forest, SVMs and what are their pros and cons?
Decision Trees:
Decision Trees (DTs) are a non-parametric supervised learning method used for classification and regression. The goal is to create a model that predicts the value of a target variable by learning simple decision rules inferred from the data features. A tree can be seen as a piecewise constant approximation.
Pros:
Clear Visualization, Simple and easy to understand, used for both classification and regression problems, can handle both continuous and categorical variables., No feature scaling required, Handles non-linear parameters efficiently, automatically handle missing values, usually robust to outliers and can handle them automatically, and Less Training Period
Cons:
Overfitting, High variance, Unstable, Affected by noise
Random Forest:
A random forest is a machine learning technique that's used to solve regression and classification problems. It utilizes ensemble learning, which is a technique that combines many classifiers to provide solutions to complex problems. A random forest algorithm consists of many decision trees and randomly selecting a subset of variables at each step of the decision tree. Random Forest is based on bagging method which is one of the Ensemble Learning techniques.
Pros:
reduces overfitting in decision trees, helps to improve the accuracy, handles both classification and regression problems, works well with both categorical and continuous values, automates missing values present in the data, Normalising of data is not required as it uses a rule-based approach, and can determine feature importance.
Cons:
requires much computational power, requires much time for training as it combines a lot of decision trees to determine the class, suffers interpretability.
SVMs:
It is a classification technique that finds a hyperplane or a boundary between the two classes of data that maximizes the margin between the two classes. There are many planes that can separate the two classes, but only one plane can maximize the margin or distance between the classes.
Pros:
works relatively well when there is a clear margin of separation between classes, effective in high dimensional spaces, effective in cases where the number of dimensions is greater than the number of samples, memory efficient
Cons:
not suitable for large data sets, does not perform very well when the data set has more noise, will underperform if the number of features for each data point exceeds the number of training data samples.
2. What is the meaning of ACF and PACF?
ACF (autocorrelation function) is a tool that is used to find patterns in the data, specifically in terms of correlations between points separated by various time lags.
For example, ACF(0)=1 means that all data points are perfectly correlated with themselves and ACF(1)=0.9 means that the correlation between one point and the next one is 0.9.
PACF (Partial autocorrelation function) is different from ACF as in instead of finding correlations of present with lags like ACF, it finds correlation of the residuals (which remains after removing the effects which are already explained by the earlier lag(s)) with the next lag value hence ‘partial’ and not ‘complete’ as we remove already found variations before we find the next correlation.
ACF and PACF plots are used in determining the order of auto-regressive (AR) and moving average (MA) series.
3. What is bias-variance trade off?
In order to understand bias-variance trade off, we should know what is bias & variance.
Bias are the simplifying assumptions made by a model to make the target function easier to learn.
Low Bias: Suggests less assumptions about the form of the target function.
High-Bias: Suggests more assumptions about the form of the target function.
Examples of low-bias machine learning algorithms include: Decision Trees, k-Nearest Neighbors and Support Vector Machines.
Examples of high-bias machine learning algorithms include: Linear Regression, Linear Discriminant Analysis and Logistic Regression.
Variance is the amount that the estimate of the target function will change if different training data was used.
Low Variance: Suggests small changes to the estimate of the target function with changes to the training dataset.
High Variance: Suggests large changes to the estimate of the target function with changes to the training dataset.
Examples of low-variance machine learning algorithms include: Linear Regression, Linear Discriminant Analysis and Logistic Regression.
Examples of high-variance machine learning algorithms include: Decision Trees, k-Nearest Neighbors and Support Vector Machines.
Bias-Variance Trade-Off
The ideal goal of any supervised machine learning algorithm is to achieve low bias and low variance.
If our model is too simple and has very few parameters then it may have high bias and low variance. On the other hand if our model has large number of parameters then it’s going to have high variance and low bias. So we need to find the right/good balance without overfitting and underfitting the data.
Total Error
To build a good model, we need to find a good balance between bias and variance such that it minimizes the total error. An optimal balance of bias and variance would never overfit or underfit the model.
How XGBoost handles Bias-Variance Trade off:
XGBoost is an ensemble Machine Learning algorithm that leverages the gradient boosting algorithm. XGBoost handles bias and variance similar to that of any boosting technique. Boosting is an ensemble meta-algorithm that reduces both bias and variance by takes a weighted average of many weak models. By focusing on weak predictions and iterating through models, the bias error is reduced. Similarly, because it takes a weighted average of many weak models, the final model has a lower variance than each of the weaker models themselves.
Resources:
FAANG Interview Concepts series:
Comments