Train-test split & Cross-Validation

-

 Train-test split :

Let's say we have a dataset with 100 samples.

Now, if we train the model on all 100 samples and test the results from this 100 samples only, then even if the data is overfitted, we won't know!

So, to avoid this, we use train-test split



In train-test split, we divide the data into 2 parts:

1) Training Set :

We use 70-80% data as a training set. So, for above example we won't train all 100 samples but we will train the model on 80 samples.

2) Test set :

The test set usually contains the remaining 20-30% of the data. So, for above example we will test model for the rest 20 samples.

The test set is used to evaluate how well the model generalizes to new, unseen data. The performance metrics (such as accuracy, precision, recall, etc.) are calculated based on the model's predictions on the test set.


Cross-validation :

Divides the dataset into multiple folds. Some of the folds as training set & some of the folds as testing set. This process is repeated multiple times, each fold is used as the test set exactly once.

Here's a simple example to illustrate cross-validation:

Let's say we have a dataset with 100 samples. We decide to use 5-fold cross-validation, meaning we will split the data into 5 subsets, each containing 20 samples.

  1. Split the Data: We divide the dataset into 5 folds:

    • Fold 1: Samples 1-20
    • Fold 2: Samples 21-40
    • Fold 3: Samples 41-60
    • Fold 4: Samples 61-80
    • Fold 5: Samples 81-100

  2. Iterative Training: We then iterate through the following process 5 times, each time using a different fold as the validation set:

    • For iteration 1: Train on Folds 2,3,4,5, Validate on Fold 1
    • For iteration 2: Train on Folds 1,3,4,5, Validate on Fold 2
    • For iteration 3: Train on Folds 1,2,4,5, Validate on Fold 3
    • For iteration 4: Train on Folds 1,2,3,5, Validate on Fold 4
    • For iteration 5: Train on Folds 1,2,3,4, Validate on Fold 5

  3. Model Evaluation: After the iterations are complete, we have used each fold as the validation set exactly once. We can then average the performance metrics (such as accuracy, precision, recall, etc.) from each iteration to get an overall estimate of the model's performance.

Cross-validation helps to provide a more accurate estimate of the model's performance compared to using a single train-test split, as it allows the model to be trained and tested on different subsets of the data. This can help to identify potential issues like overfitting or data bias.


Generally, in the code you will use below variables:

x_train, y_train, x_test, y_test, y_pred

Assume that you want to predict salary based on the experience in years.

So, x = experience

y = salary

x_train : 70% data from experience

x_test : rest 30% data from experience

y_train : 70% data from salary

y_test : rest 30% data from salary

y_pred : we give x_test (experience) as input and it will predict salary(y_pred)


So, here y_test = actual salary

y_pred = salary predicted by model


Types of Cross-Validation :

(Using Example of 100 samples)

1. K-Fold Cross-Validation: For our example dataset of 100 samples, if we choose K=5 for 5-fold cross-validation, each fold would consist of 20 samples. we train on 4-folds and test on 1-fold


2. Stratified K-Fold Cross-Validation: If our dataset has two classes - Male(M) & Female(F), each with 80 and 20 samples making total of 100, each fold in 5-fold stratified cross-validation would contain 16 samples for M and 4 samples foe F.


3. Leave-One-Out Cross-Validation (LOOCV): In LOOCV, the model would be trained on 99 samples and tested on 1 sample, repeated for all 100 samples.


Image Source: ISLR 4. Leave-P-Out Cross-Validation: For leave-3-out cross-validation, the model would be trained on 98 samples and tested on 3 samples, repeated for all possible combinations of leaving out 3 samples.


5. Repeated K-Fold Cross-Validation: You might perform 5-fold cross-validation 10 times, each time with a different random split of the data.

First iteration:

Fold 1: Random samples 1-20 Fold 2: Random samples 21-40 Fold 3: Random samples 41-60 Fold 4: Random samples 61-80 Fold 5: Random samples 81-100

Second iteration:

Fold 1: Random samples 11-30 Fold 2: Random samples 31-50 Fold 3: Random samples 51-70 Fold 4: Random samples 71-90 Fold 5: Random samples 91-100, 1-10 (wrap around)

and so on...


Comments

Post a Comment

Popular posts from this blog

Extracting Tables and Text from Images Using Python

-
Image
In this blog, we'll explore a complete Python solution that detects and extracts tables and text from images using libraries like Transformers, OpenCV, PaddleOCR, and easyOCR. This step-by-step breakdown includes code to detect tables, extract content from individual table cells, and retrieve any remaining text in the image. Overview When working with scanned documents, such as invoices or forms, it is essential to accurately extract both structured information (like tables) and unstructured text. The approach we’ll explore uses Microsoft's pretrained object detection model to locate tables and OCR techniques to extract the text from both table cells and the rest of the image. Steps: 1. This code first detects table using microsoft's model. and save that image which contains detected table only  2. After that, from the detected table , we make a seperate image for each cell.  3. Then we read text from the image of each cell  4. Now, to read the extra texts except for the ...
Read more

Getting Started with ML

-
Image
 Types of ML :- 1) Supervised ML :- Before going for supervised learning, you need to be aware of two terminologies: Suppose, you have the data of student's age and weight. Now, for new student, just by knowing age, you want to predict his weight. So, in this case i. Dependent Variable : Age ii. Independent Variable : Weight Regression : It is applied on Numeric Data such as temperature, weight etc. Classification : It is applied on Categorial Data. There are 2 types of classification: 1.Binary Classification - Yes/No, Spam/Not Spam, etc. 2. Multiclass Classification - Small/Medium/Large, Bad/Average/Good/Excellent etc. 2) Unsupervised ML :- i. Clustering : If we have some data where there is no dependent/independent variable, then we can use clustering. suppose, we have data for salary and age. There is no relation between salary and age. So, clustering groups the data and keep the data with same characteristics together. ii. Dimensionality Reduction : Suppose, we have 100...
Read more

Linear Regression

-
Image
 Linear Regression shows the Linear Relationship between dependent features(Y-axis) and independent features(X-axis).  There are 2 types of Linear Regression: 1. Simple Linear Regression -  It has only one independent variable. Hypothesis Function for Simple Linear Regression : where: Y is the dependent variable X is the independent variable  β0 is the intercept ( the value where the regression line crosses the y-axis) β1 is the slope/weight ex : calculate weight based on age. In above figure Red dots represent Actual Data Points and black dots represent predicted values. error is calculated by (Actual Value - Predicted Value) 2. Multiple Linear Regression -  It has more than one independent features.  Hypothesis Function for Simple Linear Regression : y= a0 + a1x1 + a2x2 + a3x3 + ..  Let's understand this by example, we all know that if the size of the house is bigger, then the price also increases. But along with this, there are many other factors ...
Read more