Now it applies the rule it learned to actually scale the data.
Why Separate Them?
Training Data
python
train_data =[10,20,30,40,50]# Learn rules from training datascaler.fit(train_data)# Apply rules to training data train_scaled = scaler.transform(train_data)# Result: [0, 0.25, 0.5, 0.75, 1.0]
New Test Data
python
test_data =[15,25,35]# DON'T fit again! Use the same rules learned from trainingtest_scaled = scaler.transform(test_data)# Result: [0.125, 0.375, 0.625]
Key Point: Test data uses the same min (10) and max (50) learned from training!
What if You Fit Again on Test Data? BAD!
python
# WRONG WAY:scaler.fit(test_data)# This learns NEW rules: min=15, max=35test_scaled = scaler.transform(test_data)# Result: [0, 0.5, 1.0] # Different scale! Can't compare with training!
Ordinal: Small → Medium → Large (Order matters)
0 → 1 → 2
Nominal: Red, Blue, Green (No order)
[1,0,0] [0,1,0] [0,0,1] (one-Hot Encoding)
If 500 cols, you will have to make vector of 500 dims
Label: Cat, Dog, Fish (Target only)
0, 1, 2
Let's dive deeper into the one hot encoding.
When, you make a sum or each vector , is will be 1, so model will start maintaining relationship between the cols, so to avoid dependency in input columns, we just skip one column from the vector and take n-1. It is the issue of multi colinearity and is called a dummy variable trap.
what if there are 1000 cols? we can keep n (lets say 800 important cols) and one col as 'other' which are less frequent or unimportant.
Lets take one example :
Notice the increased number of columns.
To remove the 1st col from this new dataset, drop_first = True. or drop = 'first'
Manual way for encoding(tedious):
# Step by step - messy!
age_salary_scaled = StandardScaler().fit_transform(df[['Age','Salary']])
city_encoded = OneHotEncoder().fit_transform(df[['City']])
education_encoded = OrdinalEncoder().fit_transform(df[['Education']])# Then combine them back - pain!
final_data = np.concatenate([age_salary_scaled, city_encoded, education_encoded], axis=1)
column transformer :
Applies different transformations to different columns in one step
Pipeline in ML :
It is same as langchain pipeline.
1. Train Script Without Pipeline:
# Step 1: Data Cleaning
X_train_clean = clean_data(X_train)
# Step 2: Feature Scaling
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train_clean)
# Step 3: Model Training
model = LogisticRegression()
model.fit(X_train_scaled, y_train)
# Step 4: Hyperparameter Tuning (Optional)# Let's assume hyperparameter tuning is done here
2. Eval Script Without Pipeline:
# Step 1: Data Cleaning
X_test_clean = clean_data(X_test)
# Step 2: Feature Scaling
X_test_scaled = scaler.transform(X_test_clean) # Use the same scaler from training# Step 3: Model Evaluation
y_pred = model.predict(X_test_scaled)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
3. Train Script With Pipeline:
from sklearn.pipeline import Pipeline
# Define the pipeline with all 4 steps
pipeline = Pipeline([
('cleaner', DataCleaner()), # Step 1: Custom Data Cleaning
('scaler', StandardScaler()), # Step 2: Feature Scaling
('classifier', LogisticRegression()) # Step 3: Model Training
])
# Train the pipeline
pipeline.fit(X_train, y_train)
4. Eval Script With Pipeline:
# Evaluation using the pipeline (no need to manually transform)
y_pred = pipeline.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")
Key Differences:
Without Pipeline: You manually apply each step (cleaning, scaling, and model training). During evaluation, you must repeat those steps (like transforming the test data).
With Pipeline: The pipeline takes care of the entire workflow. When calling pipeline.fit() and pipeline.predict(), it applies all the necessary steps automatically.
For the normal distribution of data, some sort of transformation is required.
what is normal distribution?
- there is one concept called QQplot, which shows
1. if black line is exactly on the green line, its good, it's a normal distribution.
2. If the black line is far from the green line, it is not the normal distribution.
For some algorithm to work properly, normal distribution is required.
Algorithm
Requires Normality?
Notes
Linear Regression
Residuals only
For valid p-values and confidence intervals
Logistic Regression
No (but helps)
Performance improves with scaled/normalized data
LDA
✅ Yes
Assumes normal distribution within classes
QDA
✅ Yes
Like LDA but allows different covariances
Gaussian Naive Bayes
✅ Yes
Assumes Gaussian features per class
PCA
No (but helps)
Normality helps in better variance capture
Tree-based models
❌ No
Insensitive to feature distribution
Neural Networks
❌ No
Benefit from normalization, not normality
Transforms:
1. log transform : log x (base 2 or 10, depends on requirement)
[log transform is mostly used on right skewed data]
2. reciprocal : 1/x
3. square : x^2
4. sqrt : square root of x
Function transformer contains all these functions :
Example :
we can pass our custom logic as well in function transformer.
Power Transformer :
1. Box-Cox
2. Yeo - Johnson
There is one more transformer called quantile transformer.
Binning and Binarization
Binning :
used to convert numerical data into categorical.
whats the need? can be useful in decision tree.
how does it work? Data gets converted into categories.
- let's say app A has been downloaded by 6000 users and B
has been downloaded by 10,34,333 users, this things cab be simplified.
Downloaded by 5k+ and 1M+ users
Another example : Age
# Example: A list of ages
ages = [5, 12, 18, 25, 30, 40, 50, 65, 70]
# Define the bin edges
bins = [0, 18, 35, 50, 100]
# Define the labels for each bin
labels = ['0-18', '19-35', '36-50', '51-100']
# Apply binning using pandas 'cut' function
age_groups = pd.cut(ages, bins=bins, labels=labels)
print(age_groups)
It is called Equal width or uniform because the width of the interval is the same or uniform.
Suppose, we have the age data of some people and the maximum age in the data is 100 and minimum is 0. Let’s take no. of bin is 10 then the width of the interval will be 10.
Here, the graph will be Histogram which has 10 bins and width is also 10.
It is used because :-
It handled the outliers as outliers came into the last bins.
No change in the spread of data.
2. Equal Frequency/Quantile Binning
It is also known as Quantile binning. Here, we talk about the quantiles like 10 percentile, 20 percentile. The width of the interval is not the same as in equal frequency.
Let’s understand this with the help of an example. Suppose the above example of age data.
In Quantile Binning, let’s say we want 10 percentile.
The age width where 10 percentile lies like here is 0–16 then again 10 percentile i.e., 16–20 and if we start from 0–20 then it will be 20 percentile. Now, 20 -22 will be 10 percentile but from 0–22 will be 30 percentile.
Each interval contains 10% of total observations.
Zoom image will be displayed
Why to use equal frequency :-
Works better for outliers.
Improve the value spread.
3. KMeans Binning
4. Custom Binning
we use our knowledge base to perform this binning, for example:
0-5 = kids
6-12 = children
13-18 = Teens
etc.
Binarization :
✅ What is Binarization?
Binarization is the process of converting continuous numerical values into binary (0 or 1) values.
It applies a threshold to the data:
Values greater than or equal to the threshold become 1
Values less than the threshold become
🧪 Example: Exam Scores
Suppose you have students' exam scores, and you want to classify whether they passed (score ≥ 50) or failed (score < 50).
In this blog, we'll explore a complete Python solution that detects and extracts tables and text from images using libraries like Transformers, OpenCV, PaddleOCR, and easyOCR. This step-by-step breakdown includes code to detect tables, extract content from individual table cells, and retrieve any remaining text in the image. Overview When working with scanned documents, such as invoices or forms, it is essential to accurately extract both structured information (like tables) and unstructured text. The approach we’ll explore uses Microsoft's pretrained object detection model to locate tables and OCR techniques to extract the text from both table cells and the rest of the image. Steps: 1. This code first detects table using microsoft's model. and save that image which contains detected table only 2. After that, from the detected table , we make a seperate image for each cell. 3. Then we read text from the image of each cell 4. Now, to read the extra texts except for the ...
1. Why Position Matters in Transformers? Transformers rely on self‑attention, which processes tokens in parallel. This means, unlike RNNs, they don’t inherently know the order of words. So, sentences like “Ravi killed the lion” vs. “The lion killed Ravi” would look identical to a vanilla Transformer—clearly problematic! 🧪 Idea #1: The Naïve Approach A simple fix would be to add index/position of the token in an embedding vector. Issues: Unbounded values: Position IDs can become huge (e.g. 100,000+ in long texts), destabilizing training. Discrete steps: Sharp jumps between integers disrupt gradient flow. 🧪 Idea #2: Normalize the Position Numbers What if we divide the position numbers by a constant to make them small and smooth? That helps a bit—values don’t explode anymore. Issues: Now, if you observe, in both the sentences, the word at second position has got the different values. 1 for sentence1, and 0.5 for sentence2. so, the Neural network will get confused while training, what a...
LangChain is a powerful framework for building applications with Large Language Models (LLMs). One of its core features is the Chain Component , which allows developers to link multiple operations together to create complex workflows. In this blog, we'll explore different types of chains in LangChain and demonstrate their usage with code examples. 1️⃣ LLMChain (Basic Single-Step Chain) 🔹 Executes a single LLM call based on a prompt. 🔹 The simplest chain type. ✅ Use Case: When you need a single prompt-response from an LLM. Example: 2️⃣ SequentialChain (Step-by-Step Execution) 🔹 Executes multiple chains in sequence where each step depends on the output of the previous step. ✅ Use Case: Multi-step processes where each step depends on the previous output . 📌 Example: Blog Outline → Content Expansion → Summary Example: 3️⃣ SimpleSequentialChain (Basic Sequential Execution) 🔹 Similar to SequentialChain, but only passes the last step’s output forward (less flexible). 🔹 ...
what if there are 1000 cols? we can keep n (lets say 800 important cols) and one col as 'other' which are less frequent or unimportant.
Comments
Post a Comment