Positional Encoding in Transformer

-

1. Why Position Matters in Transformers?

  • Transformers rely on self‑attention, which processes tokens in parallel.
  • This means, unlike RNNs, they don’t inherently know the order of words.
  • So, sentences like “Ravi killed the lion” vs. “The lion killed Ravi” would look identical to a vanilla Transformer—clearly problematic!

🧪 Idea #1: The Naïve Approach



A simple fix would be to add index/position of the token in an embedding vector.





Issues:

Unbounded values: Position IDs can become huge (e.g. 100,000+ in long texts), destabilizing training.


Discrete steps: Sharp jumps between integers disrupt gradient flow.


🧪 Idea #2: Normalize the Position Numbers



What if we divide the position numbers by a constant to make them small and smooth?





That helps a bit—values don’t explode anymore.


Issues:

Now, if you observe, in both the sentences, the word at second position has got the different values.

1 for sentence1, and 0.5 for sentence2.


so, the Neural network will get confused while training, what actually is the second position.


The neural network prefers continuous values, not discrete values.


Can't capture relative positioning. we won't know that word_X is 3 steps ahead of word_Y.

Discrete Position:





Relative Position:




The model should know:

How far apart two words are


Whether a word comes before or after another - 'the' comes after the word 'Nitish'


So, we need a function which is :


Bounded (values between –1 and 1)
Continuous
Periodic (captures repeating patterns, and hence relative distances).

🧪 Idea #3: Use the sine function



It is bounded, continuous and periodic.








Issues :

If we only use sin(position), some positions might get the same values again and again due to the repeating nature of the wave. That causes clashes—different positions looking the same!

🧪 Idea #4: Use Both Sin and Cos with Different Angles




We combine sin and cos functions


We use different angles (frequencies) for each dimension in the vector


Issues:

But what if, the text is too large and this too, starts repeating.

🧪 Idea #5: How do we decide this sine cosine frequency?



Solution given in paper : Attention is all you need.





Where:

i = current dimension number


d = total dimensions


Let's say total dimensions=6, now we have 2 formulas, so will go till halfway only, ( 0,1,2).


And because of the presence of 2 formulas(sin and cos), we will get the values of 6 dim by traversing only till 3.










Final Step :

We will add this positional vector with its word embedding vector.





Why Do We Add the Position Vector Instead of Appending It?

If we append the positional vector, the input size will double (say, 512 for word + 512 for position = 1024). That would increase model complexity.
Instead, we simply add the positional vector to the word embedding. This keeps the size same (512), and the model gets both meaning + position together in one vector. It’s simple and effective.
Reference:


https://youtu.be/GeoQBNNqIbM?si=ZTpSLl-jJdAiNJeh

Comments

Post a Comment

Popular posts from this blog

Extracting Tables and Text from Images Using Python

-
Image
In this blog, we'll explore a complete Python solution that detects and extracts tables and text from images using libraries like Transformers, OpenCV, PaddleOCR, and easyOCR. This step-by-step breakdown includes code to detect tables, extract content from individual table cells, and retrieve any remaining text in the image. Overview When working with scanned documents, such as invoices or forms, it is essential to accurately extract both structured information (like tables) and unstructured text. The approach we’ll explore uses Microsoft's pretrained object detection model to locate tables and OCR techniques to extract the text from both table cells and the rest of the image. Steps: 1. This code first detects table using microsoft's model. and save that image which contains detected table only  2. After that, from the detected table , we make a seperate image for each cell.  3. Then we read text from the image of each cell  4. Now, to read the extra texts except for the ...
Read more

Semantic Search and Vector Database

-
Image
Why do we need Vector DB? Let’s start with a common scenario: You’re searching for “affordable laptop with a good battery life” on an e-commerce site. A keyword-based search engine might return results that include the words “affordable,” “laptop,” and “battery life.” But it might also miss products that are described as “budget-friendly” or “long-lasting battery,” simply because they don’t match the exact keywords. The result? A frustrating search experience where you might not find what you’re really looking for, even though the perfect product is just a few clicks away. Keyword-based search systems struggle with: Synonyms : Different words can mean the same thing (e.g., “cheap” and “affordable”). Polysemy : The same word can have multiple meanings depending on context (e.g., “bank” as in a financial institution vs. “bank” as in the side of a river). Contextual Understanding : Words and phrases can have different meanings based on the context in which they’re used. These limitations ...
Read more

Chain Component in LangChain

-
Image
LangChain is a powerful framework for building applications with Large Language Models (LLMs). One of its core features is the Chain Component , which allows developers to link multiple operations together to create complex workflows. In this blog, we'll explore different types of chains in LangChain and demonstrate their usage with code examples. 1️⃣ LLMChain (Basic Single-Step Chain) 🔹 Executes a single LLM call based on a prompt. 🔹 The simplest chain type. ✅ Use Case: When you need a single prompt-response from an LLM. Example: 2️⃣ SequentialChain (Step-by-Step Execution) 🔹 Executes multiple chains in sequence where each step depends on the output of the previous step. ✅ Use Case: Multi-step processes where each step depends on the previous output . 📌  Example: Blog Outline → Content Expansion → Summary Example: 3️⃣ SimpleSequentialChain (Basic Sequential Execution) 🔹 Similar to SequentialChain, but only passes the last step’s output forward (less flexible). 🔹 ...
Read more