Decision Tree For Regression

What is a Decision Tree for Regression?

They help us predict continuous values.

How Does It Work?

1. Start at the Root: Begin with the entire dataset at the root node.
2. Split the Data: Choose the best feature to split the data into two groups. The goal is to minimize the error in each group.
3. Repeat: Continue splitting each group until you meet a stopping criterion (like a maximum depth or minimum samples per leaf).
4. Make Predictions: The value at each leaf node is the predicted value for data points that fall into that leaf.

Example with Calculation

Let's take a simple example to illustrate how a decision tree works for regression.

Dataset

Consider a small dataset of house prices based on the size of the house, first, sort the data, By sorting the data and evaluating potential split points, a decision tree for regression accurately predicts continuous values. :

House Size (sq ft)	Price (in $1000)
1100	199
1400	245
1425	319
1550	219
1600	312
1700	279
1700	255
1875	308
2350	405
2450	324

Step-by-Step Calculation

1. Calculate the Mean Squared Error (MSE) for the Root Node:

   $$\bar{y} = \frac{199 + 245 + 319 + 219 + 312 + 279 + 255 + 308 + 405 + 324}{10} = 286.5$$
   $$\text{MSE} = \frac{1}{10} ((199 - 286.5)^2 + (245 - 286.5)^2 + \ldots + (324 - 286.5)^2) = 4048.25$$
   

2. Evaluate Potential Split Points: Evaluate potential splits between each pair of consecutive house sizes. Let's consider a split between 1550 and 1600 sq ft (this is the midpoint of these values).
   
   

   
   
   
   

Split at 1575 sq ft

• Left Split (size ≤ 1575): $$\bar{y}_\text{left} = \frac{199 + 245 + 219 + 219}{4} = 245.5$$
  $$\text{MSE}_\text{left} = \frac{1}{4} ((199 - 245.5)^2 + (245 - 245.5)^2 + (319 - 245.5)^2 + (219 - 245.5)^2) = 2454.25$$
  
• Right Split (size > 1575): $$\bar{y}_\text{right} = \frac{312 + 279 + 255 + 308 + 405 + 324}{6} = 313.83$$
  $$\text{MSE}_\text{right} = \frac{1}{6} ((312 - 313.83)^2 + (279 - 313.83)^2 + (255 - 313.83)^2 + (308 - 313.83)^2 + (405 - 313.83)^2 + (324 - 313.83)^2) = 2757.5$$
  

3. Calculate the Weighted MSE for the Split: $$\text{Weighted MSE} = \frac{4}{10} \times 2454.25 + \frac{6}{10} \times 2757.5 = 2634.9$$Compare this with the MSE of the root node (4048.25). Since 2634.9 < 4048.25, the split improves the prediction.

Further Splits

Let's consider another split on the left and right nodes for more depth.

Split on Left Node at 1412.5 sq ft (midpoint between 1400 and 1425):

• Left-Left Split (size ≤ 1412.5): $$\bar{y}_\text{left-left} = \frac{199 + 245}{2} = 222$$
  $$\text{MSE}_\text{left-left} = \frac{1}{2} ((199 - 222)^2 + (245 - 222)^2) = 422$$
  
• Left-Right Split (size > 1412.5): $$\bar{y}_\text{left-right} = \frac{319 + 219}{2} = 269$$
  $$\text{MSE}_\text{left-right} = \frac{1}{2} ((319 - 269)^2 + (219 - 269)^2) = 2500$$
  

Split on Right Node at 2012.5 sq ft (midpoint between 1875 and 2350):

• Right-Left Split (size ≤ 2012.5): $$\bar{y}_\text{right-left} = \frac{312 + 279 + 255 + 308}{4} = 288.5$$
  $$\text{MSE}_\text{right-left} = \frac{1}{4} ((312 - 288.5)^2 + (279 - 288.5)^2 + (255 - 288.5)^2 + (308 - 288.5)^2) = 662.25$$
  
• Right-Right Split (size > 2012.5): $$\bar{y}_\text{right-right} = \frac{405 + 324}{2} = 364.5$$
  $$\text{MSE}_\text{right-right} = \frac{1}{2} ((405 - 364.5)^2 + (324 - 364.5)^2) = 3242.25$$
  

Final Predictions

The final tree and predictions are:

• Left-Left Split (size ≤ 1412.5): $$\text{Predicted Price} = 222$$
• Left-Right Split (1412.5 < size ≤ 1575): $$\text{Predicted Price} = 269$$
• Right-Left Split (1575 < size ≤ 2012.5): $$\text{Predicted Price} = 288.5$$
• Right-Right Split (size > 2012.5): $$\text{Predicted Price} = 364.5$$

Test Case :

Qs: Predict the value for a house size of 1700 sq ft 

To predict the value for a house size of 1700 sq ft using the provided decision tree, follow the path of the tree:

1. Starting at the root:

   • Check if the house size (1700) is less than or equal to 1575.
   • Since 1700 is greater than 1575, follow the "NO" branch.

2. Next node:

   • Check if the house size (1700) is less than or equal to 2012.5.
   • Since 1700 is less than or equal to 2012.5, follow the "YES" branch.

3. Final prediction:

   • The predicted price for the house size of 1700 sq ft falls under the node with the predicted price of 288.5.

Therefore, the predicted price for a house size of 1700 sq ft is $288,500.