Introduction to Machine Learning |
Univariate linear regression is a statistical technique used for modeling the relationship between a single independent variable \(x\) and a dependent variable \(y\). In this approach, we aim to find a linear equation that best represents the relationship between the two variables.
1. Prediction Equation:
\(y = \beta_0 + \beta_1x\)
2. Loss Function:
\(L(\beta_0,\beta_1) = \frac{1}{n}\sum_{i=1}^n (y_i - (\beta_0 +
\beta_1x_i))^2\)
(Note: This loss function is referred to
as L2 loss function and uses the MSE to measure the model's accuracy.)
3. Optimizing the Loss Function:
We can minimize \(L(\beta_0,\beta_1)\) by taking partial derivatives with respect to \(\beta_0\) and \(\beta_1\), and setting them to zero.
Partial derivative with respect to \(\beta_0\):
\(\frac{\partial L}{\partial \beta_0} = -\frac{2}{n}\sum_{i=1}^n (y_i - (\beta_0 + \beta_1x_i)) = 0\)
Multiply both sides by \(-\frac{n}{2}\) to simplify:
\(\sum_{i=1}^n (y_i - (\beta_0 + \beta_1x_i)) = 0\)
Isolate \(\beta_0\):
\(\beta_0 = \bar{y} - \frac{\beta_1}{n}\sum_{i=1}^n x_i\)
\( \beta_0 = \bar{y} - \beta_1\bar{x} \)Substitute \(\beta_0 = \bar{y} - \beta_1\bar{x}\) into the loss function to make it in terms of \(\beta_1\) only:
\(L(\beta_0, \beta_1) = \frac{1}{n}\sum_{i=1}^n (y_i - (\beta_0 + \beta_1x_i))^2\)
\(L(\beta_1) = \frac{1}{n}\sum_{i=1}^n \left(y_i - \left(\bar{y} - \beta_1\bar{x} + \beta_1x_i\right)\right)^2\)
Partial derivative with respect to \(\beta_1\):
\(\frac{\partial L}{\partial \beta_1} = -\frac{2}{n}\sum_{i=1}^n \left(y_i - \bar{y} + \beta_1(\bar{x} - x_i)\right)(\bar{x} - x_i) = 0\)
\(\sum_{i=1}^n (\left(y_i - \bar{y}) + \beta_1(\bar{x} - x_i)\right)(\bar{x} - x_i) = 0\)
\(\beta_1 = \frac{\sum_{i=1}^n \left(y_i - \bar{y}\right)(x_i - \bar{x})}{\sum_{i=1}^n (\bar{x} - x_i)(\bar{x} - x_i)}\)
\(\beta_1 = \frac{\sigma_{xy}}{\sigma_x^2}\)Python Example