NeuralPath // AI & ML Learning Hub

What is Linear Regression?

Linear Regression is a supervised learning algorithm used to predict a continuous numerical value.

It learns the relationship between an input feature $X$ and an output target $Y$ by fitting a straight line through the observed training data.

For example:

Years of Experience → Monthly Salary

House Size ( $m^2$ ) → Market Price ($)

If you know how much the input variable changes, the model discovers the exact rate of change so you can predict the outcome for new, unseen data points.

The Core Mental Model & Visual Intuition

Suppose we collect real-world data points comparing house sizes to sale prices. On a two-dimensional grid, they scatter like an elongated cloud:

Feature Space

Data Point (x1, y1)

Data Point (x2, y2)

Data Point (x3, y3)

↓

Fitted Straight Line (y_hat = wx + b)

↓

Minimizes Total Squared Residuals

Because real data contains noise, the points never align perfectly on a line. The algorithm finds the single straight line that passes through the dense center of that cloud.

Residual (Error): The vertical distance between an actual observed point

y_i

and the line's estimate

\hat{y}_i

Core Objective: Shift and tilt the line until the cumulative sum of these squared residuals is minimized.

How Does It Make a Prediction?

Linear Regression uses the standard slope-intercept algebraic formula:

\hat{y} = wx + b

Where:

$x$ = input feature (e.g., house square footage)
$\hat{y}$ = predicted value (e.g., estimated price)
$w$ = weight or slope (rate of change per unit of $x$ )
$b$ = bias or y-intercept (baseline value when $x = 0$ )

Step-by-Step Worked Example:

Suppose our fitted model discovers $w = 900$ and $b = 2{,}100$ :

\text{Salary} = (900 \times \text{Experience}) + 2{,}100

If an applicant has $x = 4$ years of experience:

\hat{y} = (900)(4) + 2{,}100 = 3{,}600 + 2{,}100 = \$5{,}700

The data pipeline runs through the following sequence:

Input: x

→

Multiply by w

→

Add bias b

→

Prediction: y_hat

How Does It Learn the Best Line?

The model starts with initial guesses for $w$ and $b$ , then iteratively refines them through an optimization loop:

Initialize weights w and b

↓

Compute Predictions y_hat

↓

Calculate Loss via MSE

↓

Compute Gradients

↓

Update w and b via Learning Rate

Loss Converged?

No → back to Compute Predictions y_hat

Yes → Optimal Parameters Found

The standard optimization metric used is Mean Squared Error (MSE):

MSE = \frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2

Squaring the errors achieves two critical objectives:

It eliminates negative signs so errors in opposite directions do not cancel out.
It penalizes large deviations much more severely than small ones.

What Do Weight and Bias Mean Visually?

In the equation:

\hat{y} = wx + b

Weight (w): Slope

Controls tilt, steepness, and direction

Bias (b): Y-Intercept

Shifts line vertically along Y-axis

Weight ( $w$ ): Controls the angle/slope of the line. If

w = 900

, each additional unit of

x

increases

\hat{y}

900

. A negative

w

indicates that

\hat{y}

drops as

x

rises.

Bias ( $b$ ): The point where the line crosses the vertical axis when

x = 0

. Without a bias term, the model would be constrained to always pass through

(0, 0)

Ordinary Least Squares vs. Gradient Descent

There are two primary mathematical approaches for finding the optimal weights:

Optimization Choice

Small to Medium Datasets

Ordinary Least Squares (Analytical)

Solves directly via Normal Equation:
beta = (X^T X)^(-1) X^T y

Large / Streaming Datasets

Gradient Descent (Iterative)

Updates weights step-by-step:
w = w - alpha * dLoss/dw

1. Normal Equation (Closed-form): Computes the exact minimum directly in one matrix operation. Very fast for smaller datasets, but matrix inversion scales with

\mathcal{O}(p^3)

complexity as features grow.

2. Gradient Descent (Iterative): Takes gradual downhill steps along the convex error surface. Scales effectively to millions of samples and dimensions.

Key Assumptions & Boundary Conditions

Linear regression produces reliable estimates when the following core assumptions are satisfied:

Assumption	Description	What Happens If Violated?
Linearity	The link between $X$ and $Y$ is straight and additive	Model underfits nonlinear curves
Homoscedasticity	Residual variance is uniform across all values of $X$	Confidence intervals and p-values become invalid
Independence	Errors are uncorrelated across observations	High false-positive rates on time-series data
Normality	Residuals follow a standard normal distribution	Hypothesis testing results become inaccurate
No Multicollinearity	Input features are not highly correlated with each other	Parameter weights fluctuate wildly

Regression vs. Classification

The key distinction lies in the nature of the target variable:

Dimension	Linear Regression	Classification (e.g., Logistic Regression)
Target Type	Continuous numerical	Discrete categories / classes
Example Outputs	House price, battery life, salary	Spam / Not Spam, Benign / Malignant
Output Range	$(-\infty, +\infty)$	Probabilities bounded in $[0, 1]$
Function Shape	Straight line ( $\hat{y} = wx + b$ )	S-shaped curve ( $\sigma(wx + b)$ )
Loss Function	Mean Squared Error (MSE)	Binary Cross-Entropy (Log Loss)

Simple vs. Multiple Linear Regression

Simple Linear Regression (One input variable):

\hat{y} = wx + b

Multiple Linear Regression (Multiple input variables):

\hat{y} = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b

In vector dot-product notation:

\hat{y} = \mathbf{w}^T \mathbf{x} + b

For instance, estimating a house price rarely depends on square footage alone; it also depends on number of bedrooms, distance to schools, and property age. Each feature receives its own distinct weight $w_i$ .

Key Takeaways

Linear Regression models the straight-line relationship between inputs and a continuous target by finding parameters that minimize the sum of squared prediction errors.

The fundamental equation:

\boxed{\hat{y} = wx + b}

The learning loop:

\text{Data} \longrightarrow \text{Prediction} \longrightarrow \text{Loss (MSE)} \longrightarrow \text{Update Parameters} \longrightarrow \text{Trained Model}

What to Learn Next

Linear Regression

→

Cost Functions

→

Gradient Descent

→

Regularization (Ridge/Lasso)

→

Logistic Regression