./articles/ Machine Learning Essentials: Linear vs. Logistic Regression

Linear regression and logistic regression are two fundamental machine learning algorithms that form the backbone of statistical modeling and predictive analytics. While both are regression techniques, they serve different purposes and are applied to different types of problems.

This comprehensive guide explores both algorithms in detail, covering their mathematical foundations, key differences, practical applications, and implementation considerations.

1. Overview and Key Differences

What are Linear and Logistic Regression?

Linear Regression is a statistical method used to model the relationship between a dependent variable and one or more independent variables by fitting a linear equation to observed data. It assumes a linear relationship between the input features and the continuous target variable.

Logistic Regression, despite its name, is actually a classification algorithm that uses the logistic function to model the probability of binary or categorical outcomes. It transforms the linear combination of features using the sigmoid function to produce probabilities between 0 and 1.

The fundamental equation for Linear Regression is:

\[y = \beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_nx_n + \epsilon\]

The fundamental equation for Logistic Regression is:

\[P(y=1) = \frac{1}{1 + e^{-(\beta_0 + \beta_1x_1 + \beta_2x_2 + ... + \beta_nx_n)}}\]

When to Use Each Algorithm?

Use Linear Regression when:

• Your target variable is continuous (e.g., price, temperature, height)
• You want to predict numerical values
• The relationship between features and target appears linear
• You need to understand feature importance and interpretability

Use Logistic Regression when:

• Your target variable is categorical (binary or multinomial)
• You want to predict probabilities of class membership
• You need a probabilistic output for decision making
• You require model interpretability for classification tasks

If you want to keep reading, I’ll now dive deeper into each strategy, then compare them, and finally explore some advanced strategies based on them.

2. Linear Regression Deep Dive

Mathematical Foundation of Linear Regression

Linear regression finds the best-fitting straight line through a set of data points by minimizing the sum of squared residuals. The mathematical foundation involves several key components:

Simple Linear Regression (One Feature):

\[y = \beta_0 + \beta_1x + \epsilon\]

Where:

• \(y\) = dependent variable (target)
• \(x\) = independent variable (feature)
• \(\beta_0\) = y-intercept
• \(\beta_1\) = slope (coefficient)
• \(\epsilon\) = error term

Multiple Linear Regression:

\[y = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots + \beta_nx_n + \epsilon\]

This equation extends simple linear regression to multiple features. It says that the predicted value \(y\) is given by an intercept \(\beta_0\) plus the sum of each feature \(x_j\) multiplied by its coefficient \(\beta_j\), and an error term \(\epsilon\) that captures noise or factors not explained by the model. Each coefficient \(\beta_j\) represents the average change in \(y\) when \(x_j\) increases by one unit, keeping all other variables fixed.

Matrix Form:

\[\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\epsilon}\]

This is the same model rewritten in compact matrix notation. Here, \(\mathbf{X}\) is a matrix where each row corresponds to one training example and each column to a feature, with the first column being all ones for the intercept term. Multiplying \(\mathbf{X}\) by the parameter vector \(\boldsymbol{\beta}\) produces all predicted values at once. The vector \(\boldsymbol{\epsilon}\) contains the residuals — the differences between actual outputs and predictions — for all \(m\) observations.

In matrix form:

• \(\mathbf{y}\): column vector of observed outputs (\(m \times 1\))
• \(\mathbf{X}\): design matrix of features with an intercept column (\(m \times (n+1)\))
• \(\boldsymbol{\beta}\): column vector of parameters (\((n+1) \times 1\))
• \(\boldsymbol{\epsilon}\): residuals vector (\(m \times 1\))

This compact representation makes it possible to apply linear algebra methods to solve for \(\boldsymbol{\beta}\) efficiently.

Cost Function (Mean Squared Error):

\[J(\boldsymbol{\beta}) = \frac{1}{2m} \sum_{i=1}^{m} \big( h_{\boldsymbol{\beta}}(x^{(i)}) - y^{(i)} \big)^2\]

The cost function measures how far the model’s predictions \(h_{\boldsymbol{\beta}}(x^{(i)})\) are from the actual outputs \(y^{(i)}\), by averaging the squared differences across all examples. The division by \(2m\) is used for convenience, as it simplifies derivative expressions in optimization. Minimizing \(J(\boldsymbol{\beta})\) finds the parameters that make predictions as close as possible to actual values in the least squares sense.

Assumptions of Linear Regression

Linear regression relies on several key assumptions. Violations can lead to unreliable predictions and misleading interpretations. The table below summarizes each assumption, its meaning, how to check it, and practical examples:

Step-by-Step Example: Normal Equation (Closed-form Solution)

Derived by setting the gradient of the cost function to zero, this equation gives the exact parameters that minimize the MSE without iterative optimization. It is efficient for problems with a small to moderate number of features, as it uses matrix operations to compute the optimal solution in one step.

Let's solve a simple regression problem step-by-step using the normal equation. Suppose we have 3 training examples with 1 feature (\(x_1\)) and an intercept:

Step-by-Step Example: Gradient Descent

While the Normal Equation provides an exact solution, Gradient Descent offers an iterative approach that's more scalable for large datasets. Let's apply it to the same toy example to compare both methods.

We will use the same dataset as before, with 3 training examples and 1 feature (\(x_1\)). Here's the python code:

Convergence Analysis

The gradient descent algorithm iteratively updates the parameters using the formula:

\[\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} - \alpha \nabla J(\boldsymbol{\beta}^{(t)})\]

where \(\alpha\) is the learning rate and \(\nabla J\) is the gradient of the cost function.

Method Comparison

The same results were achieved using different methods.

Key Insights

Evaluation Metrics for Linear Regression

Evaluating linear regression models requires metrics that quantify how well the model predicts the target variable. The table below summarizes common metrics, their formulas, and when to use them:

3. Logistic Regression Deep Dive

Mathematical Foundation of Logistic Regression

Logistic regression uses the sigmoid (logistic) function to map any real-valued input to a value between 0 and 1, making it perfect for probability estimation and binary classification.

Sigmoid Function:

\[\sigma(z) = \frac{1}{1 + e^{-z}}\]

The sigmoid (or logistic) function maps any real-valued input \(z\) into the range \((0, 1)\), making it ideal for modeling probabilities. When \(z\) is large and positive, \(\sigma(z) \approx 1\); when \(z\) is large and negative, \(\sigma(z) \approx 0\); and when \(z = 0\), the output is exactly \(0.5\). Its smooth, S-shaped curve ensures small changes in \(z\) near zero cause significant changes in \(\sigma(z)\), which is useful for classification.

Linear Combination:

\[z = \beta_0 + \beta_1x_1 + \beta_2x_2 + \dots + \beta_nx_n\]

This is the weighted sum of input features used in logistic regression. Each feature \(x_j\) is multiplied by its corresponding coefficient \(\beta_j\), and \(\beta_0\) is the intercept (bias term). The value \(z\) is not yet constrained between 0 and 1 — it can be any real number — and will be passed through the sigmoid function to obtain a probability.

Probability of Positive Class:

\[P(y=1 \mid x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1x_1 + \dots + \beta_nx_n)}}\]

This applies the sigmoid function to the linear combination \(z\) to get the probability that the output class \(y\) equals 1, given the features \(x\). The exponent term flips sign depending on the direction of \(z\), producing probabilities near 1 when \(z\) is strongly positive and near 0 when \(z\) is strongly negative.

Odds and Log-Odds:

The odds of an event is the ratio between the probability of the event occurring and the probability of it not occurring:

\[Odds = \frac{P(y=1)}{P(y=0)} = \frac{P(y=1)}{1 - P(y=1)}\]

If \(P(y=1)\) is 0.75, the odds are \(0.75 / 0.25 = 3\), meaning the event is three times as likely to happen than not.

Taking the natural logarithm of the odds gives the log-odds (logit), which in logistic regression is a linear function of the input:

\[\text{logit}(p) = \ln\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1x_1 + \dots + \beta_nx_n\]

This equation shows that while probabilities themselves are not linear in the inputs, their log-odds are — making the model suitable for estimation with linear methods while still producing probabilistic outputs.

This shows that logistic regression is actually modeling the log-odds as a linear function of the features.

Maximum Likelihood Estimation

Unlike linear regression, logistic regression doesn't have a closed-form solution. Instead, it uses Maximum Likelihood Estimation (MLE) to find the optimal parameters. This means we look for the values of \(\boldsymbol{\beta}\) that make the observed training data most probable.

Likelihood Function:

For a binary classification problem, the likelihood of observing the data given the parameters is:

\[L(\boldsymbol{\beta}) = \prod_{i=1}^{m} P(y^{(i)} \mid x^{(i)};\boldsymbol{\beta})\]

This formula says the likelihood \( L(\boldsymbol{\beta}) \) is the product, over all \( m \) training examples, of the predicted probability of the actual observed class. If \( y^{(i)} = 1 \), the term is \( h_{\boldsymbol{\beta}}(x^{(i)}) \); if \( y^{(i)} = 0 \), it is \( 1 - h_{\boldsymbol{\beta}}(x^{(i)}) \). Multiplying these terms gives the joint probability of the dataset under the model.

Log-Likelihood Function:

Taking the logarithm makes the optimization easier:

\[\ell(\boldsymbol{\beta}) = \sum_{i=1}^{m} \left[ y^{(i)} \log(h_{\boldsymbol{\beta}}(x^{(i)})) + (1-y^{(i)})\log(1-h_{\boldsymbol{\beta}}(x^{(i)})) \right]\]

The log transforms the product into a sum, helping numerical stability and simplifying derivatives. When \( y^{(i)} = 1 \), only the first log term remains; when \( y^{(i)} = 0 \), only the second term remains. Adding these over all samples yields the log-probability of the data given \(\boldsymbol{\beta}\).

Cost Function (Negative Log-Likelihood):

\[J(\boldsymbol{\beta}) = -\frac{1}{m} \sum_{i=1}^{m} \left[ y^{(i)}\log(h_{\boldsymbol{\beta}}(x^{(i)})) + (1-y^{(i)})\log(1-h_{\boldsymbol{\beta}}(x^{(i)})) \right]\]

Maximizing the log-likelihood is equivalent to minimizing the negative log-likelihood. This is our cost function. The minus sign converts the maximization problem into a minimization problem suitable for gradient descent, and dividing by \( m \) gives the average cost per training example.

Gradient Descent Update:

The parameters are updated using gradient descent:

\[\beta_j := \beta_j - \alpha \frac{\partial J(\boldsymbol{\beta})}{\partial \beta_j}\]

Starting from initial values, each parameter \(\beta_j\) is adjusted in the opposite direction of the gradient of the cost function to reduce it. Here, \(\alpha\) is the learning rate, controlling step size.

Where the gradient is:

\[\frac{\partial J(\boldsymbol{\beta})}{\partial \beta_j} = \frac{1}{m}\sum_{i=1}^{m} \left( h_{\boldsymbol{\beta}}(x^{(i)}) - y^{(i)} \right) x_j^{(i)}\]

This derivative measures the average prediction error \((h_{\boldsymbol{\beta}}(x^{(i)}) - y^{(i)})\) multiplied by the \( j^{th} \) feature value \( x_j^{(i)} \). If the predictions are perfect, the gradient becomes zero and the parameters stop changing.

Step-by-Step Example: Logistic Regression with Gradient Descent

Let's walk through a simple logistic regression example to illustrate how the algorithm works in practice.

Problem: Predict whether a student passes (1) or fails (0) an exam based on the number of hours studied.

Dataset:

This step-by-step process demonstrates how logistic regression models the probability of a binary outcome and makes decisions based on a chosen threshold.

Evaluation Metrics for Logistic Regression

4. Detailed Comparison

Understanding the fundamental differences between linear and logistic regression is crucial for choosing the right algorithm for your specific problem. This comprehensive comparison examines mathematical foundations, practical considerations, and performance characteristics.

Mathematical Foundations Comparison

Hypothesis Functions and Transformations

While both use linear combinations of features, logistic regression applies a non-linear transformation (sigmoid) that fundamentally changes the problem from regression to classification.

Output Characteristics and Interpretation

Cost Functions and Optimization

Parameter Estimation Methods

Decision Boundaries and Geometric Interpretation

Comprehensive Assumptions Analysis

Performance Characteristics and Trade-offs

Computational Efficiency

Robustness and Sensitivity Analysis

Advantages and Limitations

Linear Regression: Detailed Analysis

Logistic Regression: Detailed Analysis

5. Practical Considerations

Successfully implementing linear and logistic regression requires moving beyond theory into careful data preparation, feature engineering, and rigorous model validation. This section provides a comprehensive guide to the practical steps that distinguish a mediocre model from a high-performing, reliable one.

Data Preprocessing Pipeline

Essential Preprocessing Steps for Both Algorithms

Algorithm-Specific Preprocessing Requirements

Advanced Feature Engineering Techniques

Domain-Specific Feature Creation

Model Validation and Selection Strategies

Cross-Validation Methodologies

Regularization Techniques

Regularization adds a penalty to the cost function based on the size of the coefficients, preventing overfitting and improving generalization.

Hyperparameter Tuning Strategies

Performance Optimization and Diagnostics

Computational Efficiency Considerations

Model Diagnostics and Debugging

6. Implementation Examples

Linear Regression: From Scratch Implementation

Understanding linear regression requires implementing it from first principles. This implementation demonstrates the mathematical concepts in practice.

Logistic Regression: From Scratch Implementation

Logistic regression implementation from scratch demonstrates the sigmoid function, maximum likelihood estimation, and gradient descent optimization.

Advanced Topics and Extensions

Polynomial Regression: Mathematical Foundation

Polynomial regression extends linear regression by adding polynomial terms of the features (e.g., squared, cubed) to the model. This transformation allows the model to fit non-linear patterns in the data while still leveraging the efficient estimation techniques of linear regression.

Mathematical Formulation:

For a single feature, a polynomial regression of degree \(d\) is defined as:

\[y = \beta_0 + \beta_1x + \beta_2x^2 + \beta_3x^3 + \dots + \beta_dx^d + \epsilon\]

This equation models the target \(y\) as a \(d\)-degree polynomial function of the feature \(x\). The model learns coefficients \(\beta_j\) for each power of \(x\), allowing it to fit curves instead of just straight lines. The \(\epsilon\) term represents the irreducible error.

For multiple features, the model also includes interaction terms:

\[y = \beta_0 + \sum_{i=1}^{n}\beta_i x_i + \sum_{i=1}^{n}\sum_{j=i}^{n}\beta_{ij}x_i x_j + \dots + \epsilon\]

This form captures not only the polynomial effect of each feature (e.g., \(x_1^2\)) but also how features interact with each other (e.g., \(x_1x_2\)). This allows the model to fit complex, multi-dimensional surfaces.

Feature Transformation:

The core idea is to transform the original feature set into a new, higher-dimensional one that includes the polynomial terms. Polynomial regression is fundamentally a linear regression model applied to these transformed features.

\[\mathbf{X}_{\text{poly}} = [1, x_1, x_2, \dots, x_n, x_1^2, x_1x_2, \dots, x_n^2, x_1^3, \dots]\]

We create a new feature matrix, \(\mathbf{X}_{\text{poly}}\), where each row corresponds to an observation and each column represents a term in the polynomial expansion (e.g., \(x_1\), \(x_1^2\), \(x_1x_2\)).

Matrix Form:

Using the transformed features, the model is expressed in a familiar linear form:

\[\mathbf{y} = \mathbf{X}_{\text{poly}}\boldsymbol{\beta} + \boldsymbol{\epsilon}\]

This equation is identical in structure to multiple linear regression. The vector \(\mathbf{y}\) contains the target values, \(\mathbf{X}_{\text{poly}}\) is the transformed feature matrix, \(\boldsymbol{\beta}\) is the vector of coefficients to be learned, and \(\boldsymbol{\epsilon}\) is the vector of errors.

Cost Function:

The cost function is the Mean Squared Error (MSE), identical to that of linear regression, but applied to the polynomial features:

\[J(\boldsymbol{\beta}) = \frac{1}{2m}||\mathbf{X}_{\text{poly}}\boldsymbol{\beta} - \mathbf{y}||^2\]

This function calculates the sum of the squared differences between the predicted values (\(\mathbf{X}_{\text{poly}}\boldsymbol{\beta}\)) and the actual values (\(\mathbf{y}\)). The goal is to find the coefficient vector \(\boldsymbol{\beta}\) that minimizes this cost.

Normal Equation:

Because the model is linear in its parameters, we can use the Normal Equation to find the optimal \(\boldsymbol{\beta}\) analytically:

\[\boldsymbol{\beta} = (\mathbf{X}_{\text{poly}}^T\mathbf{X}_{\text{poly}})^{-1}\mathbf{X}_{\text{poly}}^T\mathbf{y}\]

This provides a direct, closed-form solution without needing iterative methods like gradient descent. It works by finding the projection of \(\mathbf{y}\) onto the column space of \(\mathbf{X}_{\text{poly}}\). However, this method involves inverting a matrix, which can be computationally expensive (\(O(n^3)\) where n is the number of features) and numerically unstable if the features are highly correlated.

Polynomial Regression: From Scratch Implementation

Multinomial Logistic Regression: Mathematical Foundation

Multinomial logistic regression, often called Softmax Regression, generalizes binary logistic regression to handle classification problems with more than two classes (K > 2). Instead of modeling a single probability for one class, it simultaneously models the probabilities for all K classes, ensuring they sum to one.

Softmax Function:

The softmax function is the cornerstone of this model, converting a vector of raw linear scores for each class into a valid probability distribution.

\[P(y=k \mid \mathbf{x}) = \frac{e^{\boldsymbol{\beta}_k^T \mathbf{x}}}{\sum_{j=1}^{K} e^{\boldsymbol{\beta}_j^T \mathbf{x}}}\]

For a given input \(\mathbf{x}\), we first compute a linear score \(\boldsymbol{\beta}_k^T \mathbf{x}\) for each class \(k\). The exponentiation \(e^{(\cdot)}\) makes all scores positive. Dividing by the sum of all exponentiated scores ensures that the final probabilities for all classes sum to 1. The class with the highest score will receive the highest probability.

Matrix Formulation:

To manage the parameters efficiently, we organize the coefficient vectors for all K classes into a single parameter matrix \(\mathbf{B}\).

\[\mathbf{B} = [\boldsymbol{\beta}_1, \boldsymbol{\beta}_2, \dots, \boldsymbol{\beta}_K]\]

If you have \(n\) features (plus an intercept), each \(\boldsymbol{\beta}_k\) is a vector of size \((n+1) \times 1\). The full parameter matrix \(\mathbf{B}\) will therefore have dimensions \((n+1) \times K\).

Linear Scores:

The linear scores (or logits) for all classes and all observations can be computed in a single matrix operation:

\[\mathbf{Z} = \mathbf{X}\mathbf{B}\]

Here, \(\mathbf{X}\) is the feature matrix (size \(m \times (n+1)\)) and \(\mathbf{B}\) is the parameter matrix. The resulting matrix \(\mathbf{Z}\) (size \(m \times K\)) contains the raw score \(z_{ik}\) for the \(i\)-th sample belonging to the \(k\)-th class.

Probability Matrix:

The softmax function is then applied to each row of the score matrix \(\mathbf{Z}\) to produce a matrix of probabilities:

\[P_{ik} = \frac{e^{z_{ik}}}{\sum_{j=1}^{K} e^{z_{ij}}}\]

Each element \(P_{ik}\) in the resulting probability matrix \(\mathbf{P}\) represents the model's predicted probability that sample \(i\) belongs to class \(k\). Each row of \(\mathbf{P}\) sums to 1.

Cross-Entropy Loss:

The cost function for multinomial logistic regression is the cross-entropy loss, which is the negative log-likelihood averaged over all samples.

\[J(\mathbf{B}) = -\frac{1}{m}\sum_{i=1}^{m}\sum_{k=1}^{K} y_{ik} \log(P_{ik})\]

Here, \(y_{ik}\) is a binary indicator from the one-hot encoded target matrix, which is 1 if sample \(i\) truly belongs to class \(k\) and 0 otherwise. This structure cleverly ensures that for each sample, the loss is simply the negative log of the probability assigned to the *correct* class. Minimizing this loss is equivalent to finding the parameters \(\mathbf{B}\) that maximize the probabilities of the true classes for all samples.

Gradient Computation:

The gradient of the cost function with respect to the parameters of a single class \(k\) is given by:

\[\frac{\partial J}{\partial \boldsymbol{\beta}_k} = \frac{1}{m}\mathbf{X}^T(\mathbf{P}_k - \mathbf{y}_k)\]

This elegant formula calculates the gradient by taking the average difference between the predicted probabilities (\(\mathbf{P}_k\)) and the true labels (\(\mathbf{y}_k\)) for class \(k\), weighted by the input features \(\mathbf{X}\). This gradient is then used in an iterative optimization algorithm like gradient descent to update the parameters \(\boldsymbol{\beta}_k\).

Multinomial Logistic Regression: From Scratch Implementation

Linear and logistic regression remain fundamental algorithms in machine learning due to their simplicity, interpretability, and effectiveness for many real-world problems. Understanding their mathematical foundations, assumptions, and practical considerations will help you make informed decisions about when and how to apply these powerful techniques in your data science projects.