Economics

Weighted Least Squares Estimator

Published Sep 8, 2024

### Weighted Least Squares Estimator

Definition of Weighted Least Squares Estimator

The Weighted Least Squares (WLS) estimator is a generalization of the ordinary least squares (OLS) method in regression analysis. Unlike OLS, which assumes equal variance among all observations, WLS assigns different weights to each data point to account for heteroscedasticity (i.e., non-constant variance) within the data. By assigning weights, WLS minimizes the weighted sum of squares of the residuals, giving more importance to observations with smaller variances.

  • Formula: The WLS estimator of the regression coefficients is obtained by solving the modified normal equations, taking into account the weights assigned to the observations.
  • Purpose: The main purpose of WLS is to improve the efficiency of the parameter estimates when heteroscedasticity is present, ensuring that more reliable and robust results are achieved in regression analysis.

Example

Consider a dataset on the relationship between advertising expenditure and sales revenue for a company. Suppose each data point represents a different region, and the variance of sales revenue (error variance) differs across regions due to varying market sizes. In this case, heteroscedasticity is present.

To perform WLS:

  1. Estimate Variances: Begin by estimating or obtaining the variance for each region’s sales revenue. This could involve statistical techniques or domain-specific knowledge.
  2. Assign Weights: Compute the weights as the inverse of the variances. For instance, if the variance of region A’s sales revenue is 4, its weight would be 1/4 = 0.25.
  3. Modify the Model: Form the weighted objective function and solve for the regression coefficients using the weighted normal equations.

By doing so, regions with higher variability in sales revenue have less influence on the regression estimates, leading to more efficient and unbiased results.

Why Weighted Least Squares Estimator Matters

Weighted Least Squares estimators are crucial for several reasons:

  • Handling Heteroscedasticity: WLS addresses the problem of heteroscedasticity, which violates one of the key assumptions of OLS. By accounting for non-constant variance, WLS provides more accurate parameter estimates.
  • Improving Efficiency: In the presence of heteroscedasticity, OLS estimates can be inefficient. WLS improves the efficiency of estimators by minimizing the weighted sum of squared residuals, resulting in better statistical properties.
  • Enhancing Robustness: WLS makes regression analysis more robust to outliers. Observations with larger variances (potential outliers) are down-weighted, which reduces their disproportionate influence on the model.

Frequently Asked Questions (FAQ)

How do you determine the appropriate weights in Weighted Least Squares?

In practice, weights can be determined through various methods:

  • Direct Measurement: Sometimes, variances can be directly observed or reasonably estimated from supplementary data or domain expertise.
  • Model-Based Estimation: In certain cases, a preliminary OLS regression can be conducted, and the residuals from this model can be used to estimate the heteroscedasticity structure, subsequently determining appropriate weights.
  • Iterative Methods: Algorithms such as Feasible Generalized Least Squares (FGLS) can iteratively estimate weights until convergence is achieved, balancing complexity with accuracy.

Are there limitations to using Weighted Least Squares Estimators?

Like any statistical method, WLS has its limitations:

  • Complexity: Implementing WLS requires accurate estimation of weights, which can be computationally intensive, particularly for large datasets or complex heteroscedastic patterns.
  • Assumption Sensitivity: WLS assumes that appropriate weights have been accurately determined. Incorrect weights can lead to biased estimates, undermining the advantages of WLS.
  • Data Requirements: To determine accurate weights, additional data or robust statistical methods are needed to estimate the error variances correctly.

Can Weighted Least Squares Estimators be used in non-linear regression models?

Yes, the concept of weighting observations can be extended to non-linear regression models. While the specifics of the estimation procedure may vary, the principle of weighting to account for heteroscedasticity remains applicable. In non-linear models, iterative algorithms like Non-Linear Least Squares (NLLS) can be adapted to include weights, ensuring that the model adequately addresses varying variances in the data.

Conclusion

The Weighted Least Squares estimator is a powerful tool in regression analysis, specifically designed to address the challenges posed by heteroscedasticity. By assigning different weights to observations based on their variances, WLS improves the efficiency and reliability of parameter estimates, making it a valuable method for analysts and researchers dealing with non-constant variance in their data.

Uncategorized