Some remarks on linear least squares fits with errors in both coordinates

• Published in: Acta geodaetica et geophysica Hungarica 2012-12, Vol.47 (4), p.441-452
• Main Author: Banyai, L
• Format: Article
• Language: English
• Subjects: Earth and Environmental Science; Earth Sciences; Geophysics/Geodesy
• ISSN: 1217-8977; 1587-1037
• DOI: 10.1556/AGeod.47.2012.4.5

The exact least squares line fit with errors in both coordinates (Reed 1992) is investigated together with the approximate solution based on the formalism of the linear Gauss-Helmert model or the unified adjustment approach of the classical textbook by Mikhail (1976). The similarities and the differences are described in details. In spite of the small differences the exact solution is preferable and the calculations are simpler. This paper does not deal with the errors-in-variables (EIV) models solved by the total least squares (TLS) principle, since the exact line fit solution is used to validate this general approach, which is basically designed to solve more sophisticated nonlinear tasks. In the most general case the fit of Person’s data with York’s weights is iteratively solved starting with the arbitrary zero initial value of the slope. The test computation with different but systematically chosen weights proved that in special cases — e.g. the weighted least squares sum of the distances between the data points and the estimated line is minimised — there is no need for iterations at all. It is shown that methods described by Detrekői (1991) and Závoti (2012) are special cases of the general exact solutions. Reed (1992) derived the variances of the slope and intercept parameters without their covariance. The simple linear estimation of variance-covariance matrix of the exact solution is also demonstrated. The importance of the stochastic models coupled with exact solution is also demonstrated.