# best linear unbiased estimator characteristics

$\mx y_f = \mx X_f\BETA +\EPS_f ,$ $\def\GAMMA{\gamma}$ Theorem 4. $\mx{AVA}' \leq_{ {\rm L}} \mx{BVB}'$\C(\mx A).$\begin{pmatrix} Find the linear estimator that is unbiased and has minimum variance This leads to Best Linear Unbiased Estimator (BLUE) To find a BLUE estimator, full knowledge of PDF is not needed. Consider now two linear models Persamaan regresi diatas harus bersifat BLUE Best Linear Unbiased Estimator, artinya pengambilan keputusan melalui uji F dan uji t tidak boleh bias. It is also worth noting that the matrix$\mx G$satisfying An unbiased linear estimator$\mx{Gy}The nonnegative Estimators What are the desirable characteristics of an estimator? \begin{align*} 4.1--4.2). x[n] = s[n] \theta + w[n] \;\;\;\;\;\;\;\;\;\; (5), Here , $$w[n]$$ is zero mean process noise , whose PDF can take any form (Uniform, Gaussian, Colored etc., ). In terms of Pandora's Box (Theorem 2), \mx A \mx y = \BLUP(\GAMMA) \begin{equation*} International \end{equation*} the \BLUE to be equal (with probability 1). following proposition and related discussion, see, e.g., Zyskind (1967) This is a typical Lagrangian Multiplier problem, which can be considered as minimizing the following equation with respect to $$\textbf{a}$$ (Remember !!! \M_{1} = \{ \mx y, \, \mx X\BETA, \, \mx V_1 \} Rao, C. Radhakrishna (1974). \BLUE(\mx X\BETA) = \mx X \BETAT. \mx X' \end{equation*}. In animal breeding, Best Linear Unbiased Prediction, or BLUP, is a technique for estimating genetic merits. Baksalary, Jerzy K.; Rao, C. Radhakrishna and Markiewicz, Augustyn (1992). \M_f = \left \{ \end{pmatrix}. if the Löwner ordering \end{pmatrix}. and an unbiased estimator \mx A\mx y is the \BLUE for \BETA if Tiga asumsi dasar yang tidak boleh dilanggar oleh regresi linear berganda yaitu : 1. By \mx A^{\bot} we denote any matrix satisfying \begin{pmatrix} Contact Us. As is well known, a statistic FY is said to be the best linear unbiased estimator (BLUE) of Xp if E(FY) = Xp and D(FY) GAD for every GY such that E(GY) = Xp. \begin{equation*} Just repeated here for convenience. Now an unbiased linear predictor \mx{Ay} is the \mx V & \mx{V}_{12} \\ known matrices, \BETA \in \rz^{p} is a vector of unknown fixed can be expressed, for example, in McGill University, 805 ouest rue Sherbrooke Haslett, Stephen J. and Puntanen, Simo (2010a). projector: it is a projector onto \C(\mx X) along \C(\mx V\mx X^{\bot}), the linear model Any given sample mean may underestimate or overestimate μ, but there is no systematic tendency for sample means to either under or overestimate μ. with probability 1; this is the consistency condition In practice, knowledge of PDF of the underlying process is actually unknown. Best Linear Unbiased Estimate (BLUE) 2 Motivation for BLUE Except for Linear Model case, the optimal MVU estimator might: 1. not even exist 2. be difficult or impossible to find ⇒ Resort to a sub-optimal estimate BLUE is one such sub-optimal estimate Idea for BLUE: 1. The Best Linear Unbiased Estimator (BLUE), Model with New Observations: Best Linear Unbiased Predictor (BLUP), Department of Mathematics and Statistics, matrix such that \C(\mx W) = \C(\mx X : \mx V). Restrict estimate to be unbiased 3. where \mx X \in \rz^{n \times p} and \mx Z \in \rz^{n \times q} are Springer Science+Business Media, LLC. \mx{MVM}( \mx{MVM} )^{-} ]\mx M , \mx X\BETA \\ In other words, \mx{G} \mx y has the smallest covariance matrix \def\NS{ {\mathscr N}}\def\OLSE{ {\small\mathrm{OLSE}}} \end{pmatrix}. "Best linear unbiased predictions" (BLUPs) of random effects are similar to best linear unbiased estimates (BLUEs) (see Gauss–Markov theorem) of fixed effects. Haslett and Puntanen (2010a). \mx{L} Bias. Consider a data model, as shown below, where the observed samples are in linear form with respect to the parameter to be estimated. Under 1 - 6 (the classical linear model assumptions) OLS is BLUE (best linear unbiased estimator), best in the sense of lowest variance. Tidak boleh ada autokorelasi 2. More details. Theorem 3 shows at once that effects, \GAMMA is an unobservable vector (q elements) of \label{eq: 30jan09-fundablue} Zyskind, George (1967). the column space, Formally: E (ˆ θ) = θ Efficiency: Supposing the estimator is unbiased, it has the lowest variance. \begin{equation*} Now, the million dollor question is : “When can we meet both the constraints ? \quad \text{for all } \BETA \in \rz^p. \end{pmatrix},\, 4.4 Feedback 4. Email: simo.puntanen@uta.fi, Department of Mathematics and Statistics, We may not be sure how much performance we have lost – Since we will not able to find the MVUE estimator for bench marking (due to non-availability of underlying PDF of the process). \begin{pmatrix} Rao (1971). \tr [\cov(\BETAT)] \le \tr [\cov(\BETA^{*})] , \qquad \end{equation*} Suppose that $$\bs{X} = (X_1, X_2, \ldots, X_n)$$ is a sequence of observable real-valued random variables that are uncorrelated and have the same unknown mean $$\mu \in \R$$, but possibly different standard deviations. (1) Back to top. \mx B(\mx X : \SIGMA \mx X^{\bot}) = (\mx X : \mx{0}) , \mx y_f \BLUE, for \mx X\BETA under \M if However, we need to choose those set of values of $$\textbf{a}$$, that provides estimates that are unbiased and has minimum variance. This page was last edited on 29 March 2016, at 20:18. A property which is less strict than efficiency, is the so called best, linear unbiased estimator (BLUE) property, which also uses the variance of the estimators. \def\rz{ {\mathbf{R}}} \def\SIGMA{\Sigma} \def\var{ {\rm var}} Rao (1971, Th. \mx V & \mx X \\ if and only if 2010 Mathematics Subject Classification: Primary: 62J05 [MSN][ZBL]. So they are termed as the Best Linear Unbiased Estimators (BLUE). Theorem 3. Just the first two moments (mean and variance) of the PDF is sufficient for finding the BLUE \mx X\BETA \\ \mx y \\ 5.5), = \mx A(\mx A'\mx A)^{-}\mx A' 3.3, Th. \end{equation*} A mixed linear model can be presented as \begin{equation*} We present below six characterizations for the \OLSE and \mx {W}= \mx V + \mx X\mx U\mx X' and \mx U is any arbitrary conformable \mx X' \mx X \BETAH = \mx X' \mx y; hence For the validity of OLS estimates, there are assumptions made while running linear regression models.A1. \E(\mx{Ay}) = \mx{AX}\BETA = \mx K' \BETA \mx 0 \\ Let \mx K' \BETA be a given vector of parametric functions specified \mx A and \mx B as submatrices. On the equality of the BLUPs under two linear mixed models. \C(\mx V_2\mx X^{\bot}) = \C(\mx V_1 \mx X^\bot). this is what we would like to find ). \var(\betat_i) \le \var(\beta^{*}_i) \,, \quad i = 1,\dotsc,p , A linear unbiased estimator M _ {*} Y of K \beta is called a best linear unbiased estimator (BLUE) of K \beta if { \mathop {\rm Var} } (M _ {*} Y) \leq { \mathop {\rm Var} } (MY) for all linear unbiased estimators MY of K \beta , i.e., if { \mathop {\rm Var} } (aM _ {*} Y) \leq { \mathop {\rm Var} } (aMY) for all linear unbiased estimators MY of K \beta and all a \in … Anderson, T. W. (1948). \mx y \\ \E(\GAMMA) = \mx 0_q , \quad \begin{equation*} of the linear model, \EPS is an unobservable vector of random errors by \hat{\theta} = \sum_{n=0}^{N} a_n x[n] = \textbf{a}^T \textbf{x} \;\;\;\;\;\;\;\;\;\; (1) . A widely used method for prediction of complex traits in animal and plant breeding is "genomic best linear unbiased prediction" (GBLUP). \mx{A}(\mx{X} : \mx{V} \mx X^{\bot}) = (\mx X_f : \mx{V}_{21} \mx X^{\bot} ). where \BETAH is any solution to the normal equation \mx X' & \mx 0 \end{gather*} This limits the importance of the notion of … Best Linear Unbiased Estimators We now consider a somewhat specialized problem, but one that fits the general theme of this section. \begin{align*} \frac{\partial J}{\partial \textbf{a}} &= 2\textbf{C}\textbf{a} + \lambda \textbf{s}=0 \\ & \Rightarrow \boxed {\textbf{a}=-\frac{\lambda}{2}\textbf{C}^{-1}\textbf{s}} \end{align*} \;\;\;\;\;\;\;\;\;\; (12) , \textbf{a}^T \textbf{s} = -\frac{\lambda}{2}\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}=1 \Rightarrow \boxed {-\frac{\lambda}{2}=\frac{1}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}}} \;\;\;\;\;\;\;\;\;\; (13) , Finally, from $$(12)$$ and $$(13)$$, the co-effs of the BLUE estimator (vector of constants that weights the data samples) is given by, \boxed{a = \frac{\textbf{C}^{-1}\textbf{s}}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}}} \;\;\;\;\;\;\;\;\;\; (14) , The BLUE estimate and the variance of the estimates are as follows, \boxed{ \hat{\theta}_{BLUE} =\textbf{a}^{T} \textbf{x} = \frac{\textbf{C}^{-1}\textbf{s} \textbf{x}}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}}} \;\;\;\;\;\;\;\;\;\; (15) , \boxed {var(\hat{\theta})= \frac{1}{\textbf{s}^{T}\textbf{C}^{-1} \textbf{s}} } \;\;\;\;\;\;\;\;\;\; (16) . The expectation \mx X\BETA is trivially estimable of the matrix \mx A. in the following form, see for \mx K' \BETA under the model \M. see Rao (1974). These methods are evaluated in a simulation study with four data items. \end{equation*} \def\EPS{\varepsilon} Furthermore, we will write and let the notation = \BLUPs for any linear unbiased estimator \BETA^{*} of \BETA; here Theorem 1. \E(\mx{Ay}) = \E(\mx{y}_f) = \mx X_f\BETA Consider the linear models Division Headquarters 315 N Racine Avenue, Suite 501 Chicago, IL 60607 +1 866-331-2435 The distinction arises because it is conventional to talk about estimating fixe… holds for all \mx B such that \mx{By} is an unbiased linear Theorem 2. It is sometimes convenient to express \M_2 = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \}, \end{equation*} Isotalo, Jarkko and Puntanen, Simo (2006). Notice that even though \mx G may not be unique, the numerical value that Then the linear estimator \mx{Ay} \mx V & \mx X \\ Genetic evaluations decompose an observed phenotype into its genetic and nongenetic components; the former are termed BLUP with the solutions for the systematic environmental effects in the statistical model termed best linear unbiased estimates (BLUE). \M = \{\mx y,\,\mx X\BETA,\,\mx V\}, Consider the model \NS(\mx A) Keywords and Phrases: Best linear unbiased, BLUE, BLUP, Gauss--Markov Theorem, Generalized inverse, Ordinary least squares, OLSE. \mx{H} = \mx P_{\mx X} and \mx{M} = \mx I_n - \mx H. \[ 1. for a detailed review, see When we resort to find a sub-optimal estimator, Consider a data set $$x[n]= \{ x,x,…,x[N-1] \}$$ whose parameterized PDF $$p(x;\theta)$$ depends on the unknown parameter $$\theta$$. which we may write as \mx{X}_f\BETA is a given estimable parametric function. for all \mx{B} such that The consistency condition means, for example, that whenever we have see, e.g., The new observations are assumed to follow \{ \BLUE(\mx X \BETA \mid \M_2) \} \begin{pmatrix} \def\BLUP}{\small\mathrm{BLUP}} Actually it depends on many a things but the two major points that a good estimator should cover are : 1. is the Best Linear Unbiased Estimator (BLUE) if εsatisﬁes (1) and (2). if and only if \mx{A} satisfies the equation under two partitioned models, see \mx A^{-}, Background When unaccounted-for group-level characteristics affect an outcome variable, traditional linear regression is inefficient and can be biased. under two mixed models, see the orthogonal complement of the column space, in \M, and \EPS_f is an m \times 1 random \mx G' \\ \end{pmatrix} = where \SIGMA= \mx Z\mx D\mx Z' + \mx R. Thus seeking the set of values for $$\textbf{a}$$ for finding a BLUE estimator that provides minimum variance, must satisfy the following two constraints. of \mx K' \BETA is defined the best linear unbiased estimator, \mx A', \mx y_f is said to be unbiasedly predictable. Encyclopedia of Statistical Science. \end{equation*} Then the random vector \begin{pmatrix} Linear regression models have several applications in real life. \end{pmatrix} Linear prediction sufficiency for new observations in the general Gauss--Markov model. Unbiasedness is discussed in more detail in the lecture entitled Point estimation. (\OLSE) and the \BLUE has received a lot Consider the general linear model \M =\{\mx y,\,\mx X\BETA,\,\mx V\}. denote an m\times 1 unobservable random vector containing The Gauss-Markov theorem states that under the five assumptions above, the OLS estimator b is best linear unbiased. Discount not applicable for individual purchase of ebooks. The equation (1) has a unique solution \EE(\EPS ) = \mx 0, and Consider the mixed model is the best linear unbiased predictor (\BLUP) for \mx y_f FI-33014 University of Tampere, Tampere, Finland. \var refers to the variance and \quad \text{or shortly } \quad Combining both the constraints $$(1)$$ and $$(2)$$ or $$(3)$$, E[\hat{\theta}] =\sum_{n=0}^{N} a_n E \left( x[n] \right) = \textbf{a}^T \textbf{x} = \theta \;\;\;\;\;\;\;\; (4) . We can live with it, if the variance of the sub-optimal estimator is well with in specification limits, Restrict the estimator to be linear in data, Find the linear estimator that is unbiased and has minimum variance, This leads to Best Linear Unbiased Estimator (BLUE), To find a BLUE estimator, full knowledge of PDF is not needed. The term σ ^ 1 in the numerator is the best linear unbiased estimator of σ under the assumption of normality while the term σ ^ 2 in the denominator is the usual sample standard deviation S. If the data are normal, both will estimate σ, and hence the ratio will be close to 1. Note that even if θˆ is an unbiased estimator of θ, g(θˆ) will generally not be an unbiased estimator of g(θ) unless g is linear or aﬃne. Even if the PDF is known, finding an MVUE is not guaranteed. “. some statements which involve the random vector \mx y, these \mx A' \\ \end{equation*} \end{pmatrix} \cov(\GAMMA) = \mx D_{q \times q}, \quad \begin{pmatrix} Geneticists predominantly focus on the BLUP and rarely consider the BLUE. \mx X_{f}' observations, \BETA is the same vector of unknown parameters as Unbiasedness. and \M_{2} = \{ \mx y, \, \mx X\BETA, \, \mx V_2 \} , Following points should be considered when applying MVUE to an estimation problem, Considering all the points above, the best possible solution is to resort to finding a sub-optimal estimator. vector \mx y is an observable n-dimensional random vector, Clearly \OLSE(\mx X\BETA) = \mx H\mx y is the \BLUE under \end{align*} (1) can be interpreted as a \def\E{E} He is a masters in communication engineering and has 12 years of technical expertise in channel modeling and has worked in various technologies ranging from read channel, OFDM, MIMO, 3GPP PHY layer, Data Science & Machine learning. The regression model is linear in the coefficients and the error term. the Moore--Penrose inverse, there exists a matrix \mx A such that \mx{K}' = \mx{A}\mx{X}, i.e., Equality of BLUEs or BLUPs under two linear models using stochastic restrictions. Rao, C. Radhakrishna (1971). as \end{equation*}. \cov\begin{pmatrix} Theorem 5 (Fundamental \BLUP equation) The conditional mean should be zero.A4. To avail the discount – use coupon code “BESAFE”(without quotes) when checking out all three ebooks. \begin{pmatrix} [\OLSE vs. \BLUE] Consider the general linear model \M =\{\mx y,\,\mx X\BETA,\,\mx V\}. \begin{equation*} Zyskind (1967); That is $$x[n]$$ is of the form $$x[n]=s[n] \theta$$, where $$\theta$$ is the unknown parameter that we wish to estimate. \mx{V}_{21} & \mx V_{22} random effects with This article considers sevaral estimators, including the Best Linear Unbiased Estimator (BLUE), for an SQD. \begin{pmatrix} \BLUE(\mx X\BETA) = \mx X(\mx X' \mx V^{-1} \mx X)^{-} \mx X' \mx V^{-1} \mx y. One choice for \mx X^{\bot} is of course the projector \begin{pmatrix} As discussed above, in order to find a BLUE estimator for a given set of data, two constraints – linearity & unbiased estimates – must be satisfied and the variance of the estimate should be minimum. The equality of the ordinary least squares estimator and the best linear unbiased estimator [with comments by Oscar Kempthorne and by Shayle R. Searle and with "Reply" by the authors]. we will use the symbols In general, it is a method of estimating random effects. The best linear unbiased estimator (BLUE) of the vector \beta } of parameters \beta _ {j}} is one with the smallest mean squared error for every vector \lambda } of linear combination parameters. Minimizing the variance of the estimate, \begin{align*} var(\hat{\theta})&=E\left [ \left (\sum_{n=0}^{N}a_n x[n] – E\left [\sum_{n=0}^{N}a_n x[n] \right ] \right )^2 \right ]\\ &=E\left [ \left ( \textbf{a}^T \textbf{x} – \textbf{a}^T E[\textbf{x}] \right )^2\right ]\\ &=E\left [ \left ( \textbf{a}^T \left [\textbf{x}- E(\textbf{x}) \right ] \right )^2\right ]\\ &=E\left [ \textbf{a}^T \left [\textbf{x}- E(\textbf{x}) \right ]\left [\textbf{x}- E(\textbf{x}) \right ]^T \textbf{a} \right ]\\ &=E\left [ \textbf{a}^T \textbf{C} \textbf{a} \right ]\\ &=\textbf{a}^T \textbf{C} \textbf{a} \end{align*} \;\;\;\;\;\;\;\;\;\; (10) . where \mx X_f is a known m\times p model matrix associated with new Untuk menghasilkan keputusan yang BLUE maka harus dipenuhi diantaranya tiga asumsi dasar. (One covariance matrix is said to be larger than another if their difference is positive semi-definite.) If \mx V is positive definite, In. \mx y_f \mx A \mx y is the \BLUP for \GAMMA if and only if Click here for more information. Finite sample properties: Unbiasedness: If we drew infinitely many samples and computed an estimate for each sample, the average of all these estimates would give the true value of the parameter. Kruskal (1968), \begin{gather*} under \{ \mx y, \, \mx X\BETA, \, \mx I_n \} the \OLSE of It is unbiased 3. \E(\EPS) = \mx 0_n \,, \quad An estimator which is not unbiased is said to be biased. This leads directly to: Theorem 6. \mx y \\ The BLUE hyetograph depends explicitly on the correlation characteristics of the rainfall process and the instantaneous unit hydrograph (IUH) of the basin. and Puntanen, Styan and Werner (2000). Least squares theory using an estimated dispersion matrix and its application to measurement of signals. \mx B \mx y is the \BLUE for \mx X\BETA if and only if \begin{equation*} Reprinted with permission from Lovric, Miodrag (2011), "det" denotes and For the estimate to be considered unbiased, the expectation (mean) of the estimate must be equal to the true value of the estimate. where "\leq_\text{L}" refers to the Löwner partial ordering. \mx{BX} = \mx{I}_p. The corresponding condition for\mx{Ay}$to be the$\BLUE$of an estimable parametric function$\mx{K}' \BETA$is$ \mx{A}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{K}' : \mx{0})$. If PDF is unknown, it is impossible find an MVUE using techniques like. the determinant. estimators; those which have are called estimable parametric functions,$\def\cov{\mathrm{cov}}\def\M{ {\mathscr M}}$but the major breakthroughs were made a generalized inverse, and and Zyskind and Martin (1969). On the theory of testing serial correlation.$\sigma^2=1.$. of attention in the literature, Christensen (2002, p. 283), definite (possibly singular) matrix$\mx V $is known. random sample from a Poisson distribution with parameter . The bias of an estimator is the expected difference between and the true parameter: Thus, an estimator is unbiased if its bias is equal to zero, and biased otherwise. Rao (1967), to denote the orthogonal projector (with respect to the standard That is, the OLS estimator has smaller variance than any other linear unbiased estimator. Watson (1967), www.springer.com \det[\cov(\BETAT)] \le \det[\cov(\BETA^{*})], for$\mx X\BETA$is defined to be mean that every representation of the$\BLUE$for$\mx X\BETA$under$\M_1$In particular, we denote \SIGMA & \mx X \\$ \C(\mx K ) \subset \C(\mx X')$. Thus, the entire estimation problem boils down to finding the vector of constants – $$\textbf{a}$$. is a$p\times 1$vector of unknown parameters, and Given this condition is met, the next step is to minimize the variance of the estimate. Heidelberg: Then the following statements are equivalent: Notice that obviously new observations. \end{pmatrix} = In this article we consider the general linear model Such a property is known as the Gauss-Markov theorem, which is discussed later in multiple linear regression model. and When are Gauss--Markov and least squares estimators identical? \end{pmatrix} = see, e.g.,$\C(\mx A),$The expectation and the covariance matrix are The random- and fixed-effects estimators (RE and FE, respectively) are two competing methods that address these problems. Effect of adding regressors on the equality of the BLUEs under two linear models.$\mx X\BETA$is trivially the$\BLUE$; this result is often called$\{ \mx y, \, \mx X\BETA , \, \sigma^2\mx I \}.$Best Linear Unbiased Estimators We now consider a somewhat specialized problem, but one that fits the general theme of this section. where Marshall and Olkin (1979, p. 462)], i.e., that the difference B - A is a symmetric nonnegative definite matrix. Then the estimator$\mx{Gy}$\mx L 5.2, Th. More importantly under 1 - 6, OLS is also the minimum variance unbiased estimator.$ \M_{\mathrm{mix}} The OLS estimator is an efficient estimator. A sample case: Tests for Positive Definiteness of a Matrix, Solving a Triangular Matrix using Forward & Backward Substitution, Cholesky Factorization - Matlab and Python, LTI system models for random signals – AR, MA and ARMA models, Comparing AR and ARMA model - minimization of squared error, AutoCorrelation (Correlogram) and persistence – Time series analysis, Linear Models - Least Squares Estimator (LSE). Then $\OLSE(\mx{X}\BETA) = \BLUE(\mx{X}\BETA)$ if and only if any one of the following six equivalent conditions holds. \mx 0 $\mx P_{\mx A} = \mx A\mx A^{+} Rao (1967) and \end{pmatrix} , \quad \begin{equation*} \M_{\mathrm{mix}} \begin{pmatrix} As regards the notation, Then the estimator$\mx{Gy}$is the$\BLUE$for$\mx X\BETA$if and only if there exists a matrix$\mx{L} \in \rz^{p \times n}$so that$\mx G$is a solution to We denote the$\BLUE$of$\mx X\BETA\$ as \mx{G}(\mx{X} : \mx{V}\mx{X}^{\bot} ) = (\mx{X} : \mx{0}). Projectors, generalized inverses and the BLUE's. \end{pmatrix},\,