Lecture 2

Econ 496/895 Introduction to Design and Analysis of Economics Experiments Professor Daniel Houser Background A. Useful definitions and results from probability theory. 1. A “random experiment” is an experiment for which the outcome cannot be predicted with certainty, although the set of all possible outcomes can be described. 2. The sample space S of a random experiment is the set of all possible outcomes. 3. Any function X : S   is a “random variable.” Note that the choice of function X generally depends on the purpose of the experiment. For example, an experiment on brain activation one could look for activation in a single area or for the average over several areas or for certain activation paths. Ex: Tossing a six-sided die and observing the number of spots facing up is a random experiment. The sample space is S  1,2,3,4,5,6 and the random variable is X ( s )  s, the identity map. An alternative random variable would be X ( s )  1 if s  4 and X ( s )  0 otherwise, which provides an indicator for whether the realized outcome exceeded four.

4. Let A be a subset of S , A  S . Suppose P is a set function defined on the set of all subsets of S, P( A)  for all A  S . P is called a “probability” if and only if it satisfies the following. (i) (ii) (iii)

For all subsets A of S, P( A)  0, P ( S )  1, If A1 , A2 , ..., Ak are such that Am  An  , m  n, then P  A1  ...  Ak  = P( A1 )  P( A2 )  ...  P( Ak ), for k  .

Ex: In the die tossing example P(1)  P(2)  ...  P(6)  1/ 6, P(1  2)  1/ 3, and P([1  2]  [2  3])  0.5. 5. The “conditional probability” of A given that B has occurred is P( A  B ) P( A | B )  . P( B ) Ex: In the die tossing example, the probability of a “2” given that a “1” or “2” has been observed is P(2 |1 or 2)  P(2  [1 or 2]) / P(1 or 2) = (1/ 6) /(1/ 3)  0.5.

1

6. Events A and B are “independent” if and only if P( A  B )  P( A) P( B ). From the definition of conditional probability we see that if two events are independent then knowledge about the occurrence of one does not help to improve predictions with regard to the likelihood of the other. 7. Events A, B and C are mutually independent if they are pairwise independent and P( A  B  C )  P( A) P( B ) P(C ). This can be extended to four or more events easily: each pair, triplet, etc… must satisfy independence. Ex: Suppose three coins are tossed simultaneously but that these are trick coins and, with equal probability, one of the four outcomes occurs. {H,T,H}, {H,H,T},{T,T,T} or {T,H,H}, where H is heads, T is tails, and the order is {Coin 1, Coin 2, Coin 3}. Then it is easy to see that the realizations are pairwise independent, P(Coin 1=H|Coin 2 = T)=0.5, etc… although there is obviously dependence in their joint realization since P(H,H,H) is zero, not 1/8. 8.

From the definition of conditional probability one can deduce Bayes’ Formula: P( A) P( B | A) P( A | B )  . P( B ) This is often referred to as the posterior probability of A given B.

9. If X is a random variable, we will often denote its “probability density function” (pdf) by f  . If X is discrete, then f ( x )  P( X  x ). If X is continuous then P( X  A)   f ( x )dx. A pdf is nonnegative and integrates to unity over the A

sample space. 10. The “cumulative distribution function” (cdf) of a random variable X is defined by F ( x )  P ( X  x )   f ( x ). A cdf lies between zero and one and is X x

nondecreasing. 11. The “expected value” of a random variable X, E(X), is

 xf ( x ),

providing that the

X

value of this integral is finite. This is the weighted average of all possible values of X, where the weights are interpreted as the chance that the observation will occur. If the value of the integral is infinite, then the expectation is said not to exist. Expected values satisfy:  E(c) = c for every constant c.  E(cX)=cE(X) for constant c and rv X.  E(u(X)+v(X))=E(u(X)) + E(v(X)) for functions u and v.

2

12. The variance of a rv X is its expected squared deviation from its mean,  X2   ( X  E ( X )) 2 f ( x )dx. The positive square root of the variance is called the X

standard deviation. The variance is a measure of the dispersion of the distribution of X about its mean. Chebyshev’s inequality shows that the probability that a realization of X is within k  1 standard deviations of its mean is at least 1  1/ k 2 . Hence, at least 84% of the mass of any rv lies within 2.5 sd’s of its mean. 13. The covariance of two rv’s X and Y is Cov ( X , Y )  E ([ X   x ][Y   y ]). Their correlation coefficient is  xy 

Cov( X , Y )

 x y

. The correlation coef. lies between –1

and +1. Intuition about the meaning of covariance and correlation is gained by noting that the least squares regression of Y on X is y   y  

y ( x   x ). x

14. A rv X has a “normal” (or Gaussian) distribution if its pdf is  ( x   x )2  1 f ( x)  exp    ,    x  . 2 X2  2 X2  If X follows a normal distribution we write X ~ N (  x ,  X2 ). 

If X ~ N (  ,  2 ) then Z 

( X  )

~ N (0,1).

  The finite sum of normally distributed rv’s follows a normal distribution. B. Useful definitions and results from sampling distribution theory. 1. A “statistic” is a function of a random sample of realizations of random variables. The mean and variance of a sample are both statistics. Since statistics are functions of random variables, they are themselves random variables. 2. “Sampling distribution theory” is concerned with determining the distribution of statistics. 3. Example: Finding the sampling distribution of a function of random variables in a random sample. An unbiased four-sided die is cast twice with outcomes X1 and X2. Find the expected value and variance of Y = X1 + X2. Y takes values in {2, 3, …, 8}. Let g(Y) denote the pdf of Y.

3

g(2) = P(Y=2) = P(X1=1 and X2=1) = (1/4)(1/4)=1/16. g(3) = P(Y=3) = P(X1=1 and X2=2) + P(X1=2 and X2=1) = 2/16. similarly: g(4) = 3/16. g(5) = 4/16. g(6) = 3/16. g(7) = 2/16. g(8) = 1/16. Hence: E[Y] = 2(1/16)+3(2/16)+4(3/16)+5(4/16)+6(3/16)+7(2/16)+8(1/16)=5. One may then calculate Var(Y) = E([Y-5]2) = 5/2. 

Note that in many cases sampling distributions can be simulated more quickly and easily than they can be computed.

4. In many empirical applications the sampling distributions of the statistics of interest turn out to be chi-square, t or F. We next discuss each of these distributions. -

The chi-square distribution. (i)

A rv X is said to have a chi-square distribution with r degrees of freedom,  2 ( r ), if its pdf is 1 f ( x)  x r / 21e  x / 2 , 0  x  . r/2 ( r / 2)2 E[X] = r, Var(X) = 2r.

(ii)

If the rv X is N (  , 2 ) then the rv V   X    /  2 is  2 (1). That is, the square of a standard normal rv follows a chi-square distribution with one degree of freedom. 2

(iii)

If X 1 is  2 ( r1 ) and X 2 is  2 ( r2 ) and if they are independent, then X 1  X 2 ~  2 ( r1  r2 ).

(iv)

Let Z1 ,..., Z k be independent standard normal rv’s. Then the sum of the square of these rv’s has a chi-square distribution with k degrees of freedom: W  ( Z12  ...  Z k2 ) ~  2 ( k ).

(v)

If X 1 ,..., X n have mutually independent normal distributions n

N ( i ,  i2 ) then the distribution of W   i 1

4

( X i  i )2



2 i

is  2 (n).

Suppose X 1 ,..., X n are independent random draws from a N (  X , X2 ) distribution. Define 1 n X   X i (the sample mean) n i 1

(vi)

S2 

2 1 n X i  X  , (estimate of the variance)   n  1 i 1

then (a) X and S 2 are independent. n

(b)  n  1

S



2

2 X



 X i 1

i



X 2 X

2

is  2 ( n  1).

Use of the sample mean instead of the true mean reduces the degrees of freedom by one. This reduces the expected value of the statistic relative to the value under the true mean. The reason is that the squared deviation of the observations in the sample from the sample mean will never be greater than the difference from the mean of the population’s distribution. -

The t distribution (i)

If Z is a rv that is N(0,1), if U is a rv that is chi-square with r degrees of freedom, and if Z and U are independent, then Z T U /r has a t distribution with r degrees of freedom. 

This distribution was discovered by W. Gosset who published under the name of “Student.”

(ii)

The pdf of a t distribution is  ( r  1) / 2 g (t )  , - 4. q( r  2)2 ( r  4)

two independent normal distributions N (  X , X2 ) and N ( Y , Y2 ). Then (n  1) S X2 /  X2 is  2 (n  1) and (n  1) SY2 /  Y2 is  2 (m  1). Since the distributions, and hence S X2 and SY2 , are independent we have h



(m  1) SY2 /  Y2 (m  1) SY2 /  Y2  ~ F (m  1, n  1). (n  1) S X2 /  X2 (n  1) S X2 /  X2

This statistic is often useful when testing for the statistical significance of treatment effects.

(5) The central limit theorem Suppose X 1 ,..., X n is a random sample from a N (  X , X2 ) distribution. What is the sampling distribution of the mean X ? Note that:

6

E[ X ] 

1 n 1 E[ X i ]  n X   X .  n i 1 n

1 n 1  X2 2 X )  n    i n2 X n . n i 1 Moreover, since X is a scaled sum of independent normal rv’s it also follows a normal distribution. Thus, Var ( X )  Var (

X ~ N (X ,

 X2 n

) and W=

X  X ~ N (0,1). X / n

As n   the distribution of X   X degenerates to zero. However, the distribution of W does not degenerate. The scale factor  X / n spreads out the mass so that the variance of W is unity for all n. The statistic W is well defined even if the underlying sample is not normal. Suppose X 1 ,..., X n is a random sample from an arbitrary distribution. What is the sampling distribution of the W statistic in this case? It is easy to show (and you should) that E[W] = 0 and Var(W) = 1 in this case as well. However, since the distribution of the X is not normal the distribution of W will not generally be normal. Nevertheless, the central limit theorem (clt) assures us that a standard normal distribution closely approximates the true sampling distribution of W, as long as fairly weak regularity conditions hold and n is sufficiently large. Typically, n = 15 is large enough for practical purposes. Central Limit Theorem: If X is the mean of a random sample X 1 ,..., X n of size n from a distribution with a finite mean  X and a finite, positive variance  X2 then the distribution of X  X W X / n is N(0,1) in the limit as n  . 

It is often the case that assessing treatment contrasts involves comparing means from two treatment groups. One way to determine the significance of the contrasts is to use parametric tests that assume the means are normally distributed. The clt can be invoked to justify this normality assumption, particularly if the number of observations in each treatment is 15 or more.

7

C. Comments on Tests of Statistical Hypotheses 1. Definitions - A statistical hypothesis is an assertion about the distribution of one or more rv’s. -

A test of a statistical hypothesis is a procedure, based on the observed values of the rv’s, that leads to the acceptance or rejection of the hypothesis.

-

The critical region C is the set of points in the sample space that leads to the rejection of the null hypothesis.

-

A statistic used to define the critical region is referred to as a test statistic. The critical region can be defined as the set of values of the test statistic that leads to rejection of the null.

-

A type I error is made when the null is rejected even though it is true. A type II error is made when the null is accepted even though it is false.

-

The significance level of the test is the probability of a type I error.

-

The power function of a test of a null H 0 against an alternative H1 is the function that gives the probability of rejecting H 0 for each parameter point in

H 0 and H1 . The value of the power function at a particular point is called the “power” of the test at that point. -

The p-value is the probability, under H 0 , of all values of the test statistic that are as extreme (in the direction of rejection of H 0 ) as the observed value of the test statistic.

2. Example Suppose we have a random sample of 25 observations X i from an unknown distribution with a known variance of 100. We believe the mean of this distribution is 60, but we want to entertain the possibility that it exceeds 60. Hence, we will test the null hypothesis H 0 :   60 against the alternative hypothesis H1 :   60. Note that each of these are statistical hypotheses, since the make assertions about the distribution of X.

8

We decide to reject the null in favor of the alternative if the mean of the observations is at least 62, or X  62. This defines the critical region for the test: 1   C  ( X i )i 1,25 :  X i  62  . If the realized sample does not fall in the critical 25   region will accept the null. Since the critical region is known we may now determine the significance level (or size) and power of this test. The power function is  (  )  P( X  62;  ). Since by the clt the distribution of X should be well approximated by a N(  ,100/25=4) distribution, we have that X   62    (  )  P(  ; ) 2 2 62    1  ( ), 60   . 2 where  () denotes the standard normal cdf. Some values of the power function follow.

 60 61 62 63 64 65 66

 ( ) 0.16 0.31 0.50 0.69 0.84 0.93 0.98

Hence, the significance level of this test is 0.16: there is a 16% chance of rejecting the null when it is true. The power of the test when the true mean is 65 is 93%. The p-value associated with a realized mean of 63 is X  60 63  60 p-value = P( X  63 ;   60 )=P( )  2 2 63  60 ) = 0.067 = 1  ( 2 so we would not reject at a 5% significance level, but would reject at a 10% level. 

Generally, we would like a power of 100% and a size of 0%. This is not usually attainable. In fact, for a fixed sample size, more powerful tests usually have greater chances of generating a type I error.

9

3. The power function and sample size The power function can be used to help determine an appropriate sample size. Ex: Let X have a N(  X ,36) distribution. We want to use a sample of size n from this distribution to test the hypothesis that  X  50 against the alternative that  X  50. We want the size of this test (the probability of a type I error) to be 0.05 and the power of the test at  X  55 to be 0.90. This requires us to solve simultaneously the following two equations.

X  50 c  50  ), 6/ n 6/ n X  55 c  55 0.10  P( X  c;  X  55)  P(  ). 6/ n 6/ n 0.05  P( X  c; H 0 )  P(

Thus, we must solve c  50  1.645 6/ n c  55  1.282 6/ n

which has solution {c, n}  {52.8,12}.

10