class: center, middle, inverse, title-slide # NHST II --- ## Review of key Points - The null hypothesis ( `\(H_0\)` ) is a claim about the particular value that a population parameter takes. - The alternative hypothesis ( `\(H_1\)` ) states an alternative range of values for the population parameter. - We test the null hypothesis by determining if we have sufficient evidence that contradicts or nullifies it. - We reject the null hypothesis if the data in hand are rare, unusual, or atypical if the null were true. The alternative hypothesis gains support when the null is rejected, but `\(H_1\)` is not proven. --- ### Review example Healthy adults send an average of 32 text messages a day `\((\sigma = 7.3)\)`. I recruit a sample of 16 adults with diagnoses of Generalized Anxiety Disorder. In my sample, the average number of texts sent per day is 33.2. _Does my sample of adults with GAD come from the same population as healthy adults?_ -- **Define hypotheses** - `\(H_0: \mu_{GAD} = 32\)` - `\(H_0: \mu_{GAD} \neq 32\)` -- **Choose alpha** - `\(\alpha = .05\)` --- ### Review example **Collect data.** **Define your sampling distribution using your null hypothesis** - `\(\mu_0 = 32\)` - `\(\sigma_M = \frac{\sigma}{\sqrt{N}} = \frac{7.3}{\sqrt{16}} = 1.825\)` **Calculate the probability of your data or more extreme under the null.** - `\(Z_{statistic} = \frac{\bar{X} - \mu_0}{\sigma_M} = \frac{33.2 - 32}{1.825} = 0.658\)` - `\(p(Z >= 0.658) = 0.255\)` - `\(p = 0.511\)` --- Compare your probability ( `\(p\)`-value) to your `\(\alpha\)` level and decide whether your data are "statistically significant" (reject the null) or not (retain the null). -- Our `\(p\)`-value is larger than our `\(\alpha\)` level, so we retain the null. --- ### Review - If we do not reject `\(H_0\)`, that does not mean that we accept it. We have simply failed to reject it. It lives to fight another day. - A _Z_-statistic summarizes how unusual the sample estimate of a mean compared to the value of the mean as specified by the null hypothesis. - More broadly, we refer to the **test statistic** as the statistic that summarizes how unusual the sample estimate of a parameter is from the point value specified by the null hypothesis. --- ### Review - The test statistic is derived from a probability distribution that represents the likelihood of any sample estimated given that the null hypothesis is true. - We choose the probability assumption based on assumptions we are willing to make about the data. The **sampling distribution** of the test statistic tells us what counts as rare or unusual. --- ### Review - We summarize the probability of the sample data (or even more extreme data), assuming the null is true, by the ** `\(p\)`-value** (signified by `\(p\)`). This value is the proportion of the sampling distribution of the test statistic that is as extreme, or more extreme, as the test statistic value. If `\(p\)` is small, the observed data are unusual *if the null hypothesis is true*. If `\(p\)` is sufficiently small ( `\(p < .05\)` ), we may choose to reject the null hypothesis as a description of the data. --- ## `\(p\)`-values - Fisher established (rather arbitrarily) the sanctity of the .05 and .01 significance levels during his work in agriculture, including work on the effectiveness of fertilizer. A common source of fertilizer is cow manure. Male cattle are called bulls. A common misinterpretation of the `\(p\)`-value ( `\(\alpha\)` ) is that it is the probability of the null hypothesis being wrong. Another common misunderstanding is that `\(1-\alpha\)` is the probability that results will replicate. --- - In most research, the probability that the null hypothesis is true is very small. - If the null hypothesis is false, then the only mistake to be made is a failure to detect a real effect. We collect a sample and compare this to a null hypothesis: `\(H_0: \mu_0 = 100\)`. Holding our sample mean `\((\bar{X} = 105)\)` constant, how does our `\(p\)`-value change as our sample size gets larger? --- We collect a sample and compare this to a null hypothesis: `\(H_0: \mu_0 = 100\)`. Holding our sample mean `\((\bar{X} = 105)\)` constant, how does our `\(p\)`-value change as our sample size gets larger? <!-- --> --- - If the null hypothesis is false, then the significance test is akin to a test of whether the sample size was large enough. - Because Null Hypothesis Significance Testing (NHST) is beginning to seem like a bit of a sham, some have suggested we start calling it Statistical Hypothesis Inference Testing. --- ## Errors In hypothesis testing, we can make two kinds of errors. | | Reject `\(H_0\)` | Do not reject | |------------:|:----------------:|:----------------:| | `\(H_0\)` True | Type I Error | Correct decision | | `\(H_0\)` False | Correct decision | Type II Error | Falsely rejecting the null hypothesis is a **Type I error**. Traditionally, this has been viewed as particularly important to control at a low level (akin to avoiding false conviction of an innocent defendant). --- ## Errors In hypothesis testing, we can make two kinds of errors. | | Reject `\(H_0\)` | Do not reject | |------------:|:----------------:|:----------------:| | `\(H_0\)` True | Type I Error | Correct decision | | `\(H_0\)` False | Correct decision | Type II Error | Failing to reject the null hypothesis when it is false is a **Type II error**. This is sometimes viewed as a failure in signal detection. --- ## Errors In hypothesis testing, we can make two kinds of errors. | | Reject `\(H_0\)` | Do not reject | |------------:|:----------------:|:----------------:| | `\(H_0\)` True | Type I Error | Correct decision | | `\(H_0\)` False | Correct decision | Type II Error | Null hypothesis testing is designed to make it easy to control Type I errors. We set a minimum proportion of such errors that we would be willing to tolerate in the long run. This is the significance level ( `\(\alpha\)` ). By tradition this is no greater than .05. --- ## Errors In hypothesis testing, we can make two kinds of errors. | | Reject `\(H_0\)` | Do not reject | |------------:|:----------------:|:----------------:| | `\(H_0\)` True | Type I Error | Correct decision | | `\(H_0\)` False | Correct decision | Type II Error | Controlling Type II errors is more challenging because it depends on several factors. But, we usually DO want to control these errors. Some argue that the null hypothesis is usually false, so the only error we can make is a Type II error -- a failure to detect a signal that is present. **Power** is the probability of correctly rejecting a false null hypothesis. --- ### Some Greek letters `\(\alpha\)` : The rate at which we make Type I errors, which is the same `\(\alpha\)` as the cut-off for `\(p\)` -values. `\(\beta\)` : The rate at which we make Type II errors. `\(1-\beta\)` : statistical power. Note that all these probability statements are being made in the frequentist sense -- in the long run, we expect to make Type I errors `\(\alpha\)` proportion of the time and Type II errors `\(\beta\)` proportion of the time. --- Controlling Type II errors is the goal of power analysis and must contend with four quantities that are interrelated: - Sample size - Effect size - Significance level ( `\(\alpha\)` ) - Power When any three are known, the remaining one can be determined. Usually this translates into determining the power present in a research design (**post-hoc power analysis** ), or, determining the sample size necessary to achieve a desired level of power. We must specify a specific value for the alternative hypothesis to estimate and control Type II errors. --- Suppose we have a measure of social sensitivity that we have administered to a random sample of 25 psychology students. This measure has a population mean ( `\(\mu\)` ) of 100 and a standard deviation ( `\(\sigma\)` ) of 20. We suspect that psychology students are more sensitive to others than is typical and want to know if their mean, which is 110, is sufficient evidence to reject the null hypothesis that they are no more sensitive than the rest of the population. We would also like to know how likely it is that we could make a mistake by concluding that psychology students are not different when they really are: A Type II error. --- .left-column[ .small[ We begin by defining the location in the null hypothesis distribution beyond which empirical results would be considered sufficiently unusual to lead us to reject the null hypothesis. We control these mistakes (Type I errors) at the chosen level of significance ( `\(\alpha = .05\)` ). ] ] <!-- --> ??? ### How do we figure out where this region is? Start with our `\(\alpha\)` level (.05) Given our distribution parameters, what Z-value corresponds to .05 above that line mu = 100 x = 110 s = 20 n = 25 sem = 4 z = 1.96 --- .pull-left[ `$$\text{Critical Value} = \mu_0 + Z_{.975}\frac{\sigma}{\sqrt{N}}$$` ```r qnorm(.975) ``` ``` ## [1] 1.959964 ``` `$$\text{Critical Value} = 100 + 1.96\frac{20}{\sqrt{25}} = 107.84$$` What if the null hypothesis is false? How likely are we to correctly reject the null hypothesis in the long run? ] .pull-right[ <p>&nbsp;</p> <p>&nbsp;</p> <p>&nbsp;</p> <p>&nbsp;</p> <p>&nbsp;</p> <!-- --> ] --- <!-- --> --- <!-- --> --- <!-- --> --- <!-- --> --- .left-column[ .small[ To determine the probability of a Type II error, we must specify a value for the alternative hypothesis. We will use the sample mean of 110. In the long run, if psychology samples have a mean of 110 ( `\(\sigma = 20\)`, `\(N = 25\)` ), we will correctly reject the null with probability of 0.71 (power). We will incorrectly fail to reject the null with probability of 0.29 ( `\(\beta\)` ). ] ] <!-- --> --- .pull-left[ <!-- --> Once the critical value and alternative value is established, we can determine the location of the critical value in the alternative distribution. ] -- .pull-right[ `$$Z_1 = \frac{CV_0 - \mu_1}{\frac{\sigma}{\sqrt{N}}}$$` `$$Z_1 = \frac{107.84 - 110}{\frac{20}{\sqrt{25}}} = -0.54$$` The proportion of the alternative distribution that falls below that point is the probability of a Type II error (.29); power is then .71. ```r pnorm(-.54) ``` ``` ## [1] 0.2945985 ``` ] --- The choice of 110 as the mean of `\(H_1\)` is completely arbitrary. What if we believe that the alternative mean is 115? This larger signal should be easier to detect. .pull-left[ <!-- --> ] .pull-right[ `$$Z_1 = \frac{107.84 - 115}{\frac{20}{\sqrt{25}}} = -1.79$$` ```r 1-pnorm(-1.79) ``` ``` ## [1] 0.963273 ``` ] --- What if instead we increase the sample size? This will reduce variability in the sampling distribution, making the difference between the null and alternative distributions easier to see. (New `\(N\)` is 50.) .pull-left[ <!-- --> ] .pull-right[ `\(\text{CV} = 100 + 1.96\frac{20}{\sqrt{50}} = 105.54\)` `$$Z_1 = \frac{105.54 - 110}{\frac{20}{\sqrt{50}}} = -1.58$$` ```r 1-pnorm(-1.57) ``` ``` ## [1] 0.9417924 ``` ] --- What if we decrease the significance level to .01? <p>&nbsp;</p> .pull-left[ <!-- --> ] .pull-right[ That will move the critical value: `\(\text{CV} = 100 + 2.58\frac{20}{\sqrt{25}} = 110.3\)` `$$Z_1 = \frac{110.3 - 110}{\frac{20}{\sqrt{25}}} = 0.08$$` ```r 1-pnorm(.08) ``` ``` ## [1] 0.4681186 ``` ] --- Oops, we've been ignoring the other tail. So far it hasn't mattered (the area has been so small) but it makes a difference when `\(H_0\)` and `\(H_1\)` overlap significantly. .pull-left[ <!-- --> ] .pull-right[ `$$Z_1 = \frac{107.84 - 102}{\frac{20}{\sqrt{25}}} = 1.46$$` `$$Z_2 = \frac{92.16 - 102}{\frac{20}{\sqrt{25}}} = -2.46$$` ```r 1-pnorm(1.46) ``` ``` ## [1] 0.07214504 ``` ```r pnorm(-2.46) ``` ``` ## [1] 0.006946851 ``` ] --- More generally we can determine the relationship between effect size and power for a constant `\(\alpha\)` (.05) and sample size ( `\(N = 20\)` ). <!-- --> --- Likewise, we can display the relationship between sample size and power for a constant `\(\alpha\)` and effect size. <!-- --> --- Power changes as a function of significance level for a constant effect size and sample size. <!-- --> --- For a one-tailed test, we put the entire rejection area into a single tail. If `\(\alpha = .05\)`, then one tail contains .05 and critical values will be either 1.64 or -1.64 standard errors away from the null mean. <!-- --> --- A two-tailed test is less powerful (more conservative) than a one-tailed test for the same sample size. <!-- --> --- <!-- --> --- ### How can power be increased? .pull-left[ <!-- --> ] .pull-right[ `$$Z_1 = \frac{CV_0 - \mu_1}{\frac{\sigma}{\sqrt{N}}}$$` - increase `\(\mu_1\)` - decrease `\(CV_0\)` - increase `\(N\)` - reduce `\(\sigma\)` ] I strongly recommend playing around with different configurations of `\(N\)`, `\(\alpha\)` and the difference in means ( `\(d\)` ) in this [online demo](https://rpsychologist.com/d3/NHST/). --- class: inverse ## Next time... critiques of NHST