Hypothesis testing - exploring research studies

Studying research reports to extract the basic information about the study design, analysis and conclusions can help broaden one's perspective on how hypothesis testing is used to do research in the real world. The goal of this activity is to carve out an understanding of the 4 steps involved in hypothesis testing as implemented in the study:

1. Specify null and alternative hypotheses
2. Collect and summarize the data (calculate the test statistic)
3. Assess the evidence provided by the data (determine the p-value)
4. Make conclusions in the context of the original hypotheses

Instructions

Choose a journal article which reports the results of a research study, preferably in your field of study. Try to choose an article which uses one of the statistical methods that we will be studying: t-test, chi-square, regression, ANOVA. The article may be something which you have already read. If possible, collaborate with one or more students in the class to choose the same article. (However, once chosen, please do the rest of this exercise independently; you will get a lot more out of it.)

Read and study your chosen article. Summarize your findings for each of the four steps. Copy and paste the following headers and guiding statements into a word processor. For each bullet, enter the information about your article.

Summarize the four steps in hypothesis testing

Background Information:

• Article citation: Using your preferred standard (e.g., APA style), enter a complete citation for this article.
• Research question: State what question this research is designed to answer.
• Explanatory variable:
• Response variable:
• Motivation for study: Briefly describe why this research is of interest.
• Study design: Describe the overall study design, indicating whether it is experimental (include description of treatments) or observational.

Step 1: Specify the null and alternative hypotheses.

• H₀:
• H_a:

Step 2: Collect and summarize the data

• Sample: Describe the sample used in the study.
• Exploratory analyses: Describe what exploratory analysis results were included (summary stats, graphs).
• Alpha level: State the alpha level.
• Statistical test: Specify the test that was used.
• Conditions met: If provided in the article, indicate in what ways the sample met the conditions for using this test.
• Test value: State the value(s) of the test statistic.

Step 3: Assess the evidence provided by the data

• P-value: State the p-value of the test statistic.
• Confidence interval: If provided in the article, give the estimate and the confidence interval.

Step 4: Make conclusions in the context of the original hypotheses

• Statistical significance: State whether or not H₀ is rejected in favor of H_a.
• Conclusions: Briefly describe what conclusions were made based on the data results.
• Limitations: List ways in which this research is limited in what it can conclude.

Follow-up in-class discussion

Organize students into groups of 2-4 students according to research interest, if possible. Post the following to guide group discussion:

1. Begin the discussion with each person presenting their analysis of chosen research article. If more than one person analyzed the same article, each successive student presents their results in comparison (similar/different?).
2. Following each student presentation of article analysis,
   • Group members may want to comment on analysis (ask for clarification; note something similar to their analysis)
   • The student will describe any issues with how the author presented the statistical analysis. (e.g., anything important not included?)
   • Students should discuss ideas for how analysis results could be better communicated.
3. When all studies have been discussed organize a full group discussion of the following questions:
   • What aspects of the 4-step hypothesis testing process are almost always included in report?
   • What aspects are often not included?
   • Generally speaking, do the presented results provide enough detail to evaluate the "validity" of the research?
   • Any other comments from group discussions?