Common Threats to Internal Validity

|

Common Threats to Internal Validity

START: Athlone Flatline Half Marathon 2014

A study's internal validity has to do with the ability of its design to support a causal conclusion. Unsurprisingly, experimental research tends to have the highest internal validity, followed by quasi-experimental research, and then correlational research, with case studies at the bottom of the list. Nonetheless, there are several potential threats to internal validity that are especially relevant to nonexperimental designs. These include the following eight threats, which may be remembered by using the acronym "MRS. SMITH":

Maturation

Physiological processes occurring within the participants that could account for any changes in their behaviour. Subjects may change between test sessions of the experiment such that any changes in scores between testing sessions may simply be due to the passage of time rather than any treatment effects.

Examples:

• Aging Processes: simply growing older; changes in motor coordination; cognitive development.
• Physiological States: hunger, fatigue; attention span; motivation

Regression to the Mean

The tendency that participants who receive extreme scores when tested, tend to have less extreme scores on subsequent retesting even in the absence of any treatment effects.

• This phenomenon is the result of the fact that all measurement instruments are not perfectly reliable (i.e., there is measurement error present). It is this error that most likely accounts for the extreme score, not some inherent characteristic within the individual. As a result, a person’s score tends to fluctuate on repeated testing. Extreme scores typically become less extreme. The implication is that the difference between groups formed based on extreme scores tend to become smaller even in the absence of any treatment effects.

Selection of Subjects

Any bias in selecting and assigning participants to groups that results in systematic differences between the participants in each group.

• The differences exist before one group is exposed to the experimental treatment.
• This threat to validity is great in quasi-experiments where the random assignment to treatment conditions is not possible.

Selection by Maturation Interaction

The treatment and no-treatment groups, although similar at one point, would have grown apart (developed differently) even if no treatment had been administered.

• Even though pretest scores may have been the same, groups that are not matched as well on other relevant variables that may cause the groups to naturally become different after a period of time

• Example: Long-term Head Start research comparing middle-class and disadvantaged children.

Mortality

Differential dropping out of some subjects from the comparison groups before the experiment is finished, resulting in differences between the groups that may be unrelated to the treatment effects.

• The problem is that the subjects who drop out of the study for whatever reasons may be different than those who complete it. This may inflate, obscure, or confuse the treatment effects of interest.
• The researcher excluding the data of particular subjects based on some criterion can also cause this bias.

Instrumentation

Changes in the measurement procedures may result in differences between the comparison groups that are confused with the treatment effects. For example:

• Observers may become more experienced or careless over time which results in differences between the pretest and posttest measurements that are unrelated to the treatment effects.
• Calibration of testing apparatus may change from one test to another.
• A change to a “better way of collecting the data” between the pretest and the posttest such as finding better ways to ask the same question of the participants.

Testing

When participants are repeatedly tested, changes in test scores may be more due to practice or knowledge about the test procedure gained from earlier experiences rather than any treatment effects

• Similar to maturation except that the change is caused by the testing procedure itself.

History

Extraneous events occurring during the course of the experiment that may affect the participants’ responses on the dependent measure.

• Could be major events occurring in society (e.g., social upheaval) or minor events occurring within the experimental situation (e.g., equipment malfunction)
• These events may account for the participants’ responses in the experiment more so than the treatment of interest.

Activity

Read through this "We're only human" news story and try to answer the following questions:

1. Clearly describe one correlational result reported in the news study. Your description should include the variables involved as well as the direction of the correlation. (6 marks)

2. Draw and clearly label a scatterplot that illustrates this correlation using 10 data points. You may wish to use the Interpreting Correlations interactive visualization to guide your drawing. (8 marks)

3. Did the researcher(s) consider whether a third variable might have influenced this correlation? If so, which variable did they measure? (2 marks)

4. How do the researchers interpret this correlation? Do they explain the correlation in a particular causal direction? Can you suggest an alternative interpretation of this correlation? (6 marks)

5. Consider how you might be able to address the same question using a different non-experimental research design. Your proposed study should use the same variables but should operationalize them differently. Briefly outline your proposed study. (8 marks)

Answers to the questions are available here.

For additional practice you can browse through the studies analyzed by Beth Morling on her website: Everyday Research Methods.

Share your favourite example on the micro-blog discussion forum (found at the bottom of the Home Page) or on Twitter (using the hashtag #OERuPSYC2111).

|