Introduction to research methods in psychology/IRMP102/Common Threats to Internal Validity
A study's internal validity has to do with the ability of its design to support a causal conclusion. Unsurprisingly, experimental research tends to have the highest internal validity, followed by quasi-experimental research, and then correlational research, with case studies at the bottom of the list. Nonetheless, there are several potential threats to internal validity that are especially relevant to nonexperimental designs. These include the following eight threats, which may be remembered by using the acronym "MRS. SMITH":
Contents
Maturation
Physiological processes occurring within the participants that could account for any changes in their behaviour. Subjects may change between test sessions of the experiment such that any changes in scores between testing sessions may simply be due to the passage of time rather than any treatment effects.
Examples:
- Aging Processes: simply growing older; changes in motor coordination; cognitive development.
- Physiological States: hunger, fatigue; attention span; motivation
Regression to the Mean
The tendency that participants who receive extreme scores when tested, tend to have less extreme scores on subsequent retesting even in the absence of any treatment effects.
- This phenomenon is the result of the fact that all measurement instruments are not perfectly reliable (i.e., there is measurement error present). It is this error that most likely accounts for the extreme score, not some inherent characteristic within the individual. As a result, a person’s score tends to fluctuate on repeated testing. Extreme scores typically become less extreme. The implication is that the difference between groups formed based on extreme scores tend to become smaller even in the absence of any treatment effects.
Selection of Subjects
Any bias in selecting and assigning participants to groups that results in systematic differences between the participants in each group.
- The differences exist before one group is exposed to the experimental treatment.
- This threat to validity is great in quasi-experiments where the random assignment to treatment conditions is not possible.
Selection by Maturation Interaction
The treatment and no-treatment groups, although similar at one point, would have grown apart (developed differently) even if no treatment had been administered.
- Even though pretest scores may have been the same, groups that are not matched as well on other relevant variables that may cause the groups to naturally become different after a period of time
- Example: Long-term Head Start research comparing middle-class and disadvantaged children.
Mortality
Differential dropping out of some subjects from the comparison groups before the experiment is finished, resulting in differences between the groups that may be unrelated to the treatment effects.
- The problem is that the subjects who drop out of the study for whatever reasons may be different than those who complete it. This may inflate, obscure, or confuse the treatment effects of interest.
- The researcher excluding the data of particular subjects based on some criterion can also cause this bias.
Instrumentation
Changes in the measurement procedures may result in differences between the comparison groups that are confused with the treatment effects. For example:
- Observers may become more experienced or careless over time which results in differences between the pretest and posttest measurements that are unrelated to the treatment effects.
- Calibration of testing apparatus may change from one test to another.
- A change to a “better way of collecting the data” between the pretest and the posttest such as finding better ways to ask the same question of the participants.
Testing
When participants are repeatedly tested, changes in test scores may be more due to practice or knowledge about the test procedure gained from earlier experiences rather than any treatment effects
- Similar to maturation except that the change is caused by the testing procedure itself.
History
Extraneous events occurring during the course of the experiment that may affect the participants’ responses on the dependent measure.
- Could be major events occurring in society (e.g., social upheaval) or minor events occurring within the experimental situation (e.g., equipment malfunction)
- These events may account for the participants’ responses in the experiment more so than the treatment of interest.
