What does the pretest posttest design measure?

We consider multiple designs for the simple case of a single treatment group and a single control group, as shown in Figure 1.

Figure 1 – Pretest-Posttest Data

T-test of the Gain

One approach is to simply compare the gain after treatment for the two groups. Here the gain is the difference between the posttest and pretest scores. This can be done by using a two-sample t-test based on the data in range E4:E8 vs. E9:E13, as shown in Figure 2.

Figure 2 – t-test on the gain

With p-value = .003, we see there is a significant difference between the Treatment and Control groups.

Note that the study as a whole depends on the fact that there isn’t a significant difference between the Treatment and Control groups in the pretest phase. This can be checked by running a two-sample t-test based on the data in range C4:C8 vs. C9:C13, as shown in Figure 3.

Figure 3 – t-test on pretest data

The p-value = .58 confirms that there is no significant difference between the Treatment and Control groups, although there is a larger medium-sized effect (d = .36) with a non-negligible difference between the group means (19 vs. 35).

Repeated Measures ANOVA

Another approach is to perform a repeated-measures ANOVA with one between-subjects factor (treatment and control) and one within-subjects factor (pre- and post), as shown in Figure 4.

Figure 4 – Repeated measures analysis

Since there are only two between-subjects levels, we don’t need to worry about sphericity. For this analysis, p-value = .002875, which again shows that there is a significant difference between the Treatment and Control groups. Note that it is the p-value of the Interaction effect that is most relevant. Also, note that this value is identical to the p-value shown in cell K13 of Figure 2. This will be so for any similar analysis.

The pretest-posttest control group design, also called the pretest-posttest randomized experimental design, is a type of experiment where participants get randomly assigned to either receive an intervention (the treatment group) or not (the control group). The outcome of interest is measured 2 times, once before the treatment group gets the intervention — the pretest — and once after it — the posttest.

The objective is to measure the effect of the intervention which can be:

• A medical treatment
• An education program
• A policy change, etc.

The pretest-posttest control group design has 3 major characteristics:

1. The study participants are randomly assigned to either the treatment or the control group (this random assignment can occur either before of after the pretest).
2. Both groups are exposed to the same conditions except for the intervention: the treatment group receives the intervention, whereas the control group does not.
3. The outcome is measured simultaneously for both groups at 2 points in time — the pretest and the posttest.

The pretest-posttest control group is the most commonly used design in randomized controlled trials.

Advantages of the pretest-posttest control group design

By using a pretest, a control group, and random assignment, this design controls all internal threats to validity.

Advantage of having a pretest measurement

This design is better than the posttest-only control group design because it adds a pretest.

Adding a pretest:

1. Increases the power of the design to detect an effect.
2. Allows studying the effect of the intervention at different sublevels of the pretest.
3. Helps analyzing initial differences between groups (and therefore quantifying their effect on the study outcome).
4. Helps controlling attrition bias i.e. the unequal loss to follow-up of participants between the treatment and the control group which can affect the outcome measured at the posttest.

Advantage of using random assignment and having a control group

Random assignment and the control group will both limit the effects of:

1. Selection bias: Which happens when participants themselves get to choose if they receive the intervention or not. This may create unequal and incomparable study groups. Randomization allows unbiased assignment of participants to treatment options, and therefore makes the study groups comparable.
2. Maturation: Which is the effect of time (between the pretest and the posttest) on study participants (e.g. participants growing older, or getting tired over time) which might influence the outcome, thus becoming a rival explanation for the intervention regarding the study outcome. Participants are subject to maturation both in the treatment and the control group, therefore, any difference between the outcome of these groups will be due to the effect of the treatment alone and will not be affected by maturation.
3. History: Which is any event that might co-occur with the intervention and has the potential to influence the outcome. Co-occurring events affect both the treatment and the control group, and therefore any difference between the outcome of these groups will be due to the effect of the treatment alone and will not be affected by history.
4. Testing: Which is the effect of taking a pretest on the result of a posttest. For instance, if the pretest sensitizes participants and compels them to behave in a certain way that affects the outcome of the posttest. The presence of a control group protects against testing effects, as these will affect both groups and therefore any difference between the outcome of these groups will be due to the effect of the treatment alone and will not be affected by testing.
5. Regression to the mean: When pretest scores are exceptionally good by chance, the posttest scores will naturally regress toward the mean. This happens because an exceptionally good performance is hard to maintain. Regression toward the mean can be mistaken for the effect of the treatment, and therefore is a source of bias. Since participants from both groups are subject to regression, therefore, comparing the outcome of the treatment group with that of the control group will take care of this regression effect.

Limitations of the pretest-posttest control group design

Participants included in any randomized study might not be typical people in the population i.e. they may not represent well the population of interest, this is because:

1. Not everyone in the population of interest is eligible for the experiment,
2. and not everyone who is eligible can be recruited,
3. and not everyone who is recruited will give us their consent to be included in the study,
4. and not everyone who consented will be randomized.

So the outcome of a randomized study may not generalize well to the population.

More specifically, this design:

• Does not allow us to study how the effect of the treatment changes over time: To do so, we need to add more posttest measures.
• Is susceptible for interactions between the intervention and other factors (such as the pretest, history, instrumentation, etc.): One solution for this problem is to use the Solomon four-group design.

Example of a study that used the pretest-posttest control group design

Koenig et al. used a pretest-posttest control group design to study the effect of a yoga program on the classroom behavior of autistic children. These children were randomly assigned to either receive the yoga program or their standard morning routine.

The study concluded that yoga can significantly improve the classroom behavior of autistic children.

But because the researchers used a convenience sample from a particular school and the classrooms that were allowed to participate were hand-picked by administrators, the study outcome may not generalize well to all children with autism.

What is the pretest and posttest design in research?

What does a post test measure?

What does a pre/post experimental design requires the researcher to measure?

What is the purpose of the pretest in an experiment?