1. National Program Education

As a consequence of the COVID-19 pandemic, many students, from pre-primary to university education institutions, have developed substantial educational delays, both cognitively and social-emotionally, and especially students from low socioeconomic and ethnic/immigrant backgrounds.^[1][2][3][4] To counter the negative effects of the school closures, several policies and support strategies on attainment and social-emotional well-being have been proposed and implemented.^[5][6]

In The Netherlands, the Ministry of Education has introduced an unprecedented National Program Education (NPE) to combat Covic-19 related educational delays.^[7] The core of this program consists of providing schools with additional budgets which they can spend on implementing evidence-based interventions. The total budget amounts to €8.5 billion for a 2.5 year period. Primary and secondary schools receive at least €700 per student per year; depending on the social-ethnic backgrounds of their students, schools may receive even more. Schools are required to spend the money on evidence-based interventions, denoting approaches that emphasize the practical application of the findings of the best available current scientific research, preferably from experiments using randomized controlled designs. The Ministry has developed a so-called menu card with (for now) 22 interventions schools can choose from.^[8] Most of them (19) come from the Teaching and Learning Toolkit developed by the English Education Endowment Foundation (EEF);^[9] the rest (3) is added by the Dutch Ministry.

2. Timeline

For 2021, the Ministry has published a timeline for the schools, depicted in Table 1.^[10] This is a very tight one, as the schools must do a lot of work in a very short period of time (and while at the same time reopening the schools after a lockdown) and - inevitable - partly during the summer holidays.

Table 1. Timeline National Program Education (February 2021 – December 2021)

3. Effect sizes

Most of the evidence-based interventions on the NPE menu card are translations of the EEF’s Teaching and Learning Toolkit.^[9] The Toolkit’s interventions specifically aim at boosting the attainment of disadvantaged children. In the Toolkit’s Technical Appendices the effectiveness of each of the interventions is accounted for. EEF uses three criteria to rate the interventions: (1) the cost; (2) the strength of the evidence; (3) the impact in terms of months additional progress, where it is assumed that 1 SD is equivalent to 1 year of progress, and thus 0.09 SD is about 1 month of progress^[11] (a procedure which is severely crititized^{[12][13][14][15]}).

The interventions’ effects are calculatexpressed by EEF in terms of so-called effect sizes. There are many types of effect sizes, but in its simplest form an effect size represents the standardized difference between two groups, ideally the experimental group and the control group (or the group with the intervention and the group without the intervention, respectively). Because they are standardized, they can be compared across studies. A common effect size is Cohen’s d. Cohen (1988)^[16] himself provided the following rule of thumb for interpreting d: <0.20, negligible effect; 0.20-0.50, small effect; 0.50-0.80, medium effect; >0.80, large effect. However, he cautions that these values are necessarily somewhat arbitrary and acknowledges that the qualifications should be used in the proper context. According to Hattie (2009)^[17], who performed many meta-meta analyses using d, most interventions in education 'work' with an average effect size of about 0.40, and that this should therefore be a benchmark. EEF (2018)^[11] uses the following classification: <0.02, negative, no or very low effect; 0.02-0.18, low effect; 0.19-0.44, moderate effect; 0.45-0.69, high effect; 0.70-1.00, very high effect. On the one hand, EEF argues that a ‘small’ effect may nevertheless be educationally important if it is easy or cheap to attain or achievable with groups who are otherwise hard to influence. On the other hand, a ‘large’ effect size may not be important if it is unrealistic to bring about in normal circumstances. One of the contributors to the NPE menu card is the Netherlands Youth Institute (NJI, 2021),^[18] which employs Cohen’s rule of thumb interpretation: <0.20, no or a negligible effect; 0.20- 0.50, a small effect; 0.50-0.80, a medium effect; >0.80, a large or very large effect. But NJI, too, points to the fact that one should also consider the context when interpreting effect sizes.

4. Evidence-based interventions?

The main question here is how much evidence there really is for the evidence-based interventions included in the menu card. To get an impression of the effectiveness, the evidence supplied by EEF in the Toolkit’s Technical Appendices^[9] has been critically reviewed and summarized. EEF discerns two types of studies, single studies and meta-analyses (which comprise a quantitative summary of a series single studies). Table 2 provides per intervention the number of underlying meta-analyses and single studies, the ‘range’ (minimum – maximum) of the effect sizes (d), the average effect, and prominent features as regards content and methodology.

Table 2. Overview effects evidence based interventions (effect sizes d; source: EEF Toolkit)

# Not part of the EEF Toolkit; * Weighted mean; ** Median; *** Indicative.

When looking at the average effect sizes in Table 2 and focusing on the 19 EEF interventions (the other 3 are added by the Dutch Ministry of Education), it can be concluded that according to Cohen’s (1988)^[16] rule of thumb, 37 percent of the interventions have a negligible effect, 53 percent a small effect, and only 2 a medium effect. According to Hattie (2009)^[17], only 4 out of 19 interventions, or 21 percent, achieve at or above his benchmark level of 0.40. It is obvious that EEF’s^[11] rule of the thumb is the most lenient one:. According to EEF, 26 percent of the interventions have a low effect, 58 percent a moderate effect, and 16 percent a high effect. To summarize these findings: in so far interventions are effective, only very few of them really show a robust effect. In addition,But from a further look at the range of the effect sizes, it becomes clear that this conclusion is strongly embellished. There exists an enormous variation in the magnitude of the effect sizes, they vary from negative to strongly positive. (In fact, there is so much variation that computing averages is not a statistical sound procedure.) What is lacking in most of the cases is consistency. This variability probably is caused by the fact that under one intervention (e.g., parental engagement), there are many practical interpretations (e.g., helping with homework, being a member of the parent council, volunteering in the classroom, or visiting libraries), with each having its specific effects (e.g., negative or positive) on different output domains (e.g., language or motivation) for different groups of students (e.g., immigrant or high social class). Therefore, when head teachers choose an intervention to solve the specific problems and needs of their schools, they have no guarantee whatsoever that their local interpretations will lead to the hoped for results. The situation clearly is considerably more complex than assumed at first sight. In addition, there are many more problematic considerations with regard to the evidence base of the interventions. Table 3 provides an overview.

Table 3. – Overview problematic aspects studies into evidence based interventions (trends)

5. Conclusions

On the basis of these findings, the obligation for schools to choose so-called evidence-based interventions from the NPE’s menu card is a highly dubious one. First of all, it appears that many interventions are really not that evidence-based as one would expect. In addition, it cannot be expected from the schools’ head teachers to make a valid and reliable choice as they simply do not have the statistical and methodological knowledge to interpret the EEF’s scientific foundation of the interventions correctly. Moreover, the evidence presented in the EEF’s Toolkit only holds for a specific interpretation of an intervention, for specific groups (e.g., age, socioeconomic background, ethnicity) and domain (e.g., language, motivation). Probably it is a better idea to now start well-designed studies, preferably randomized controlled experiments, to have really proven interventions available in the future. The present situation, with all that money and data available under the Dutch National Program Education, offers an ideal starting point.