School Effectiveness Research Paper

Assessment of school effects occurs in various types of applied contexts like the evaluation of school improvement programs or comparing schools for accountability purposes, by governments, municipalities, or individual schools.

School effectiveness research attempts to deal with the causal aspects inherent in the effectiveness concept by means of scientific methods. Not only are assessment of school effects considered, but particularly the attribution of differences in school effects to malleable conditions. Usually, school effects are assessed in a comparative way, e.g., by comparing average achievement scores between schools. In order to determine the ‘net’ effect of malleable conditions, like the use of different teaching methods or a particular form of school management, achievement measures have to be adjusted for intake differences between schools. For this purpose student background characteristics like socioeconomic status, general scholastic aptitudes, or initial achievement in a subject are used as control variables. This type of statistical adjustment in research studies has an applied parallel in the strive for ‘fair comparisons’ between schools, known under the label of ‘value-added.’

2. Strands Of Educational Effectiveness Research

School effectiveness research has developed as a gradual integration of several research traditions. The roots of current ‘state-of-the-art’ school effectiveness research are sketched by briefly referring to each of these research traditions.

The elementary design of school effectiveness research is the association of hypothetical effectiveness-enhancing conditions of schooling and output measures, mostly student achievement. A basic model from systems theory, where the school is seen as a black box, within which processes or ‘throughput’ take place to transform inputs into outputs. The inclusion of an environmental or context dimension completes this model (see Fig. 1). The major task of school effectiveness research is to reveal the impact of relevant input characteristics on output and to ‘break open’ the black box in order to show which process or throughput factors ‘work,’ next to the impact of contextual conditions. Within the school it is helpful to distinguish a school and a classroom level and, accordingly, school organizational and instructional processes.

Research tradition in educational effectiveness varies according to the emphasis that is put on the various antecedent conditions of educational outputs. These traditions also have a disciplinary basis. The common denominator of the five areas of effectiveness research that will be distinguished is that in each case the elementary design of associating outputs or outcomes of schooling with antecedent conditions (inputs, processes, or contextual) applies.

The following research areas or research traditions can be distinguished:

(a) Research on equality of opportunities in education and the significance of the school in this.

(b) Economic studies on education production functions.

(c) The evaluation of compensatory programs.

(d) Studies of unusually effective schools.

(e) Studies on the effectiveness of teachers, classes, and instructional procedures.

For a further discussion of each of these research traditions the reader is referred to Scheerens (1999). A schematic characterization of research orientation and disciplinary background is given in Table 1.

3. Integrated School Effectiveness Research

In recent school effectiveness studies these various approaches to educational effectiveness have become integrated. Integration was manifested in the conceptual modeling and the choice of variables. At the technical level multilevel analysis has contributed significantly to this development. In contributions to the conceptual modeling of school effectiveness, schools became depicted as a set of ‘nested layers’ (Purkey and Smith 1983), where the central assumption was that higher organizational levels facilitated effectiveness enhancing conditions at lower levels (Scheerens and Creemers 1989). In this way, a synthesis between production functions, instructional effectiveness, and school effectiveness became possible by including the key variables from each tradition, each at the appropriate ‘layer’ or level of school functioning (the school environment, the level of school organization and management, the classroom level, and the level of the individual student). Conceptual models that were developed according to this integrative perspective are those by Scheerens (1990), Creemers (1994), and Stringfield and Slavin (1992). The Scheerens model is shown in Fig. 2.

Exemplary cases of integrative, multilevel school effectiveness studies are those by Brandsma (1993), Sammons et al. (1995), and Grisay (1996).

In Table 2 (cited from Scheerens and Bosker 1997) the results of three meta-analyses and a re-analysis of an international data set have been summarized and compared to results of more ‘qualitative’ review of the research evidence. The qualitative review was based on studies by Purkey and Smith (1983), Levine and Lezotte (1990), Scheerens (1992), and Sammons et al. (1995). The results concerning resource input variables are based on the re-analysis of Hanushek’s (1979) summary of results of production function studies that was carried out by Hedges et al. (1994). As stated before this re-analysis was criticized, particularly the unexpectedly large effect of per pupil expenditure.

The results on ‘aspects of structured teaching’ are taken from meta-analyses conducted by Fraser et al. (1987). The international analysis was based on the IEA Reading Literacy Study and carried out by Bosker (Scheerens and Bosker 1997, Chap. 7). The meta-analysis on school organizational factors, as well as the instructional conditions ‘opportunity to learn,’ ‘time on task,’ ‘homework,’ and ‘monitoring at classroom level,’ were carried out by Witziers and Bosker and published in Scheerens and Bosker (1997, Chap. 6). The number of studies that were used for these meta-analyses varied per variable, ranging from 14 to 38 studies in primary and lower secondary schools.

The results in this summary of reviews and meta- analyses indicate that resource-input factors on average have a negligible effect, school factors have a small effect, while instructional variables have an average to large effect. The conclusion concerning resource-input factors should probably be modified and somewhat ‘nuanced,’ given the results of more recent studies referred to in the above, e.g., the results of recent studies concerning class-size reduction.

There is an interesting difference between the relatively small effect size for the school level variables reported in the meta-analysis and the degree of certainty and consensus on the relevance of these factors in the more qualitative research reviews.

It should be noted that the three blocks of variables depend on types of studies using different research methods. Education production function studies depend on statistics and administrative data from schools or higher administrative units, such as districts or states. School effectiveness studies focusing at school level factors are generally carried out as field studies and surveys, whereas studies on instructional effectiveness are generally based on experimental designs.

4. Foundational And Fundamental School Effectiveness Studies

Foundational school effectiveness studies refer to basic questions about the scope of the concept of school effectiveness. Can a school be called effective on the basis of achievement results measured only at the end of a period of schooling, or should such a school be expected to have high performance at all grade levels? Can school effectiveness be assessed by examining results in just one or two school subjects, or should all subject matter areas of the curriculum be taken into account? Also, shouldn’t one restrict the qualification of a school being effective to consistently high performance over a longer period of time, rather than a ‘one shot’ assessment at just one point in time?

Fortunately all of these questions are amenable to empirical research. These type of studies that are associated with the consistency of school effects over grade-levels, teachers, subject-matter areas, and time have been referred to as ‘foundational studies’ (Scheerens 1993) because they are aimed at resolving issues that bear upon the scope and ‘integrity’ of the concept of school effectiveness.

A recent review of such foundational studies is given in Scheerens and Bosker (1997, Chap. 3). Their results concerning primary schools are presented in Table 3. Consistency is expressed in terms of the correlation between two different rank orderings of schools. Results are based on arithmetic and language achievement.

The results summarized in Table 3 indicate that there is a reasonable consistency across cohorts and subjects, while the consistency across grades is only average. Results measured at the secondary level likewise show reasonably high stability coefficients (consistency across cohorts), and somewhat lower coefficients for stability across subjects (e.g., in a French study (Grisay 1996), coefficients based on value-added results were 0.42 for French language and 0.27 for mathematics). The average consistency be- tween subjects at the secondary level was somewhat lower than in the case of primary schools (r about 0.50). This phenomenon can be explained by the fact that, at the secondary level, different teachers usually teach different subjects, so that inconsistency is partly due to variation between teachers.

The few studies in which factor analysis was used to examine the size of a stable school factor relative to year specific and subject specific effects have shown results varying from a school factor explaining 70 percent of the subject and cohort specific (gross) school effects, to 25 percent.

The picture that emerges from these studies on the stability and consistency of school effects is far from being indifferentially favorable with respect to the unidimensionality of the school effects concept.

Consistency is fair, when effects at the end of a period of schooling are examined over a relatively short time interval. When grade-level and subject matter area is brought into the picture, consistency coefficients tend to be lower, particularly when different teachers teach different grades or subjects. School effects are generally seen as teacher effects, especially at the secondary school level

The message from these ‘foundational studies’ is that one should be careful not to overgeneralize the results of school effectiveness studies when only results in one or two subject matter areas at one point in time are measured. Another implication is that hypothetical antecedent conditions of effects are not only to be sought at the school organizational level, but also at the level of teaching and the teaching and learning process.

Fundamental school effectiveness studies are theory and model-driven studies. Bosker and Scheerens (1994) presented alternative causal specifications of the conceptual multilevel models referred to in an earlier section. These models attempt to grasp the nature of the relationships between e.g., schools and classroom conditions. For example, whether such relationships are additive, interactive, reciprocal, or form a causal chain.

Other studies that have attempted to formalize these types of relationships are by Hofman (1995) and Creemers and Reezigt (1996). In general, it appeared to be difficult to establish the better ‘fit’ of one of the alternative model specifications. More complex models, based on the axioms of microeconomic theory, have been tested by de Vos (1998), making use of simulation techniques.

So far, these studies are too few to draw general conclusions about the substantive outcomes; continuation of this line of study is quite interesting, however, also from a methodological point of view.

From a substantive point of view educational effectiveness studies have indicated the relatively small effects of these conditions in developed countries, where provisions are at a uniformly relatively high level. At the same time the estimates of the impact of innate abilities and socioeconomic background characteristics—also when evaluated as contextual effects—seem to grow, as studies become more methodologically refined. Given the generally larger variation in both conditions and outcomes of schooling in developing countries—and the sometimes appallingly low levels of both—there is both societal and scientific relevance in studying school effectiveness in these settings.

5. The Future Of School Effectiveness Research

From this research paper it could be concluded that school effectiveness research could be defined in a broad and a narrow sense. In the broadest sense one could refer to all types of studies which relate school and classroom conditions to achievement outcomes, preferably after taking into account the effects of relevant student background variables. In a narrower sense, state-of-the-art integrative school effectiveness studies and foundational and fundamental studies could be seen as the core.

Following the broader definition the future of school effectiveness studies is guaranteed, particularly in the sense of ‘applied’ studies, like cohort studies, large-scale effect studies carried out for accountability purposes, monitoring studies, and assessment studies.

State-of-the-art, fundamental and foundational school effectiveness studies are a much more vulnerable species. One of the major difficulties is the organizational complexity and costs of the ‘state-of-the-art types’ of study. Given the shortage of these kind of studies the more fundamental and foundational types of studies are likely to be dependent on the quality of data-sets that have been acquired for ‘applied purposes.’

Therefore the best guarantee for continued fundamental school effectiveness research lies in the enhanced research technical quality of applied studies. One example consists of quasi-experimental evaluation of carefully designed school improvement projects. Another important development is the use of IRT (Item Response Theory) modeling and ‘absolute’ standards in assessment programs. If school effects can be defined in terms of distance or closeness in average achievement to a national or even inter- national standard, some of the interpretation weak- nesses of comparative standards belong to the past.

Bibliography:

1. Bosker R J, Scheerens J 1994 Alternative models of school effectiveness put to the test. In: Bosker R J, Creemers B P M, Scheerens J (eds.) Conceptual and Methodological Advances in Educational Effectiveness Research, special issue of the International Journal of Educational Research 21: 159–80
2. Brandsma H P 1993 Basisschoolkenmerken en de Kwaliteit an het Onderwijs [Characteristics of primary schools and the quality of education]. RION, Groningen, The Netherlands
3. Creemers B P M 1994 The Effective Classroom. Cassell, London
4. Creemers B P M, Reezigt G J 1996 School level conditions affecting the effectiveness of instruction. School Effectiveness and School Improvement 7: 197–228
5. Fraser B L, Walberg H J, Welch W W, Hattie J A 1987 Syntheses of educational productivity research. Special Issue of the International Journal of Educational Research 11
6. Grisay A 1996 Evolution des acquis cognitifs et socio-affectifs des eleves au cours des annees de college [Evolution of Cognitive and socio-affective outcomes during the years of secondary education]. Universite de Liege, Liege, Belgium
7. Hanushek E A 1979 Conceptual and empirical issues in the estimation of educational production functions. Journal of Human Resources 14: 351–88
8. Hedges L V, Laine R D, Greenwald R 1994 Does money matter? A meta-analysis of studies of the effects of differential school inputs on student outcomes. Educational Researcher 23: 5–14
9. Hofman W H A 1995 Cross-level relationships within effective schools. School Effectiveness and School Improvement 6: 146–74
10. Levine D U, Lezotte L W 1990 Unusually Effective Schools: A Review and Analysis of Research and Practice. National Center for Effective Schools Research and Development, Madison, WI
11. Purkey S C, Smith M S 1983 Effective schools: A review. The Elementary School Journal 83: 427–52
12. Sammons P, Hillman J, Mortimore P 1995 Key Characteristics of Effective Schools: A Review of School Effectiveness Research. OFSTED, London
13. Scheerens J 1990 School effectiveness and the development of process indicators of school functioning. School Effectiveness and School Improvement 1: 61–80
14. Scheerens J 1992 Effective Schooling. Research, Theory and Practice. Cassell, London
15. Scheerens J 1993 Basic school effectiveness research: Items for a research agenda. School Effectiveness and School Improvement 4: 17–36
16. Scheerens J 1999 School Effectiveness in Developed and Developing Countries: A Review of the Research Evidence. World Bank paper, Washington, DC
17. Scheerens J, Bosker R J 1997 The Foundations of Educational Effectiveness. Elsevier Science Ltd., Oxford, UK
18. Scheerens J, Creemers B P M (eds.) 1989 Developments in school effectiveness research. Special themed issue of the International Journal of Educational Research 13
19. Stringfield S C, Slavin R E 1992 A hierarchical longitudinal model for elementary school eff In: Creemers B P M, Reezigt G J (eds.) Evaluation of Effectiveness. ICO-Publication 2. GION, Groningen, The Netherlands
20. de Vos H 1998 Educational Effects: A Simulation-Based Analysis. University of Twente, Enschede, The Netherlands