Peer Review And Quality Control Research Paper

In most other spheres of institutional activity, the formalization of quality assurance has become the norm, as for example through the wide-ranging standard-setting procedures of the International Standards Organization (ISO). In academic science, however, along with cultural pursuits like the arts, the methods are still largely informal. Science has been almost unique in having self-assessment performed by practitioners rather than by external ‘critics.’ To what extent and in what ways this must change to keep pace with science’s expanding role in public life has become an urgent question in the governance of science.

The assurance of quality is not a straightforward task. This has been known since the time of the Romans, as indicated by the Latin motto quis custodiet ipsos custodes? (Who will guard the guardians themselves?). This motto implies an indefinite iteration. It is a reminder that, however, routine may be the tasks of quality control, full quality assurance demands yet higher levels of supervision at which informality and explicit value judgments are necessary.

As long as science remained mainly academic, problems of quality were assumed to be resolved by the very nature of the scientific endeavor. The informal systems of checking by peers seemed a rational response to the problem, rather than a culturally contingent mechanism characteristic of a particular epoch. Scientific facts were believed to be discovered by an infallible method, and scientists themselves were viewed as being endowed with certain superior moral qualities that protected them and their work from ordinary human failure or error. This self-correcting property of science could be explained in sociological terms, as in the ‘four norms’ of scientific practice expounded by Robert K. Merton in 1942 (Merton 1973), or philosophically, as in the committed attempts at self-refutation supposed by Karl Popper to be normal scientific practice (Popper 1959).

With the onset of the industrialization of science after World War II, the self-conscious study of science as a social activity, including the methods of quality assurance, became inevitable. Growth in size, capital investment, scale, and social differentiation within science created divisions between managers and research workers, as well as between researchers and teachers in universities. A Gemeinschaft (community) of scholars could no longer realistically be assumed. The earliest disciplined analyses of the quality of production in science were quantitative. Derek J. de Solla Price, who devised some measures of quality and provided analyses of its distribution, did the pioneering work. He noticed that at the leading British science reference library only a minority of journals was ever requested. The contents of the others could be inferred to have no interest, and hence to be of very low scientific quality (Price 1963). This phenomenon is a reminder that ‘quality’ is a relational attribute. ‘Fitness for purpose’ depends on whose purposes are dominant; not always perhaps those of a community devoted to the advancement of learning, but possibly only of those scientists working under constraints of ‘publish or perish.’

Price’s studies were continued in two directions. At the Institute for Scientific Information, Eugene Garfield produced more searching and sophisticated measures of quality, using citations rather than mere number of publications. Such attempts at quantification were bound to become controversial (Brooks 1982, Garfield 1970, 1972). It was impossible to avoid bias in the selection of the relatively small set of journals used for citations; those in conventional mainstream English-language research science were inevitably privileged at the expense of all others. Further, when quantitative measures of citations came to be used as indicators of academic merit, manipulative practices, including reciprocal citations, inevitably developed. The deep problems of a quantitative proxy for quality suddenly became acute.

In a more reflective vein, Jerome R. Ravetz applied the quis custodiet principle to analyze the vulnerability of the quality assurance system in science. He observed that the processes of peer review are conducted largely informally and (unlike research) are not themselves normally submitted to open scrutiny and review. They require a different sort of competence, which is not part of the formal training of scientists; and they are also more open to a variety of abuses, ranging from bias to plagiarism. One can understand the phenomena of low quality, both in scientific research and in technological development in these terms. Thus, while denying that the practice of science automatically produces a higher morality, Ravetz agrees that moral standards are necessary for the successful practice of science. On this basis he stresses the importance of morale and morality (and ultimately idealism and leadership) in science (Ravetz 1996).

This analysis provides a background for the increasing interest in ‘trust’ as an essential element of practice in science, in society, and in their interactions. The broader society has provided resources to the esoteric activities of science because it trusts the scientific community to make good use of them. There has always been an undercurrent of distrust, based on evidence either of pointless research or of malign applications. Once science became deeply involved in technology and related policy problems that crucially affect public health and welfare, the traditional relations of trust could no longer be assumed. It appeared to be necessary for the principles and practices of accountability to be extended from the institutions of political governance (as, e.g., representative democracy) to those institutions, which govern science and its applications.

Quality control in research science has become more difficult as the relatively inflexible technical requirements of the traditional printing process have been relaxed. There is no longer a well-defined ‘gateway’ to publication through the institutions that control reproduction of, and hence access to documents. First through inexpensive photocopying and now through the Internet, it has become easy for anyone to distribute scientific wares to an unrestricted audience. In addition, the presence of the global media tends to bypass the traditional processes of evaluation, which were conducted personally among colleagues. Isolated scientific results can become media events (Close 1991). All those with an interest in the report, such as consumers, politicians, regulators, and the stock markets, become potential stakeholders in the evaluation of the result. Thus, science arguably becomes accountable to a drastically extended peer community in the quality-assurance process. The criteria of quality applied by these heterogeneous actors need not be identical to those of ‘public knowledge’ generated within tightly knit scientific networks.

These developments may be judged in different ways. While they may seriously disrupt the procedures of quality assurance in normal science, they can also bring needed public scrutiny to bear on controversies and scandals. The demystification of scientific practice both enables such events to become news, and is fostered by their being exposed. Top scientists become like celebrities—needing the media for advertising themselves yet simultaneously hating it for its unwanted intrusions. The ‘Baltimore affair,’ centering on the US Nobel laureate David Baltimore’s laboratory at MIT, is perhaps the most notorious case in which a dispute about scientific misconduct was blown up into a lengthy, visible, political saga that damaged all the individuals and institutions involved (Kevles 1998). The episode was symptomatic of an increasingly recognized problem of ‘deviance’ in science, which carries the unspoken danger that, without timely correctives, misconduct might become the norm.

All these developments affect the maintenance of trust, which is necessary for ordinary scientific practice and even more for quality assurance. As in other professional domains, the normal tendency in science has been for misconduct to be covered up by the responsible institution (not necessarily by the community of scientists). In such situations, ultimate exposure does even more damage and further erodes the basis for mutual trust. Attempts to circumvent the need for trust by increasing bureaucratic surveillance are likely to be counterproductive in their own way, by erecting impediments to free inquiry and communication among colleagues.

The relations between social science and natural science have also been transformed during the last decades, with implications for quality control. Starting with the acceptance of natural science as the ideal of knowledge, essentially independent of social forces, there has been a gradual but accelerating shift toward recognizing all sciences as incorporating social constraints and biases. An early critical interaction was in connection with the astronomical community’s management of the eccentric Velikovsky (de Grazia 1966). Later, the social science community embraced Thomas Kuhn’s disenchanted picture of ‘normal’ science (Kuhn 1970). Finally, post-Feyerabend studies of science re-examined the whole institution of scientific inquiry without presupposing any privileged status in relation to either virtue or natural knowledge (Bloor 1991, Bloor et al. 1995, Collins and Pinch 1993, Fuller 1993).

When natural scientists, led by physicists, eventually confronted the emerging socialized picture of their discipline, the reaction was so strident that ‘science wars’ became an appropriate label (Gross et al. 1997, Nelkin 1996, Ross 1996). Sociologists of science and post modernists were indiscriminately blamed for all the ills of science, including decline of public trust, budget cuts, resurgent Creationism, and even poor teaching of science. A physicist whose hoax article (Sokal 1996) was accepted by a leading cultural studies journal, Social Text, helped to crystallize the attack (Bricmont and Sokal 1998). The implication was that the critics of science had no real quality control of their productions. The science warriors’ assumption was that within real science, such problems are prevented from occurring because of the verifiable empirical content of scientific research. In the ensuing debate, there was little mention of the ease of publication of erroneous or vacuous research in the standard scientific literature. Historical episodes, like Millikan’s manipulation of his oil-drop results in the course of a controversy on the charge of the electron, were discounted as mere embarrassments (Segerstale 1995).

It has been presupposed thus far that ‘science’ refers primarily to traditional basic research. But among contemporary forms of scientific practice, curiositydriven research with no regard for applications has been increasingly marginalized. A diversification has occurred, so that quality assurance must also be considered in such areas as mission-oriented and issuedriven research, forensic science (Foster and Huber 1997, Jasanoff 1995), and the provision of scientific advice for policy (Jasanoff 1990, Salter 1988). In addition, the products themselves and the media through which they are diffused increasingly are diversified. For example, patents are now a common outcome of a research process, and this form of intellectual property is radically different from traditional published papers (Myers 1995). Also, results are reported in unpublished consultancy advice and unpublished ‘gray literature’ or kept confidential within institutions or even totally sealed under ‘lawyer client confidentiality’ and legal settlement agreements. With traditional peer review as the norm, the challenges of quality assurance for these new products and processes are nearly unrecognizable. A genre of critical literature has developed, with some authors directing anger at the new contexts of scientific production (Huber 1991), and others more clearly appreciating the problems they present (Crossen 1994, Jasanoff 1990, 1995).

A parallel diversification has occurred in the types of knowledge production that are accepted as legitimate. The democratization of knowledge now extends beyond the juries who assess the quality of technical evidence in courts (Jasanoff 1998) to include those who master previously esoteric aspects of their predicament (e.g., illness, contamination, pollution, oppression, discrimination, exploitation) through special-interest groups or the Internet. In addition, claims of specialized or local knowledge are present in even more diverse contexts, as among indigenous peoples, and in systems of complementary or ‘traditional’ medicine. These claims are commanding increasing commercial and political support among various publics, as well as gaining explicit recognition in numerous international treaty regimes. As a result, a new philosophy of knowledge appears to be emerging, based on a new disciplined awareness of complexity, in which a plurality of legitimate perspectives is taken for granted (Funtowicz and Ravetz 1991). Modern science, with its characteristic methodology and social location, is part of this enriched whole, but not coextensive with it. The criteria and tasks of quality assurance must explicitly involve additional values and interests, incorporating even the ontological commitments of groups other than scientists. This new configuration has been termed postnormal science.

Quality assurance can, thus, be seen as a core commitment of postnormal science, replacing ‘truth’ as science’s ultimate regulative principle (Funtowicz and Ravetz 1992). Defined in terms of uncertainties and decision-stakes, quality assurance encompasses ‘public interest,’ ‘citizen,’ and ‘vernacular’ sciences. In a period of domination by globalized corporate science (Gibbons et al. 1994), this effort to make scientists accountable to interested groups presents a coherent conceptual alternative for the survival of the ‘public knowledge’ tradition of science. Collegial peer review is thereby transformed into review by an ‘extended peer community.’ This new form of quality assurance will be given its formal structure and routines by those heterogeneous actors who put it into practice.

Bibliography:

1. Bloor D 1991 Knowledge and Social Imagery. University of Chicago Press, Chicago
2. Bloor D, Edge D, Henry J 1995 Scientific Knowledge. Athlone, Chicago
3. Bricmont J, Sokal A D 1998 Fashionable Nonsense: Post-modern Intellectuals’ Abuse of Science. Picador, New York
4. Brooks H 1982 Science indicators and science priorities. In: La Follette M C (ed.) Quality in Science. MIT Press, Cambridge, MA, pp. 1–32
5. Close F H 1991 Too Hot to Handle: The Race for Cold Fusion. Princeton University Press, Princeton, NJ
6. Collins H, Pinch T 1993 The Golem: What Everyone Should Know about Science. Cambridge University Press, Cambridge, UK
7. Crossen C 1994 Tainted Truth: The Manipulation of Fact in America. Simon & Schuster, New York
8. Foster K R, Huber P W 1997 Judging Science—Scientific Knowledge and the Federal Courts. MIT Press, Cambridge, MA
9. Fuller S 1993 Philosophy, Rhetoric and the End of Knowledge: The Coming of Science and Technology Studies. University of Wisconsin Press, Madison, WI
10. Funtowicz S O, Ravetz J R 1991 A new scientific methodology for global environmental issues. In: Costanza R (ed.) Ecological Economics. Columbia University Press, New York, pp. 137–52
11. Funtowicz S O, Ravetz J R 1992 Three types of risk assessment and the emergence of post-normal science. In: Krimsky S, Golding (eds.) Social Theories of Risk. Greenwood Press, Westport, CT, pp. 251–73
12. Garfield E 1970 Citation indexing for studying science. Nature 227: 669–71
13. Garfield E 1972 Citation analysis as a tool in journal evaluation. Science 178: 471–9
14. Gibbons M C, Limoges C, Nowotny H, Schwartzman S, Scott P, Trow M 1994 The New Production of Knowledge. Sage, Beverley Hills, CA
15. de Grazia A (ed.) 1966 The Veliko sky Affair: The Warfare of Science and Scientism. University Books, New York
16. Gross P R, Levitt N, Lewis M W (eds.) 1997 The Flight from Science and Reason. Johns Hopkins University Press, Baltimore, MD
17. Huber P W 1991 Galileo’s Revenge: Junk Science in the Court- room. Basic Books, New York
18. Jasanoff S 1990 The Fifth Branch: Science Advisors as Policy as Policymakers. Harvard University Press, Cambridge, MA
19. Jasanoff S 1995 Science at the Bar: Law, Science and Technology in America. Harvard University Press, Cambridge, MA
20. Jasanoff S 1998 The eye of everyman: Witnessing DNA in the Simpson trial. Social Studies of Science 28(5–6): 713–40
21. Kevles D J 1998 The Baltimore Case: A Trial of Politics, Science and Character. Norton, New York
22. Kuhn T S 1970 The Structure of Scientific Revolutions, 2nd edn. University of Chicago Press, Chicago
23. Merton R K 1973 The Normative Structure of Science. University of Chicago Press, Chicago
24. Myers G 1995 From discovery to invention: the writing and rewriting of two patents. Social Studies of Science 25(1): 57–105
25. Nelkin D 1996 What are the science wars really about. The Chronicle of Higher Education July 26: A52
26. Popper K 1959 The Logic of Scientific Discovery. Basic Books, New York
27. Price de Solla D J 1963 Little Science, Big Science. Cambridge University Press, Cambridge, UK
28. Ravetz J R 1996 (1971) Scientific Knowledge and its Social Problems. Transaction Publishers, New Brunswick, NJ
29. Ross A (ed.) 1996 Science Wars. Duke University Press, Durham, NC
30. Salter L 1988 Mandated Science. Kluwer, Dordrecht, The Netherlands
31. Segerstale U 1995 Good to the last drop? Millikan stories as ‘canned’ pedagogy. Science and Engineering Ethics 1: 197–214
32. Sokal A D 1996 Transgressing the boundaries. Social Text 14: 217–52
33. Stampa A E III 1997 Advances in peer review research. Science and Engineering Ethics 3(1): 1–104