Abstract of Chapter:
EVALUATION OF INTERDISCIPLINARY RESEARCH
Oxford Handbook of Interdisciplinarity
Katri Huutoniemi (July 23, 2007)
While interdisciplinarity has become a major topic in discussions of knowledge production and research funding, evaluation and criteria remain poorly understood aspects. This chapter reviews current understanding of how to evaluate research that goes beyond academic disciplines, either by crossing boundaries between several disciplines (interdisciplinarity) or involving also stakeholders outside of academia (transdisciplinarity). Appropriate evaluation criteria and results are needed to give useful information not only to those who invest in research – research councils, funding agencies, universities, and research institutes – but also to the researchers whose work is being evaluated and their various stakeholders.
The evaluation of research is based on measuring quality, ranging from quality control to quality enhancement. The underlying dilemma in inter- and transdisciplinary research evaluation is that quality assessment of knowledge production is rooted in disciplinary standards and practices, but the different perspectives involved in integrative activities may be epistemologically irreconcilable. This raises questions about both the contents and procedures of research evaluation: How can a balance be achieved between different epistemic aspects, and what criteria may be used to assess them? How to organize the evaluation, and who to include in the process? The chapter provides a review of the state-of-the-art literature on these issues. Included are insights from conceptual and pragmatic discussions on integrative research, empirical studies on research evaluation, and experiences and initiatives of participating organizations.
Numerous scholars of interdisciplinarity connect the challenges of interdisciplinary evaluation to the lack of available criteria, since the epistemological standards of disciplinary communities are proving insufficient or even irreconcilable to evaluate research that expands, integrates, or challenges the disciplinary canon (Klein 2005; Laudel & Origgi 2006; Boix-Mansilla 2006; Boix-Mansilla et al. 2006; Feller 2006). Social studies of disciplinary cultures have revealed that both the assessment of competence and scholarly identity are strongly tied to a discipline-specific set of assumptions, since Ňdisciplines shine under varying lights and their members filter quality through specific prismsÓ (citation by Lamont, in press; see also Becher 1989, Knorr-Cetina 1999, Kekle 2002).
What does quality actually mean for research that transcends disciplinary boundaries? Against what standards should it be assessed, and are new ones required? Scientific quality is the basis of producing good and reliable new knowledge, but the complexity of inter- and transdisciplinary research defies a single standard (for a review of literature, see Klein 2006). Empirical observations show that the notions of interdisciplinarity and transdisciplinarity are employed to characterize a broad variety of research practices located at multiple points of intersection in the intellectual landscape of academe. Since there is no single understanding of inter- or transdisciplinarity that corresponds to the understanding of disciplines, generic operational criteria for the quality of such research hardly exists. Substantive questions need to be assessed with an eye to the specific content of the project to be evaluated. Evaluating boundary-crossing research is thus a local issue, determined by relations within the intellectual environment and goals of the particular research (Spaapen & Dijstelbloem 2005, Bezzi 2006).
The local character of evaluation does not mean, however, that one could not learn from analogical processes elsewhere. Emergent commonalties have motivated scholars to analyze and compare concrete evaluation processes and measures of quality in different contexts. Representative overview appears in a special issue of Research Evaluation (April 2006), including an investigation of how academics in established interdisciplinary research institutions define quality of their work (Boix-Mansilla 2006); studies on interdisciplinary expert panelsŐ work and its impact on quality judgments (Langfeldt 2006; Lamont et al. 2006); and an analysis of review processes of interdisciplinary research networks (Laudel 2006). These studies have contributed to our analytical understanding of the topic both by providing empirical information about what is valued in interdisciplinary science, and by illuminating the process through which the institutionalized standards of quality are set.
Other studies have looked at the system features of scientific evaluation from the perspective of inter- and transdisciplinary research. Both qualitative and quantitative accounts show that the mechanisms of the evaluation system are unable to acknowledge the benefits, innovativeness, and even revolutionary voice of inter- and transdisciplinary approaches. Particular risks have been identified in traditional peer review process (Porter & Rossini 1985, Travis & Collins 1991, Becher 1989). Studies have concluded that in the presence of administrative, funding, and cultural barriers between research departments, collaboration across disciplinary boundaries needs special support. To make sure that boundary-crossing research is not the looser in the assessment process, the evaluation system should be revised. (Ibid, Feller 2006)
Research organizations have actively promoted the pragmatic discussion of interdisciplinary evaluation, since guidelines for appropriate criteria and procedures are strongly needed. Examples of conceptual and/or programmatic frameworks of general validity are DefilaŐs and DiGiulioŐs (1999) catalogue of criteria for inter- and transdisciplinary research, KleinŐs (2005) guiding questions for integration, Stokols et al.Ős (2003) logic model for evaluating the collaborative processes and outcomes of transdisciplinary science, Bergmann et al.Ős (2005) operational criteria for the formative self-evaluation of transdisciplinary research projects, and National Academy of SciencesŐs (FacilitatingÉ2004) guide for facilitating interdisciplinary research. In addition, many national research councils have undertaken empirical analyses on evaluation as part of their own monitoring process (see e.g. Bruun et al. 2005, Grigg 1999, Sandstrm et al. 2005, Tait & Lyall 2001). These practice-oriented studies have identified many principles of evaluation that match the cross-cutting nature of inter- and transdisciplinary research.
Whereas the evaluation of interdisciplinary and transdisciplinary research has recently raised special concern, it is more rarely asked to what extent this discussion should be extended to other research as well. Although interdisciplinary research has by definition many characteristics that make it particularly difficult to evaluate, it is important to note that there is much contingency and variation within disciplinary research, too. Quality and performance are relative not only to disciplinary cultures, but also to the goals, expectations, norms, and values of stakeholders and thus vary from one evaluation context to another. In the era of knowledge society, the contextualization of problems and social accountability of knowledge production have become as important indicators of quality as scientific reliability and disciplinary rigor (Gibbons et al. 1994, Nowotny et al. 2001, Hemlin & Rasmussen 2006). The chapter concludes with a comparative analysis on the evaluation of interdisciplinary contra disciplinary research, and questions whether the challenges of the former are, in the end, so distinct from the more general problematique of research evaluation. Might interdisciplinarity give an illustrative example of the need to develop the whole evaluation system into more flexible, context-sensitive, participatory, and far-seeing direction?
References
Becher T, 1989. Academic tribes and territories. Intellectual enquiry and the culture of disciplines. Buckingham and Bristol: The Society for Research into Higher Education & Open University Press.
Bergmann M, Brohmann B, Hoffmann E, Loibl MC, Rehaag R, Schramm E, Vo§ J-P, 2005. Quality criteria of transdisciplinary research: A guide for the formative evaluation of research projects. Frankfurt am Main: Institute for Social-Ecological Research (ISOE)
Bezzi C, 2006. Evaluation pragmatics. Evaluation 12(1): 56-76.
Boix-Mansilla V, 2006. Assessing expert interdisciplinary work at the frontier: An empirical exploration. Research Evaluation 15(1): 17-29.
Boix-Mansilla V, Feller I, Gardner H, 2006. Quality assessment in interdisciplinary research and education. Research Evaluation 15(1): 69-74.
Bruun H, Hukkinen J, Huutoniemi K, Klein J T, 2005. Promoting interdisciplinary research: The case of the Academy of Finland. Publications of the Academy of Finland 8/05. Helsinki: Academy of Finland.
Defila R, DiGiulio A, 1999. Evaluating transdisciplinary research. In PANORAMA [Newsletter of the Swiss Priority Program Environment, Swiss National Science Foundation] 1/99
Facilitating interdisciplinary research, 2004. National Academy of Sciences. Washington, DC: National Academies Press.
Feller I, 2006. Multiple actors, multiple settings, multiple criteria: Issues in assessing interdisciplinary research. Research Evaluation 15(1): 5-15.
Grigg L. 1999. Cross-disciplinary research. A discussion paper. Commissioned Report No. 61. Canberra: Australian Research Council.
Gibbons M, Limoges C, Nowotny H, Schwartzman S, Scott P, Trow M, 1994. The New Production of Knowledge. The Dynamics of Science and Research in Contemporary Societies. London: SAGE Publications.
Hemlin S, Rasmussen S B, 2006. The shift in academic quality control. Science, Technology & Human Values 31(2): 173-198.
Kekle J, 2002. Conceptions of quality in four different disciplines. Tertiary Education and Management 8(1): 65-80.
Klein J T, 2006. Afterword: The emergent literature on interdisciplinary and transdisciplinary research evaluation. Research Evaluation 15(1): 75-80.
Klein J T, 2005. Guiding questions for integration. Integration Symposium 2004 - Proceedings. Canberra: Land and Water Australia.
Knorr-Cetina K, 1999. Epistemic cultures. How the sciences make knowledge. Cambridge and London: Harvard University Press.
Lamont M, In press. Cream rising: Finding and defining excellence in the social sciences and the humanities.
Lamont M, Mallard G, Guetzkow J, 2006. Beyond blind faith: Overcoming the obstacles to interdisciplinary evaluation. Research Evaluation 15(1): 43-55.
Langfeldt L, 2006. The policy challenges of peer review: Managing bias, conflict of interests and interdisciplinary assessments. Research Evaluation 15(1): 31-41.
Laudel G, 2006. Conclave in the tower of babel: How peers review interdisciplinary research proposals. Research Evaluation 15(1): 57-68.
Laudel G, Origgi G, 2006. Introduction to a special issue on the assessment of interdisciplinary research. Research Evaluation 15(1): 2-4.
Nowotny H, Gibbons M, Scott P, 2001. Re-thinking science: Knowledge and the public in an age of uncertainty. Oxford: Polity Press.
Porter A L, Rossini F A, 1985. Peer-review of interdisciplinary research proposals. Science, Technology & Human Values 10(3): 33–38.
Research Evaluation, 2006, 15(1): 1-80. Special issue on interdisciplinary research assessment.
Sandstrm U, Friberg M, Hyenstrand P, Larsson K, Wadskog D, 2005. Tvrvetenskap – en analys. Stockholm: Swedish Research Council (Vetenskapsrdet)
Spaapen J, Dijstelbloem H, 2005. Evaluating research in context: A method for assessment. The Netherlands: Consultative Committee of Sector Councils for Research and Development (COS)
Stokols D, Fuqua J, Gress J, Harvey R, Phillips K, Baezconde-Garbanati L, et al, 2003. Evaluating transdisciplinary science. Nicotine & Tobacco Research 5, Supplement 1: S21-39.
Tait J, Lyall C. 2001. Investigation into ESRC funded interdisciplinary research. Final report. Edinburg: SUPRA.
Travis G D L, Collins H M, 1991. New light on old boys: Cognitive and institutional particularism in the peer review system. Science, Technology & Human Values 16: 322–341.