Back to November 2002 CRN Table of Contents

[Published originally in the November 2002 edition of Computing Research News, Vol. 14/No. 5.]

The National Research Council is once again undertaking a study of Ph.D. programs at research universities in the United States. The last study, commonly referred to as the "NRC Rankings," was released in 1995 (http://bob.nap.edu/html/researchdoc). According to the NRC, the purpose of the current study is primarily to provide university administrators and faculty with a set of common measures, both quantitative and reputational. Additional information about the assessment is available on the Web at: http://www7.nationalacademies.org/bhew/ResDocs_about.html.

The Computing Research Association was invited by the NRC's Committee to Examine the Methodology for the Assessment of Research-Doctorate Programs (http://www7.nationalacademies.org/bhew/ResDocs_cmte.html) to provide comments and suggestions on the proposed assessment prior to the committee's meeting on September 30. Reprinted below is the text of the letter submitted to the chair of the NRC committee, Dr. Jeremiah P. Ostriker, by CRA board chair, Jim Foley.

The Computing Research Association (CRA) welcomes the opportunity to provide input to the National Research Council's assessment of doctoral programs at research universities. CRA is an association of more than 200 North American academic departments, six computing societies (American Association for Artificial Intelligence, Association for Computing Machinery, Canadian Association of Computer Science, IEEE Computer Society, Society for Industrial and Applied Mathematics, and USENIX Association), and twenty-three industrial laboratories and institutes all engaging in computing research. CRA's mission focuses on representing and strengthening research and graduate education in the computing fields. CRA's members are very aware of the NRC rankings and the importance of the results.

Your letter asks "…how might the next [NRC] study incorporate interdisciplinary programs [and] identify emerging areas that were not included in the 1995 study?" Since 1995, the year the last NRC rankings were released, computing has experienced great change. The emergence of new research areas, sub-disciplines, and interdisciplinary fields within the computer science and engineering fields has reached a critical mass and now requires a new categorization and new methods of assessment. For example, a brief survey of CRA member departments finds computing programs housed in nine different college units from Arts and Sciences to Engineering to Information Technology and Engineering (attached). Our IT Deans' group includes Deans from about 40 computing units that report directly to a Provost or similar official. The names of these units [http://www.cra.org/Activities/itdeans/participants.html] reflect the breadth of computing. Also, the attached article "Computing > Computer Science" [see p. 6 at http://www.cra.org/CRN/issues/0204.pdf] is my own take on the intellectual breadth of computing.

CRA suggests that the committee establish the high-level category "Computer and Information Science and Engineering" under which doctoral programs in computer science, computer engineering, information science, information technology, human-computer interaction, computational science, etc., would be assessed and ranked. In addition, some multidisciplinary programs such as bio-informatics and other "x"-informatics programs may be appropriately housed under this new category.

Our suggestion mirrors the policy of the National Science Foundation which, in 1986, reorganized to create CISE-the Directorate for Computer and Information Science and Engineering-in recognition of computing as an important cluster of disciplines on a par with the traditional disciplines housed in the other Directorates of Engineering, Math and the Physical Sciences, the Biological Sciences, and the Social, Behavioral and Economic Sciences.

In response to the question of how the next NRC study should assess the scholarly reputation of doctoral programs, CRA endorses a rigorous evaluation process that takes into account the differences between computing and other research disciplines in terms of publication modalities. Attached is the CRA white paper "Best Practices in Evaluating Computer Scientists and Engineers for Promotion and Tenure" [http://www.cra.org/reports/tenure_review.html]. As noted in this paper, although "…standard publication is one indicator of academic achievement, other forms of publication, specifically conference publication, and the dissemination of artifacts [such as computer programs and systems] also transmit ideas. Conference publication is both rigorous and prestigious. Assessing artifacts requires evaluation from knowledgeable peers. Quantitative measures of impact are possible, but they may not tell the entire story." CRA strongly recommends that the committee take these differences in academic culture and procedure into consideration when judging a program's "scholarly reputation."

In conclusion, CRA recommends that the committee:

1. create the new category "Computer and Information Science and Engineering" to properly capture and classify Ph.D. programs in the widening areas of computing education and research, and
2. specify a rigorous evaluation of computing programs that takes into account the assessment practices of computer science and engineering.

[end of letter text]

According to the NRC, the first phase of the NRC study will investigate new measures and methodologies so that the second phase, conducted in 2003-05, may accurately reflect changes in scholarship and graduate education that have occurred over the past 20 years.