Flawed report on Ph.D. production creates stir

By Peter A. Freeman

An article in the July 4 New York Times--"Supply exceeds demand for Ph.D.s in many science fields"--alarmed a number of people in the computing research community because of its strongly worded assertion that "there is a 1-in-4 chance of [a newly minted Ph.D.] being disappointed [in finding a job in their specialty], according to a recent survey."

The article cited computer science as being the worst field:

" 'The surplus of doctoral computer science degrees currently awarded over the number of those who get desirable jobs in their field is 50.3%,' Dr. [William A.] Massy [author of the study] said." Charles A. Goldman of RAND was co-author of the study.

My review of the study, given below, points out, among other things, that the study was merely a simulation, not a survey as reported by the Times, thus making the reported conclusions speculative at best.

The CRA Board appointed a group to review the study more thoroughly. Jeff Ullman of Stanford University and Bob Ritchie of Hewlett-Packard Co. volunteered to meet with the senior author of the study. Just before the scheduled meeting, Massy discovered a major flaw in the input data for computer science.

When corrected, Massy's model predicts a CS overproduction of less than 4% rather than the 50% originally reported--one of the lowest of all disciplines, as opposed to one of the highest. At CRN press time, Massy intended to reissue his report. Various other follow-up actions were under consideration.

In spite of the flaws in the study, however, two points in it are worth serious attention. First, it presents a model that shows how increasing research funding to employ additional Ph.D.s can actually lead to an even more serious problem of oversupply in the future. Second, it implicitly questions whether or not we are providing students in Ph.D. programs with training and education appropriate to their likely employment.

Summary of study: This report describes a simulation of the supply and demand for science and engineering doctorates, and asserts there is a credible case for significant structural underemployment when the supply-and-demand system achieves a steady state. The report implies that this will happen sometime in the future, but the time frame is not clearly laid out.

The model assumes that Ph.D.-granting departments admit students with almost no attention to the demand for completed doctorates. The report backs up this assumption with an extensive set of interviews with faculty throughout the country in all fields.

Another key assumption, based on data and simple reasoning, is that each faculty member in a research university, throughout all fields, produces an average of 10 doctorates during his or her career. This alone is enough in many situations to suggest a structural imbalance. This, in turn, leads to the simulation's conclusion that expanding research funding in the face of a surplus of Ph.D.s, in the long run, will cause an even greater oversupply, assuming that more than one of the 10 goes into a faculty position in which they produce more Ph.D.s.

The simulation is a system of detailed models: faculty career transitions, completion rates and so on. The authors observe that the simulation "boils down to a large first-order Markov process." The report might lead one to believe that the overall simulation is carefully built because of the amount of detail presented. Yet the report apparently is only a technical report, unreviewed at this point; it is dated July 1995 and has numerous typos, suggesting its draft status.

When Ritchie and Ullman visited Massy at Stanford to explore details of the study, he told them he had just discovered that a critical number--total employed doctorates--was incorrect for computer science. As a result, computer science now shows an employment gap of only 3.6% instead of the reported 50.3%.

This number is one of the lowest of all fields and suggests that the estimate of 1,000 CS Ph.D.s/year needed, formulated 15 years ago, is in close agreement with the new Massy-Goldman estimate. Others have raised additional questions about detailed aspects and underlying assumptions of the model.

Some of Massy and Goldman's results are compared to National Science Foundation data, but otherwise the authors do not seem to have done a large amount of validation. Again, it may be buried in the details.

The report seems to indicate that the authors have carefully built the component models, such as faculty transitions, validated them and then put them together into an overall simulation to arrive at their conclusions. Based on our experience so far, however, any detailed conclusions should be looked at skeptically until a more thorough validation exists.

When a more detailed analysis is available, it will be linked into the CRA home page (http://cra.org).

Conclusion: Having a validated model of Ph.D. production, such as this study hoped to provide, would be of great value. Some of the qualitative conclusions, such as those stating that some fields may be producing too many Ph.D.s and that expanding research funding only makes it worse in the long run, are worthy of careful consideration. Likewise, the implicit conclusion that many Ph.D.s are not being trained for the work they wind up doing is something we need to address.

Citation: The Production and Utilization of Science and Engineering Doctorates in the US. William F. Massy, Stanford University, and Charles A. Goldman, RAND. Copies can be obtained for $20 from Massy, Stanford Institute for Higher Education Research, Stanford University, Stanford, CA 94305. E-mail: hk.rac@forsythe.stanford.edu. The report is about 200 pages and includes lots of data tables, but the primary presentations and conclusions are in Chapter 1. The remainder of the study appears to be details of various components of the overall model, such as a model for estimating faculty career transitions.

Peter Freeman is dean of the Georgia Institute of Technology College of Computing. He also is a member of the CRA Board.