Peer review valuable; helps catch mistakes

By Peter A. Freeman

My article, "Flawed report on Ph.D. production creates stir," describes a New York Times article on a recent study. Without getting into details of the study, I think this whole episode sadly illustrates two important points that too often are overlooked.

Point 1: Peer review is extremely valuable.

The study was released without any formal peer review as far as I can tell. It is simply an unbound report available from the authors. It contains typos, and some of the underlying data--obtained from the National Science Foundation--have some anomalies in it obvious to anyone who knows the reality of particular departments. Worse, the study has some grievous errors of the "garbage in, garbage out" variety resulting from an inadequate examination of the data used to drive the model. Whether other errors are still lurking, either in their assumptions or in the base data, is unknown.

Peer review is no more perfect than program testing. But its absence can be disastrous.

Point 2: Publicity, even in reputable publications, may be totally misleading.

The Times article stated as fact--in its title--that "Supply exceeds demand for Ph.D.s in many science fields." In the discussion of computer science, specifically, the article stated, "The surplus of doctoral computer science degrees currently awarded over the number of those who get desirable jobs in their field is 50.3%." This statement strongly implies that the William F. Massy-Charles A. Goldman report is a survey of current graduates.

However, in its first sentence, the study notes: "This report describes a simulation [emphasis added] of the supply and demand for science and engineering doctorates." We all know that the correction to this misleading article, if ever published, will never have the impact the original article had.

There are important points concerning Ph.D. production that need to be discussed. A simulation--possibly even this one, when corrected--would be a useful tool to aid those discussions. Unfortunately, serious discussions now will be harder because some researchers have widely distributed work that was unreviewed and erroneous, and a reporter distorted what they said.