CSC 400 Assignment: ITRG Panel
CSC 400 Assignment: ITRG Panel NSF panels are tasked to evaluate, through discussion, a set of proposals, come up with an overall ranking, and also a classification for each proposal from the set {highly competitive, competitive, non-competitive}. Highly competitive means the proposal is of excellent quality and should be funded if at all possible. Competitive means the proposal is high quality, and worth funding if funds are available. Non-competitive means the proposal has significant deficits and should not be considered for funding in its current form, though it may be possible to correct the deficits in a future proposal. Most proposals actually funded are ranked highly competitive. For each proposal, the panel prepares a panel summary, briefly describing the reasoning behind the ranking/classification.
Individual panelists are asked to read and review a set of proposals prior to the panel meeting. The written reviews should summarize the merits and deficiencies of the proposals, and also include a ranking from the set {poor, fair, good, very good, excellent}. The two broad evaluation criteria are scientific merit and broader impact. You should read the information at the NSF web site on review criteria before reviewing your proposals. These reviews are returned, anonymously, to the proposal writer along with the panel summary.
There is a definite budget effect. The program director generally tells the panel about how many proposals can be funded overall (in a given program, most are about the same size, since researchers tend to ask for the maximum). The unspoken message is that this places a loose bound on the number of "highly competitive" rankings the panel can return, since researchers expect highly competitive proposals to be funded. This must be a bit flexible, because of small number statistics, but it is a definite constraint. For this exercise, I (as the standin for the program director) want to fund 2 or 3 proposals, which means the expected yield is about 1 highly competitive and one competitive proposal for each panel group. It may be that your group has no proposals you consider highly competitive. This is fine, and you should say so. You may also have more than one highly competitive proposal, but you should be very sure of the fact before assigning such a ranking.
Actual NSF panels often have 15-20 people on them, with 3-4 or so panelists assigned to review each proposal. They spend a couple of days going through perhaps 40 proposals, with the reviewers leading the discussion, and others kibbitzing if they have something to say. Since we don't have a couple of days discussion time, we are cutting out the listening to other dicussions part. This is actually an important part of calibrating the overall panel metric, but we'll try to work around that. Usually, on person is assigned to write the panel summary for each proposal. That should work OK for our panels as well.
Members
Proposals
Members
Proposals
Members
Proposals