Instructions for Rating the Referee Reviews

 A referee review should be "substantive, both quantitatively and qualitatively;...respectful, competent, and helpful to the author."  They should also lead to a recommendation that is knowledgeable and sustained by the review.  Personalized preferences should not appear.  The review should be clear and clearly written.

 Reviews should not concern themselves with the direction of the findings and conclusions (positive or negative) but only the logic of the paper, that is, whether the conclusions were sustained by the methodology.  Positive or negative findings are besides the point if the introduction proves the value of the research topic and if the methodology is credible.  Reviews should not editorialize, making comments about the desirability of the services except when they intrude upon the internal logic of the paper being reviewed.
 
 The enclosed two sets of reviews should be graded in the following way:

1= Excellent.  The review covered all relevant points very well, acknowledging the strengths and detailing the weaknesses.  It commented on the value of the research.  It sustained its recommendation.  It did not contain obvious errors of logic.  The review from the JCCP of the family preservation paper is an example of an excellent review.

2= Adequate.  The review covered many relevant points and sustained its recommendations.  Its deviations from logic were minor.  It sustained its recommendations.

3= Inadequate but marginal.  The review missed much, commented on findings rather their logic, and did not sustain its recommendations.

4= Poor.  The review said little about the paper's methods or relevance.

5= Very poor.  The review misunderstands research in general and lacks any awareness of the failures of the paper.  It is not really a review but simply written proof of the referee's existence.

 The length of the review should not matter although reviews that are short can not possibly cover all the problems of these papers.  On the other side, long reviews concerned with trivial matters (punctuation, sentence structure, etc.) are trivial.  Copy editing is not the major obligation of the referee; substance is.

 Both papers were greatly flawed:

 Paper 1 reported on an experiment in family preservation.  It lacked a true randomized control.  Its introduction was slipshod and occasionally inaccurate.  The interventions were not clearly described.  Its tables were poorly presented; its analysis undetailed; statistical significance was reported without identifying the statistical test; the conclusions do not flow logically out of the methodology.

 Paper 2 reported on an experiment in preventing repeat pregnancy among unwed teenage mothers.  It did contain a randomized control but this was its only virtue.  The paper contained all of the other flaws in paper 1.

 Each review is identified by a three digit number in the upper right corner: the reviews of paper 1 begin with 0 and those for paper 2 with 1.  Please enter your assessment of each review on the attached rating sheet.  Please send the completed ratings to me or fax it to (702) 895-4079.