East Carolina University
Department of Psychology
Ignorant Experts
Dear Students in PSYC 6430,
I hope you enjoyed reading Bradley's articles on his experiences with the publication process:
2. Bradley, J. V. (1984). The complexity of nonrobustness effects. Bulletin of the Psychonomic Society, 22, 250-253.
3. Bradley, J. V. (1981). Overconfidence in ignorant experts. Bulletin of the Psychonomic Society, 17, 82-84.
4. Bradley, J. V. (1981). Pernicious publication practices. Bulletin of the Psychonomic Society, 18, 31-34.
5. Bradley, J. V. (1982). Editorial overkill. Bulletin of the Psychonomic Society, 19, 271-274.
6. Bradley, J. V. (1984). Antinonrobustness: A case study in the sociology of science. Bulletin of the Psychonomic Society, 22, 463-466.
I would like to share with you a recent experience one of my colleagues had with an "ignorant expert" who reviewed one of her manuscripts. I posted the following query on EDSTAT-L and TIPS and ECUPSY-L and got some interesting responses, which I have appended.
One of my colleagues asked me about this criticism she received from
the "expert reviewer" of one of her manuscripts:
"I'm unclear as to why <construct name> scores had 'negative skewness.' The
maximum possible score was 54, and the mean was over 44. Isn't that positive
skewness? In all likelihood, a pictoral depiction of <construct name> scores
would reveal that most of them clustered toward the top end of the
distribution."
This 'expert' went on to argue that transforming the scores by squaring them was
inappropriate and constituted nothing more than a "data manipulation trick" that
invalidated the (normality assuming) analysis which was done on the transformed
data. The untransformed data had (skewness) g1 = -1.38 and
(kurtosis) g2 = 1.58. The transformed scores had g1
= -.78 and g2 = -.24.
Later this same expert reviewer wrote with respect to another variable "the
average score was low, which to me doesn't suggest 'positive skewness.' In any
event, why not keep scores 'as is' rather than performing a square root
transformation?" The untransformed scores had g1 = .78, g2
= 1.52, the transformed scores had g1 = .24, g2
= .43.
What is one to do when one receives an expert review like this?
Great idea, I'll change it. When will this be published? :o)
Anonymous
Well… that’s when the letter to the editor that is not part of one’s
revisions or replies to the reviewers is important… surely the editor
understands what negative skewness means and the rationale behind
transformations… of course, it’s also the time when it’s probably best to write
the response that one wants to send, then tear that up and write one that sounds
much more circumspect… “perhaps the reviewer was slightly confused… perhaps I
wasn’t clear enough???” certainly a frustrating experience…. k
Kathleen A. Lawler Row, Ph.D.
Let's be charitable and assume that the reviewer just got momentarily
confused about positive and negative skewness. I still get my left foot and
right foot confused during dance lessons.
If a reviewer suggests a minor change in the data analysis, and that's all it
takes to get the paper published, I generally encourage people to give in on
this point. Arguing over a minor point isn't worth it in the long run. Unless
you have a really strong reason to keep the analysis unchanged, re-run the
analysis on the untransformed data. It won't take more than a few minutes and it
virtually guarantees publication.
Getting in an argument with an obvious nit-picker is not a productive use of
your time. This reviewer used rather harsh language, and it's hard not to react
defensively. But I think the wisest course is to save your ammunition for the
truly important battles.
If you do indeed think this is an important battle, then some of the other
respondents offered good advice. But when a reviewer says "I don't like how you
analyzed the data" I try to be grateful that they didn't say "This data is so
awful that no data analysis could salvage it."
Steve Simon, ssimon@cmh.edu
Yes, your colleague should reply directly to the editor of the journal
concerned, pointing out the mistakes made by the reviewer, with references to
emphasize her arguments for things a statistician might consider to be
"obvious". That editor needs to know that the reviewer has been at fault ASAP so
that remedial action can be taken (you never know what other papers have been
affected). Editors may or may not be statistically aware, but they should be
able to follow a well-reasoned argument.
Best wishes,
Malcolm Campbell
Standard response: Reply to the editor of the journal (I presume?) in
question, reviewing the reviewer and rebutting the erroneous implied argument(s).
Indeed, scores "clustered toward the top end of the distribution" would be
negatively skewed, using a common definition of skewness (e.g. [here cite your
favourite stats text]). If I were the author, I'd be strongly tempted to supply
a simple plot of the data, to illustrate that the reviewer's expectation is
correct, even if his/her definitions aren't. A dotplot, perhaps. I might also be
snarky enough to ask that, if the reviewer chooses to abide by non-standard
definitions of statistical terms, he/she be kind enough to supply
definitions...My personal bias tends to favor untransformed data. But I would be
less interested in measures of skewness and/or kurtosis of the data to be
subjected to statistical analysis than in whether analysis on the transformed
data led to conclusions different from those arising from analysis of the
untransformed data. If the conclusions are not different, why bother
transforming? (But in any case, one could summarize the analytical results,
pointing out the differences, if any.)
------------------------------------------------------------
Donald F. Burrill dfb@mv.mv.com
Despair at the level of statistical literacy in some disciplines and
sub-disciplines?
Data transformations have a long history in psychology and other sciences; lots
of references out there. She might want to find an article in this particular
area, if possible. Anyone familiar with the literature would know that for a
negatively skewed distribution, powers greater than 1 will expand the upper end
of the distribution making it more normal. Probably if the researcher had
used x' = x^2.5, she would have come even closer to g1 = 0.
James M. Clark
clark@uwinnipeg.ca
It is exasperating when one receives a review like this one. I have
received more ludicrous complaints. Tell your colleague to correct errors
by the reviewer politely in the return letter to the action editor. In
cases where there is an issue in which reasonable people might disagree
then respond to the reviewer's concern, don't ignore it.
Kenneth M. Steele, Ph.D. steelekm@appstate.edu
If the editor of the journal can't handle an author politely explaining why a given review is incorrect, then "burning bridges" isn't an issue becuase she doesn't really want to publish there anyway (and I say this as a journal editor myself).Thanks to all who have contributed on this topic. My colleague fears that she risks burning bridges if she questions the review, especially when the editor described this reviewer as an "expert."
As you should have (because you don't really want to publish in such a journal). In the meantime, she might first try to engage the editor in a discussion about the matter. (And if he(?) suggests sending her comments back to the original reviewer, she can ask if he(?) would be willing to send the whole exchange to another "expert." (What are the odds of having two such so ill-named?)The last time I was in a similar situation I sent the editor a rather lengthy explanation of why the reviewer was mistaken in arguing that the reported statistically significant results were meaningless because I did not have sufficient power to have a good chance of detecting a medium-sized effect. DUH! The editor had described this reviewer as his "methodological expert." The editor forwarded my comments to the "expert" who wrote back to him "Who the f*^k does this guy think he is, tell him to do as I say or just go away." I just went away and published elsewhere.
Return to Readings for Students in Graduate Statistics
