I recently had a case at BRIA where in the decision letter I stated “The key for successful resolution of the issues identified by the reviewers is proof that the results are not statistical method of analysis dependent.”  The decision letter went on to suggest a variety of tests that could be done to rule out the issue of method dependency.  How did the authors reply?  A two page ramble in the text of the paper about why their method was most appropriate.

What inference can I draw as an editor from such a reply?  That the results are so substantially weaker if other valid methods are employed that the authors did not want to disclose them.  If the results are so method dependent then are they really results that we want to report in our literature?  As Bob Libby used to say about experiments, if you cannot see the difference you are reporting by examining cell means, maybe what you found is a dust mite not a mountain (okay not exactly what Bob said but close enough to get the point).