As an editor I am somewhat worried by the trend to selective reporting of results in academic papers. No I do not mean papers that report analyses in footnotes or supplementary form, but papers that only refer to “strong” results. How is this done?

1. Having two or three potential dependent variables and reporting only one. Of course only the reviewer can tell there are two or three as the paper does not disclose it. That assumes the reviewer gets the full instrument to review.

2. Substitute a measured variable for one of the manipulated variables but does not consistently do so across all manipulated variables. I have no problem with mixed designs, however if you are going to manipulate and measure, disclose both so the reader can judge. I frequently plan ex ante studies that manipulate and measure independent variables. But when I measure I use multiple items that represent the independent variable, report confirmatory and exploratory factor analyses, do not mix measured and manipulated measures (unless reviewers and editors make me and then I also disclose the results if done consistently).

3. Reducing sample without given clear rationales as to why and reporting results ONLY on reduced sample. Report what happens if you include everyone. Where there is judgment involved with some sample reductions, report with and without. ( e.g. a task takes ten minutes to read yet the respondent not only read the task but answered the questions in four minutes, exclude for certain. But what about the person that did it in 9 minutes?)

4. Selectively using high powered statistical methods and only reporting the ones that work. If a result is method dependent the reader should know.

For archival researchers an similar set of problems can be found. Years ago I heard Pat O’Brien consistently ask the question in workshop after workshop, what happens to your results if you do not take out 2% of the extreme observations ( the practice of winsorizing top 1% and bottom 1% of samples)? No analysis of whether these are truly outliers, just delete. At least footnote report the full sample.