Musings on Accounting Research by Steve

Home » Current thoughts » Full disclosure – a note for empirical researchers

Full disclosure – a note for empirical researchers

Topics

As an editor I am somewhat worried by the trend to selective reporting of results in academic papers. No I do not mean papers that report analyses in footnotes or supplementary form, but papers that only refer to “strong” results. How is this done?

1. Having two or three potential dependent variables and reporting only one. Of course only the reviewer can tell there are two or three as the paper does not disclose it. That assumes the reviewer gets the full instrument to review.

2. Substitute a measured variable for one of the manipulated variables but does not consistently do so across all manipulated variables. I have no problem with mixed designs, however if you are going to manipulate and measure, disclose both so the reader can judge. I frequently plan ex ante studies that manipulate and measure independent variables. But when I measure I use multiple items that represent the independent variable, report confirmatory and exploratory factor analyses, do not mix measured and manipulated measures (unless reviewers and editors make me and then I also disclose the results if done consistently).

3. Reducing sample without given clear rationales as to why and reporting results ONLY on reduced sample. Report what happens if you include everyone. Where there is judgment involved with some sample reductions, report with and without. ( e.g. a task takes ten minutes to read yet the respondent not only read the task but answered the questions in four minutes, exclude for certain. But what about the person that did it in 9 minutes?)

4. Selectively using high powered statistical methods and only reporting the ones that work. If a result is method dependent the reader should know.

For archival researchers an similar set of problems can be found. Years ago I heard Pat O’Brien consistently ask the question in workshop after workshop, what happens to your results if you do not take out 2% of the extreme observations ( the practice of winsorizing top 1% and bottom 1% of samples)? No analysis of whether these are truly outliers, just delete. At least footnote report the full sample.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: