We seem to do two things to make our papers size up or down.
Size up: We print all possible analyses to rule out alternative explanations. This lengthens papers considerably, especially archival ones. Why not simply report all of these analyses in one table with a hot link to detailed results? The only matters to be discussed about these robustness results are where they did not work along with as clear an explanation as to why. Doing this is only really important when it involves replication of prior results. In many areas only the main results are in the paper!
Size down: time after time I am seeing key descriptive statistics buried in the text of a paper. This is occurring more frequently in experimental papers. It is hard to understand if means are in the right direction, that is, as the alternative hypothesis suggests if you have to struggle to find them.
Neither of these practices does us any good as scholars.
I rarely get besotted (a old English word) with a handbook even though I have contributed to three of them over the years. But I must admit that for positivist behavioral accounting researchers, one should “bit the bullet” and purchase what I consider a somewhat overpriced product (especially if you want it in hard copy format and when the ebook option is more expensive than what a softcover should be).
What am I talking about? Libby and Thorne’s 2017 The Routledge Companion to Behavioural Accounting Research 1st Edition (available at https://www.routledge.com/The-Routledge-Companion-to-Behavioural-Accounting-Research/Libby-Thorne/p/book/9781138890664 if the embedded hotlink does not work). I cannot recall the last time I loaned a book to a PhD student and had to call it back to look up a reference – but I did with this book.
The book contains several chapters of interest including an elaboration on Bill Kinney Jr.’s most famous footnote (Footnote 23 of Kinney Jr. 1986) that talks about how to plan your research through a relatively straightforward mnemonic. As a Doctoral Student Advisor/Supervisor and as an Editor for most of the last decade, it is so obvious to me that many of us teach this but it goes in one ear and out the other for our students!! The quality of behavioral accounting research would improve immensely if BAR researchers followed its precepts.
The book also contains a reprint of Libby’s 1982 Chapter 1 explaining the predictive validity model. The original is so much better than the echo of it that he published in AOS in 2002, not to mention the original does not focus on an example article that has been retracted as likely based on fake data.
This is truly a reference book that should stand the test of time.
PLEASE note this is an uncompensated endorsement but also note that I have a chapter in the book so there is a potential COI. Now that I said that I have moral license to exaggerate but those who know me know that would not matter!
In a day and age of multiple statistical methods being readily available through software packages, and a decline in the number of methods courses being required of PhD students in behavioral accounting, it is little surprise when a reviewer encounters a statistical analysis in a paper where they have substantive expertise. [ Aside: When I was a boy I took two classes in probability and statistical theory, three applied methods courses and that was just the requirements. I took another three methods courses beyond what was the minimum for credit and audited two others. ]
So what does the reviewer do? Do they assume the author(s) has picked an obscure statistical methods for data analysis correctly? OR Do they educate themselves on the method and consider whether it is appropriate for the type of data?
Of course, this assumes that the reviewer is well enough trained in statistical methods to recognize what assumptions are needed to be fulfilled in order to use any given method. While the previous generation of researchers certainly knew how to check for gross violations of underlying statistical assumptions in ANOVA and other linear models, some understood the assumptions underlying SEM and its variants, when it gets to hazard rate models, logit and probit models, and other such models only a handful new. I wonder today how many newly trained behavioral PhD’s could say the same????
The beauty is that if a reviewer has any basic knowledge of model assumptions that need to be fulfilled in general, our good friend “google” can help you find what the assumptions are for models that you many not have heard of. For example, this week I had a paper that employed Poisson log linear regression models in my doctoral class. While I had heard of Poisson distribution and I had heard of log linear regressions, this is the first time I had heard this combination. Yep, after 30 years in the business, a new one.
So what did I do, I went to my trusty “google” and looked for “Assumptions underlying Poisson loglinear regression models.” The first thing that popped up was that the dependent variable is suppose to be count data that was theoretically unbounded at the upper end of the distribution. (“If the data are anything but non-negative integers that are (in principle, at least) unbounded, Poisson regression is the wrong model to use. “) Whoops, the data had only three values 0, 1, 2. Well that suggests further research is necessary by the reviewer and at least a strongly worded review note should result. [Second aside: I will find out whether my doctoral students read this blog when I see whether they bring up the point in class!]
Unfortunately this is a real example of a paper that is in the later rounds of review at a major journal (NOT BRIA of course). It will be interesting to see if the journal review process catches it and whether the data stand up to proper statistical analysis (likely an ordered probit but I could be wrong)!
Today as I prepared to teach a governance class on executive compensation, I came across a Canadian think tank study on the Double Glass Ceiling. https://www.policyalternatives.ca/publications/reports/double-pane-glass-ceiling The double glass refers to the fight women have in breaking into top executive and board ranks and then in executive ranks getting equal pay for equal work!
At the same time I read in the Globe & Mail, the closest thing Canada has to the NY Times or Times of London, about the results of the American Economic Association survey https://www.aeaweb.org/news/press-release-climate-survey-results-march-2019 on discrimination nation among current and former AEA members – predominately academic economists. The response rate was huge 25 % of current members responded. What they reported was a hugely misogynistic culture that devalued contributions of women and racial minority economists.
As I think about the difficulty I run into convincing many tenured Associate Professors that are female to go up for promotion to full prof, I wonder if the set of experiences they encountered might be similar to those in the AEA. Maybe the various accounting associations should find out!
For a long time now economists have been running off to India and similar venues to run economics experiments with real people using real incentives for real work! Those working in the nudging paridigm have similarly done field experiments here in Norh America. Often all it requires is an interested government ministry and they are in business.
Management accounting has seen a few field experiments as well, with my good friend Alan Webb being at the forefront of that movement! Even rarer but happening are field experiments in tax compliance – often done by economists or nudgers!
Recently came across an audit experiment where partner lead interventions were run on real audit teams working on real clients! Dennis and Johnstone 2018 in AOS did just that. While it looks like the review process gave it a long ride before finding a home in AOS, after all the paper won a best of award in 2015, this sort of research is the innovative kind of thing that experimental behavioural researchers need to do. And it once again shows the inability of top tier US journals to accept innovative work that is not “perfect” in its experimental design. Thank goodness for CAR and AOS that are willing to take greater chances with innovative papers. The behavioural accounting world, and accounting research in general would be worse off without them!
It seems you cannot look at a behavioural accounting paper these days without a mediation or moderation model in it. Last summer I noted this was a concern of the Libby panel at the AAA Annual meeting http://aaahq.org/Meetings/2018/Annual-Meeting/Video-Gallery/Panel-Sessions. Session 6.01 on testing process theories.
And if you see such a model being tested 8 or 9 times out of ten it will be based on the Preacher and Haynes PROCESS macro that is available for SPSS and SAS. After reading enough reviews of papers using this approach, the best I can tell is that this is a poor sample size SEM at work! Indeed I recently came across a paper by Haynes and a co-authors (Montoya and Rockwood in the Australasian Marketing Journal 2017) that basically said that! see https://www.sciencedirect.com/science/article/abs/pii/S1441358217300265
Indeed as I read it I am not certain that the use of the bootstrapping option in most SEM packages would not do the same thing! So go ahead and use PROCESS macro when you are using a process model with only observed variables! but do not make claims about it that are not warranted! There is no superiority over SEM applications that I can see despite the mistaken claims of some – some that do NOT include Preacher or Haynes!
In the underlying psychology literature there is some controversy about whether manipulation checks are needed. Mind you from reading that literature it appears that some psychology researchers were not too careful about where they placed the questions that may have had the effect of priming responses that they wanted to get to their manipulations.
What are the alternatives to manipulation checks? One thing for certain is that recall checks of case facts, even manipulations, do not prove manipulations are effective. Ah, but is not remembering a manipulation a necessary condition for it to effective? No, unconscious processing of manipulation is all that is needed in many if not all experiments. After all most experiments are not done with the goal of altering long term memory!
What is important is that we have proof that participants give the same interpretation to the manipulations as the experimenters think they are. So if you do not establish that as part of your instrument with a well placed mc, then you must do pilot tests, verbal protocols, expert panels, or some other means to convince the reader that you and participants see the world in the same way as far as the experiment goes.
So are MC’s history? Not if we want to do valid experiments! In many ways for auditors it is like doing alternative procedure if the normal audit test cannot be done. Most of the time their are workarounds, but almost always involving more work than the original approach. That’s how I see the lack of manipulation checks. The slightly shorter experiment versus the extra work to prove that operationalization was effective.