Home » 2013 (Page 2)
Yearly Archives: 2013
As I have been teaching a doctoral course this fall I have found many recent articles where the published manipulation check was “do you recall what your instructions were?”
or “did you remember what we told you about your case?”
Really???? That’s all?????
When I first studied the predictive validity framework under Bob Libby in 1989 I remember clearly that manipulation checks, pretests, etc were aimed at providing evidence that the participants in experiments interpreted the manipulations the way that the authors/researchers had meant them to be interpreted. NOT that the participants could parrot back that they were in an incentive condition where they were paid $2 per correct answer on a test.
What is the difference you ask? Well if the incentive is about motivating greater performance, why not have a measure of motivation and see if those that received the incentive exhibit greater motivation than those that do not receive the incentive. There are many very succinct scales that could be employed to quickly gather this information as part of a debriefing instrument at the end of the experiment.
An interpretation of a manipulation “manipulation check” is even more important when the author/researcher is attempting to create a manipulation that informs different participants about different states of the world. For example, in an experiment I could manipulate whether the CFO had a professional accounting designation or not as part of a multi item manipulation of earnings management expertise within the company. While it is nice to know that the participant remembered the CFO had a professional accounting designation it provides no evidence whatsoever that I was successful in my manipulation of relative differences in earnings management expertise across treatment conditions. To do that I would have to ask something like “The firm has strong technical expertise in accounting” and “the firm management has the ability to implement complex accounting schemes?” Now I would have to pretest these questions to ensure they were not leading the witness, but they would provide me with some confidence that I was manipulating what I thought I was manipulating in a way that just recalling whether the CFO had a professional accounting designation does not.
Young reviewers also seem to miss this as well. No wonder some very questionable manipulations are not being caught by reviewers if the reviewer is happy with what I like to call “factoid checking” as manipulation checks.
Surely we are better trained than that!!!!
Journals in other disciplines are beginning to require more disclosure about back of the house activities in the research process. Some economics journals require the final data set be posted. Some psychology journals are requiring disclosure of any manipulations that were omitted in the final data analysis or how many times different variants of the same experiment were run before a significant result is found!!!
I find the statement that the archival data used in our research is from publicly available sources and that is the end of the disclosure about data availability to be one of the most disingenuous statement very made. Especially by a group of scholars who are often studying the transparency of others disclosures.
It is especially worrisome as one sees paper after paper after paper eliminating large portions of the total population due to missing data, outliers, windsoring, inability to match observations across data sets etc. lets make it a required disclosure that the first data set used be disclosed so that others can replicate the exclusions and see if they are warranted.
I will elaborate on this theme in my next post or two with a concrete example from our literature.
Every now and gain I come across strange cases where one reviewer recommends acceptance of a paper and the other reviewer keeps pushing for changes. The editor instead of drawing the matter to a close continues to let the process run on and on! What happens??? The paper in the end never satisfies the reluctant reviewer and gets to the point where the reviewer who recommended acceptance either changes his/ her mind or withdraws from the process talking the favourable voice out of the process.
You know what happens in this story don’t you???? Yep, the paper gets rejected!!! Strike another blow for strong editing!