As I have been teaching a doctoral course this fall I have found many recent articles where the published manipulation check was “do you recall what your instructions were?”

or “did you remember what we told you about your case?”

Really????  That’s all?????

When I first studied the predictive validity framework under Bob Libby in 1989 I remember clearly that manipulation checks, pretests, etc were aimed at providing evidence that the participants in experiments interpreted the manipulations the way that the authors/researchers had meant them to be interpreted.  NOT that the participants could parrot back that they were in an incentive condition where they were paid $2 per correct answer on a test.

What is the difference you ask?  Well if the incentive is about motivating greater performance, why not have a measure of motivation and see if those that received the incentive exhibit greater motivation than those that do not receive the incentive.  There are many very succinct scales that could be employed to quickly gather this information as part of a debriefing instrument at the end of the experiment.

An interpretation of a manipulation “manipulation check” is even more important when the author/researcher is attempting to create a manipulation that informs different participants about different states of the world.  For example, in an experiment I could manipulate whether the CFO had a professional accounting designation or not as part of a multi item manipulation of earnings management expertise within the company.  While it is nice to know that the participant remembered the CFO had a professional accounting designation it provides no evidence whatsoever that I was successful in my manipulation of relative differences in earnings management expertise across treatment conditions.  To do that I would have to ask something like “The firm has strong technical expertise in accounting”  and “the firm management has the ability to implement complex accounting schemes?”  Now I would have to pretest these questions to ensure they were not leading the witness, but they would provide me with some confidence that I was manipulating what I thought I was manipulating in a way that just recalling whether the CFO had a professional accounting designation does not.

Young reviewers also seem to miss this as well.  No wonder some very questionable manipulations are not being caught by reviewers if the reviewer is happy with what I like to call “factoid checking” as manipulation checks.

Surely we are better trained than that!!!!