An integrated procedure to control for common method variance using random intercept factor analysis models

Response styles and response biases can introduce systematic variance in measurement models. Their aggregated effect can be captured by introducing into the model an additional general 'method' factor, a random intercept that varies across items. To support the validity of interpreting this additional factor as method variance rather than a substantive general factor, we formulate testable queries, including an efficiency hypothesis that outlines conditions where method effects are more likely. The strongest validity evidence involves integrating into the survey and model a variable that, while theoretically uncorrelated with the assessed attributes, elicits similar response mechanisms. If this marker variable remains uncorrelated with this additional factor, it supports its interpretation as substantive, rather than method. When validity evidence supports the interpretation of the additional factor as method variance, we describe methods to quantify the extent of method bias on reliability and validity coefficients. In cases where method effects are substantial, our proposed approach ensures valid inferences and scores free from method bias. Fully worked out examples and code illustrate the implementation of these methods. In a neutral scenario according to our efficiency hypothesis, no indication of method effects is observed. However, when method effects could be expected, we found non-negligible method variance (up to 17%), which we subsequently controlled for. Failure to account for method variance when present leads to poorly fitting models and over- or underestimation of validity and reliability coefficients.