Replication or Generalizability? How Flexible Inferences Uphold Unfounded Universal Claims

Metascientists commonly distinguish between tests of replicability, reproducibility, robustness, and generalizability. Whereas these distinctions are sensible, they are not often the subject of in-depth definitional exploration. This paper examines the relation between two of these—replication and generalizability—seeking to clarify their differences in the context of sample diversity. Following a brief discussion on the definitions of these terms, I advance four assertions: 1) discussions of replication assume researchers adhere to the Mertonian norm of disinterestedness, 2) researchers are motivated to maintain claims associated with their ontological stance, most commonly universalism, 3) despite universal claims, failures of replication are perceived to have greater epistemic consequences than lack of generalizability, and 4) researchers are motivated to reframe threats to replicability as limits of generalizability. I then use behavioral genetics research on polygenic score prediction of educational attainment as a case study to illustrate the assertions. Finally, I close with some brief recommendations for how to move forward.