Mapping Creativity in The PISA 2022 Context Questionnaire: An Expert Evaluation of Content Validity of the Creativity Self- and Other-Report Scales

The Programme for International Student Assessment (PISA) is one of the most costly, influential, and wide-reaching research efforts in educational psychology. But are the measures they administer as valid as they should be? As part of its 2022 cycle, PISA included a focus on creative thinking, incorporating questionnaires designed to measure various creativity-related constructs. However, information on the development of these items is limited, and the extent to which these items validly capture their intended constructs has not been systematically evaluated. If these measures lack validity, the field risks interpreting secondary analyses of PISA data that may not be fully psychologically meaningful, leading possibly to incorrect recommendations for educational practice. This problem is addressed by employing a multi-stage expert judgment procedure (involving a total of N = 11 experts) to assess the content validity of 128 creativity-related items selected from an initial pool of 936 items from PISA 2022. After an extensive pilot-phase (Study 1), experts categorized items according to predefined construct and domain definitions, evaluated item quality, and rated construct coverage (Study 2). Findings revealed substantial misalignment between the constructs assigned by the OECD and those identified by the expert panel (average of 37% overlap). The results raise concerns about the interpretability and coherence of existing PISA creativity indices and underscore the need for improved scale specification. This study ultimately offers a detailed item-content map as a ground for revised measurement of creativity-relevant constructs, which are intended to support future analyses grounded in psychologically and educationally valid quantities.