The Reliability and Validity of Problem Generation Tests: A Meta-Analysis with Implications for Problem Finding and Creativity

Problem finding (PF) is a very important part of the creative process. The problem Generation (PG) test was developed nearly 40 years ago in an attempt to assess the potential for PF using a series of paper-and-pencil tasks. This meta-analysis examined 19 previous empirical investigations of the PG test to examine internal reliability (k = 43, N = 2,029), convergent (k = 125, N = 2,573), and discriminant validity (k = 26, N = 2,145) evidence. A search identified 617 candidate studies. Analyses indicated that the overall random-effects weighted mean reliability was 𝛼 = .816, t(42) = 26.419, p < .001 (95% CI: 0.786, 0.842), indicating good internal consistency. A second analysis produced a random-effects weighted mean validity coefficient r, was .463, t(124) = 13.994, p < .001 (95% CI: 0.406, 0.518), indicating moderate agreement between PG test and other creativity measures such as divergent thinking and creative achievement. The mean correlation among the scoring indices of the PG test (i.e., fluency, flexibility, and originality) was .590, t(25) = 4.714, p < .001 (95% CI: 0.364, 0.750) which indicates a moderate level of discriminant validity of the scoring indices. Data were heterogeneous in all three analyses, but moderator analyses showed that covariates did not explain the variation. This is the first study that examined the reliability and validity of the PG test, which showed that the test is, in several ways, a valid and reliable tool for assessing PF ability.