AIScreening: A Structured-Output Pipeline and Prompt-Policy Benchmark for LLM-Assisted Title and Abstract Screening

Title and abstract screening is a major bottleneck in systematic review and meta-analytic workflows. Although large language models may support early-stage screening, false negatives are costly because they can remove potentially eligible studies before full-text review. This study introduces AIScreening, a local structured-output pipeline for AI-assisted literature screening, and evaluates how prompt design affects LLM screening decisions.Using 207 title/abstract records from an educational expectations/aspirations meta-analysis, we compared six prompt conditions: zero-shot, inclusion-only, exclusion-only, full-criteria, few-shot, and criteria-decomposition prompting. All conditions used the same model, input fields, output schema, and parsing pipeline. Model outputs were evaluated against corrected human reference labels using recall, precision, F1 score, Cohen’s kappa, Over-Inclusion Rate, and exploratory error-structure analysis.Zero-shot prompting achieved perfect recall but produced substantial over-inclusion of human-excluded records. Full-criteria prompting showed the best overall performance, producing the highest F1 score, Cohen’s kappa, and precision while maintaining low over-inclusion. Few-shot and criteria-decomposition prompting did not improve performance beyond full criteria and increased false negatives. These results suggest that prompt design controls the trade-off between semantic overgeneralization and eligibility constraint, rather than improving screening performance linearly with added information. AIScreening provides a reusable framework for prompt-policy benchmarking in AI-assisted systematic review screening.