C3E: A Multi-Round, Controllable Chain-of-Thought Critic-Editor Framework for Factual and User-Guided Summaries

While large language models (LLMs) have made significant strides in text summarization, they often produce summaries that exhibit factual inconsistency, and researchers have found it difficult to make the output controllable to users. This paper presents a new framework called Controllable Chain-of-Thought Critic-Editor (C3E), which uses a controllable, post-editing process involving a Controller, Critic (which performs two evaluations), and Editor that operate with zero-shot Chain-of-Thought reasoning, in a multi-round, post-editing process, to solve these issues. The Controller will make summaries more aligned with user instruction, the Critic will evaluate the factual consistency and adherence to user control, and the Editor will refine the summaries iteratively until it satisfies the user over multiple rounds. We evaluate C3E by further developing the FRANK dataset with user control instruction as well as introducing a new score called Control Adherence Score. Experiments with advanced LLMs showed that C3E was higher in factual accuracy and user compliance than revisions made without a C3E framework without loss of content quality. Human evaluation and ablation studies verified that performing an iterated multi-round revision process and reasoning in a Chain-of-Thought format led to accurate and controllable summaries.