LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis
Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort. Recent large language model (LLM) systems have demonstrated strong performance on individual SR phases - screening (otto-SR: 96....
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Artículo |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2606.28362 http://arxiv.org/abs/2606.28362 |
| Etiquetas: |
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1879179547177385984 |
|---|---|
| author | Huang, Yen-Hsun Lin, Yu-Shiou |
| author_browse | Huang, Yen-Hsun Lin, Yu-Shiou |
| author_facet | Huang, Yen-Hsun Lin, Yu-Shiou |
| author_sort | Huang, Yen-Hsun |
| building | arXiv |
| collection | Acceso abierto |
| description | Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort. Recent large language model (LLM) systems have demonstrated strong performance on individual SR phases - screening (otto-SR: 96.7% sensitivity), extraction (Gartlehner et al.: 91.0% accuracy), and search (TrialMind: 0.83 recall) - but no study has reported what it actually costs to run an end-to-end pipeline, how cost distributes across phases, or how architectural choices affect the cost-quality trade-off. We present LUMEN, an open-source multi-agent pipeline that automates six SR/MA phases using 11 specialized LLM agents with deliberate model routing. We evaluate LUMEN on seven datasets: five self-conducted domain reviews (psychiatry, psychology, surgery, vaccinology, cardiology) and two SYNERGY screening benchmarks. Across 13 ground-truth-comparable outcomes, LUMEN achieves 100% directional agreement with published meta-analyses, with effect sizes within 1% for homogeneous study designs. The primary contribution is the first empirical cost and operational characterization of such a pipeline: a complete review costs 19 to 29 USD (median 22.65 USD), with title-abstract screening and data extraction together dominating expenditure. A three-arm extraction ablation reveals a phase-dependent architecture reversal: multi-agent design hurts screening but is essential for extraction, producing 5.7x more poolable analyses than single-model alternatives while eliminating clinically dangerous direction errors. A two-dataset screening benchmark demonstrates that model ranking is domain-dependent and not transferable across review topics. All code and cost logs are publicly available. |
| first_indexed | 2026-10-16T01:34:27Z |
| format | Article |
| fullrecord | <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ https://www.openarchives.org/OAI/2.0/oai_dc.xsd"><identifier>arxiv-2606.28362</identifier><datestamp>2026-06-30</datestamp>
<dc:title>LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis</dc:title>
<dc:creator>Huang, Yen-Hsun</dc:creator>
<dc:creator>Lin, Yu-Shiou</dc:creator>
<dc:subject>Information Retrieval</dc:subject>
<dc:subject>Artificial Intelligence</dc:subject>
<dc:subject>Computation and Language</dc:subject>
<dc:subject>Computers and Society</dc:subject>
<dc:subject>Digital Libraries</dc:subject>
<dc:description>Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort. Recent large language model (LLM) systems have demonstrated strong performance on individual SR phases - screening (otto-SR: 96.7% sensitivity), extraction (Gartlehner et al.: 91.0% accuracy), and search (TrialMind: 0.83 recall) - but no study has reported what it actually costs to run an end-to-end pipeline, how cost distributes across phases, or how architectural choices affect the cost-quality trade-off. We present LUMEN, an open-source multi-agent pipeline that automates six SR/MA phases using 11 specialized LLM agents with deliberate model routing. We evaluate LUMEN on seven datasets: five self-conducted domain reviews (psychiatry, psychology, surgery, vaccinology, cardiology) and two SYNERGY screening benchmarks. Across 13 ground-truth-comparable outcomes, LUMEN achieves 100% directional agreement with published meta-analyses, with effect sizes within 1% for homogeneous study designs. The primary contribution is the first empirical cost and operational characterization of such a pipeline: a complete review costs 19 to 29 USD (median 22.65 USD), with title-abstract screening and data extraction together dominating expenditure. A three-arm extraction ablation reveals a phase-dependent architecture reversal: multi-agent design hurts screening but is essential for extraction, producing 5.7x more poolable analyses than single-model alternatives while eliminating clinically dangerous direction errors. A two-dataset screening benchmark demonstrates that model ranking is domain-dependent and not transferable across review topics. All code and cost logs are publicly available. </dc:description>
<dc:description>15 pages, 5 figures. Open-source implementation and cost logs available</dc:description>
<dc:date>2026-03-29</dc:date>
<dc:type>text</dc:type>
<dc:identifier>https://arxiv.org/abs/2606.28362</dc:identifier>
<dc:identifier>http://arxiv.org/abs/2606.28362</dc:identifier>
</oai_dc:dc> |
| id | arxiv-2606.28362 |
| institution | arXiv |
| last_indexed | 2026-10-16T01:34:27Z |
| publishDate | 2026 |
| publishDateRange | 2026 |
| publishDateSort | 2026 |
| record_format | oai_dc |
| spelling | arxiv-2606.283622026-06-30 LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis Huang, Yen-Hsun Lin, Yu-Shiou Information Retrieval Artificial Intelligence Computation and Language Computers and Society Digital Libraries Systematic reviews and meta-analyses (SR/MA) remain the gold standard for evidence synthesis, yet completing one typically requires 67 weeks and substantial expert effort. Recent large language model (LLM) systems have demonstrated strong performance on individual SR phases - screening (otto-SR: 96.7% sensitivity), extraction (Gartlehner et al.: 91.0% accuracy), and search (TrialMind: 0.83 recall) - but no study has reported what it actually costs to run an end-to-end pipeline, how cost distributes across phases, or how architectural choices affect the cost-quality trade-off. We present LUMEN, an open-source multi-agent pipeline that automates six SR/MA phases using 11 specialized LLM agents with deliberate model routing. We evaluate LUMEN on seven datasets: five self-conducted domain reviews (psychiatry, psychology, surgery, vaccinology, cardiology) and two SYNERGY screening benchmarks. Across 13 ground-truth-comparable outcomes, LUMEN achieves 100% directional agreement with published meta-analyses, with effect sizes within 1% for homogeneous study designs. The primary contribution is the first empirical cost and operational characterization of such a pipeline: a complete review costs 19 to 29 USD (median 22.65 USD), with title-abstract screening and data extraction together dominating expenditure. A three-arm extraction ablation reveals a phase-dependent architecture reversal: multi-agent design hurts screening but is essential for extraction, producing 5.7x more poolable analyses than single-model alternatives while eliminating clinically dangerous direction errors. A two-dataset screening benchmark demonstrates that model ranking is domain-dependent and not transferable across review topics. All code and cost logs are publicly available. 15 pages, 5 figures. Open-source implementation and cost logs available 2026-03-29 text https://arxiv.org/abs/2606.28362 http://arxiv.org/abs/2606.28362 |
| spellingShingle | Information Retrieval Artificial Intelligence Computation and Language Computers and Society Digital Libraries Huang, Yen-Hsun Lin, Yu-Shiou LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis |
| title | LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis |
| title_full | LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis |
| title_fullStr | LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis |
| title_full_unstemmed | LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis |
| title_short | LUMEN: Cost-Transparent Multi-Agent Pipeline for Automated Systematic Review and Meta-Analysis |
| title_sort | lumen cost transparent multi agent pipeline for automated systematic review and meta analysis |
| topic | Information Retrieval Artificial Intelligence Computation and Language Computers and Society Digital Libraries |
| url | https://arxiv.org/abs/2606.28362 http://arxiv.org/abs/2606.28362 |
| work_keys_str_mv | AT huangyenhsun lumencosttransparentmultiagentpipelineforautomatedsystematicreviewandmetaanalysis AT linyushiou lumencosttransparentmultiagentpipelineforautomatedsystematicreviewandmetaanalysis |