LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows
Financial institutions deploy Large Language Models (LLMs) for reconciliations, regulatory reporting, and client communications, but nondeterministic outputs (output drift) undermine auditability and trust. We quantify drift across five model architectures (7B-120B parameters) on regulated financial...
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Articolo |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2511.07585 http://arxiv.org/abs/2511.07585 |
| Tags: |
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1879177896364343296 |
|---|---|
| author | Khatchadourian, Raffi Franco, Rolando |
| author_browse | Franco, Rolando Khatchadourian, Raffi |
| author_facet | Khatchadourian, Raffi Franco, Rolando |
| author_sort | Khatchadourian, Raffi |
| building | arXiv |
| collection | Acceso abierto |
| description | Financial institutions deploy Large Language Models (LLMs) for reconciliations, regulatory reporting, and client communications, but nondeterministic outputs (output drift) undermine auditability and trust. We quantify drift across five model architectures (7B-120B parameters) on regulated financial tasks, revealing a stark inverse relationship: smaller models (Granite-3-8B, Qwen2.5-7B) achieve 100% output consistency at T=0.0, while GPT-OSS-120B exhibits only 12.5% consistency (95% CI: 3.5-36.0%) regardless of configuration (p<0.0001, Fisher's exact test). This finding challenges conventional assumptions that larger models are universally superior for production deployment.
Our contributions include: (i) a finance-calibrated deterministic test harness combining greedy decoding (T=0.0), fixed seeds, and SEC 10-K structure-aware retrieval ordering; (ii) task-specific invariant checking for RAG, JSON, and SQL outputs using finance-calibrated materiality thresholds (plus or minus 5%) and SEC citation validation; (iii) a three-tier model classification system enabling risk-appropriate deployment decisions; and (iv) an audit-ready attestation system with dual-provider validation.
We evaluated five models (Qwen2.5-7B via Ollama, Granite-3-8B via IBM watsonx.ai, Llama-3.3-70B, Mistral-Medium-2505, and GPT-OSS-120B) across three regulated financial tasks. Across 480 runs (n=16 per condition), structured tasks (SQL) remain stable even at T=0.2, while RAG tasks show drift (25-75%), revealing task-dependent sensitivity. Cross-provider validation confirms deterministic behavior transfers between local and cloud deployments. We map our framework to Financial Stability Board (FSB), Bank for International Settlements (BIS), and Commodity Futures Trading Commission (CFTC) requirements, demonstrating practical pathways for compliance-ready AI deployments. |
| first_indexed | 2026-10-16T01:08:13Z |
| format | Article |
| fullrecord | <oai_dc:dc xmlns:oai_dc="http://www.openarchives.org/OAI/2.0/oai_dc/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ https://www.openarchives.org/OAI/2.0/oai_dc.xsd"><identifier>arxiv-2511.07585</identifier><datestamp>2025-11-12</datestamp>
<dc:title>LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows</dc:title>
<dc:creator>Khatchadourian, Raffi</dc:creator>
<dc:creator>Franco, Rolando</dc:creator>
<dc:subject>Machine Learning</dc:subject>
<dc:subject>Artificial Intelligence</dc:subject>
<dc:subject>Computation and Language</dc:subject>
<dc:subject>I.2.7; K.4.1; H.3.3</dc:subject>
<dc:description>Financial institutions deploy Large Language Models (LLMs) for reconciliations, regulatory reporting, and client communications, but nondeterministic outputs (output drift) undermine auditability and trust. We quantify drift across five model architectures (7B-120B parameters) on regulated financial tasks, revealing a stark inverse relationship: smaller models (Granite-3-8B, Qwen2.5-7B) achieve 100% output consistency at T=0.0, while GPT-OSS-120B exhibits only 12.5% consistency (95% CI: 3.5-36.0%) regardless of configuration (p<0.0001, Fisher's exact test). This finding challenges conventional assumptions that larger models are universally superior for production deployment.
Our contributions include: (i) a finance-calibrated deterministic test harness combining greedy decoding (T=0.0), fixed seeds, and SEC 10-K structure-aware retrieval ordering; (ii) task-specific invariant checking for RAG, JSON, and SQL outputs using finance-calibrated materiality thresholds (plus or minus 5%) and SEC citation validation; (iii) a three-tier model classification system enabling risk-appropriate deployment decisions; and (iv) an audit-ready attestation system with dual-provider validation.
We evaluated five models (Qwen2.5-7B via Ollama, Granite-3-8B via IBM watsonx.ai, Llama-3.3-70B, Mistral-Medium-2505, and GPT-OSS-120B) across three regulated financial tasks. Across 480 runs (n=16 per condition), structured tasks (SQL) remain stable even at T=0.2, while RAG tasks show drift (25-75%), revealing task-dependent sensitivity. Cross-provider validation confirms deterministic behavior transfers between local and cloud deployments. We map our framework to Financial Stability Board (FSB), Bank for International Settlements (BIS), and Commodity Futures Trading Commission (CFTC) requirements, demonstrating practical pathways for compliance-ready AI deployments. </dc:description>
<dc:description>11 pages, 5 figures. To appear in AI4F @ ACM ICAIF '25, November 15-18, 2025, Singapore</dc:description>
<dc:date>2025-11-10</dc:date>
<dc:type>text</dc:type>
<dc:identifier>https://arxiv.org/abs/2511.07585</dc:identifier>
<dc:identifier>http://arxiv.org/abs/2511.07585</dc:identifier>
</oai_dc:dc> |
| id | arxiv-2511.07585 |
| institution | arXiv |
| last_indexed | 2026-10-16T01:08:13Z |
| publishDate | 2025 |
| publishDateRange | 2025 |
| publishDateSort | 2025 |
| record_format | oai_dc |
| spelling | arxiv-2511.075852025-11-12 LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows Khatchadourian, Raffi Franco, Rolando Machine Learning Artificial Intelligence Computation and Language I.2.7; K.4.1; H.3.3 Financial institutions deploy Large Language Models (LLMs) for reconciliations, regulatory reporting, and client communications, but nondeterministic outputs (output drift) undermine auditability and trust. We quantify drift across five model architectures (7B-120B parameters) on regulated financial tasks, revealing a stark inverse relationship: smaller models (Granite-3-8B, Qwen2.5-7B) achieve 100% output consistency at T=0.0, while GPT-OSS-120B exhibits only 12.5% consistency (95% CI: 3.5-36.0%) regardless of configuration (p<0.0001, Fisher's exact test). This finding challenges conventional assumptions that larger models are universally superior for production deployment. Our contributions include: (i) a finance-calibrated deterministic test harness combining greedy decoding (T=0.0), fixed seeds, and SEC 10-K structure-aware retrieval ordering; (ii) task-specific invariant checking for RAG, JSON, and SQL outputs using finance-calibrated materiality thresholds (plus or minus 5%) and SEC citation validation; (iii) a three-tier model classification system enabling risk-appropriate deployment decisions; and (iv) an audit-ready attestation system with dual-provider validation. We evaluated five models (Qwen2.5-7B via Ollama, Granite-3-8B via IBM watsonx.ai, Llama-3.3-70B, Mistral-Medium-2505, and GPT-OSS-120B) across three regulated financial tasks. Across 480 runs (n=16 per condition), structured tasks (SQL) remain stable even at T=0.2, while RAG tasks show drift (25-75%), revealing task-dependent sensitivity. Cross-provider validation confirms deterministic behavior transfers between local and cloud deployments. We map our framework to Financial Stability Board (FSB), Bank for International Settlements (BIS), and Commodity Futures Trading Commission (CFTC) requirements, demonstrating practical pathways for compliance-ready AI deployments. 11 pages, 5 figures. To appear in AI4F @ ACM ICAIF '25, November 15-18, 2025, Singapore 2025-11-10 text https://arxiv.org/abs/2511.07585 http://arxiv.org/abs/2511.07585 |
| spellingShingle | Machine Learning Artificial Intelligence Computation and Language I.2.7; K.4.1; H.3.3 Khatchadourian, Raffi Franco, Rolando LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows |
| title | LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows |
| title_full | LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows |
| title_fullStr | LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows |
| title_full_unstemmed | LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows |
| title_short | LLM Output Drift: Cross-Provider Validation & Mitigation for Financial Workflows |
| title_sort | llm output drift cross provider validation mitigation for financial workflows |
| topic | Machine Learning Artificial Intelligence Computation and Language I.2.7; K.4.1; H.3.3 |
| url | https://arxiv.org/abs/2511.07585 http://arxiv.org/abs/2511.07585 |
| work_keys_str_mv | AT khatchadourianraffi llmoutputdriftcrossprovidervalidationmitigationforfinancialworkflows AT francorolando llmoutputdriftcrossprovidervalidationmitigationforfinancialworkflows |