Chunking and Retrieval Strategies for Reducing Hallucination in Domain-Specific Retrieval-Augmented Generation Pipelines

David Taiwo Balogun Mr, N/A, Independent Researcher, Lagos, Lagos, Nigeria, Balogundavid98@gmail.com
Olajumoke Ore Akinremi Ms, N/A, Independent Researcher, Norfolk County, Norfolk County, United Kingdom, Akinremiolajumoke77@gmail.com
Kosisochukwu Ashara Ms, Department of Artificial Intelligence, University of Bradford, London, London, United Kingdom, Asharakosi@gmail.com
Chinenyengozi Okoronkwo Ms, Department of Computer Science, Birmingham City University, Birmingham, Birmingham, United Kingdom, Chinex2004@yahoo.com

While Retrieval-Augmented Generation (RAG) models have been an excellent way to integrate external data into language models, even RAG models struggle with hallucination, which is even more evident in finance-related topics that are high-risk and high-reward. The objective of this research is to discover how, in a finance-related document question and answer system, chunking style, and how data is pulled, affects the accuracy of the answer and the frequency of hallucinations. Three open-source components of a RAG system: a language processing module, a vector database, and a database of public finance documents, are used in this research. We implement the plain retrieval (cosine-similarity) and a reranking (cross-encoder) retrieval methods, and test the following chunking methods: fixed-size, recursive or hierarchical, and semantic chunking. To measure hallucinations, we created a metric which determines if claims are traceable to the retrieved context, and we also employed standard retrieval measurements and answer accuracy measurements. Semantic chunking with reranking was the most effective at lowering the hallucination rate. Recursive chunking was shown to have a good tradeoff between cost and accuracy. We hope to guide researchers and industry professionals working with domain-specific RAG models. We also outline future explorations related to retrieval-confidence-aware generation and chunk sizing that could be more variable.

CCS Concepts: • Information systems → Information retrieval; • Computing methodologies → Natural language generation; Machine learning;

Additional Keywords and Phrases: Retrieval-Augmented Generation, Large Language Models, Hallucination Mitigation, Chunking Strategies, Information Retrieval, Reranking, Domain-Specific Question Answering, Financial NLP

ACM Reference Format:
David Taiwo Balogun, Olajumoke Ore Akinremi, Kosisochukwu Ashara and Chinenyengozi Okoronkwo. 2026. Chunking and Retrieval Strategies for Reducing Hallucination in Domain-Specific Retrieval-Augmented Generation Pipelines. In 2026 10th International Conference on Advances in Artificial Intelligence (ICAAI 2026), September 21-24, 2026, London, United Kingdom. ACM, New York, NY, USA, 9 Pages. https://doi.org/10.1145/3859174.3859178

1 Introduction

Generative capabilities in open-domain text for large language models (LLMs) are extensive. However, such capabilities are diminished by their dangerous hallucinating tendencies, where these models produce exactly what was requested while lacking empirically-backed treatment. Retrieval-Augmented Generation (RAG) attempts to resolve such issues by basing generation on a collection of documents fetched from an external corpus to the inference instance. While RAG provides a notable improvement, the utility of this added grounding is dependent on a multitude of decisions in the preceding steps, including knowledge of how documents are segmented, and how relevant documents are scoped and scored.

As described, domain-specific LLM applications, such as in finance, law, and healthcare, are more dangerous due to the costly consequences of hallucinations on, for instance, generated financial summaries or on located contractual clauses. However, due to the nature of the source in some of these applications, such as in RAG-transcribed annual reports or in RAG-transcribed transcripts of earnings calls, basic segmentation may damage retrieval, or may damage the model's ability to support its conclusions. This study examines the investigated chunking technique, retrieval configuration, and hallucination within a focused RAG system on financial documents. We make the following contributions: (1) we implement and experimentally compare three prominent chunking strategies; (2) we study the influence of introducing a cross-encoder reranking after dense retrieval; (3) we propose and empirically support a claim-level hallucination rate measure using an LLM-as-judge framework; and (4) we suggest actionable insights to support the construction of RAG pipelines in high-stakes, document-intensive settings.

The rest of the paper is structured in the following way. In Section 2, we summarize previous studies on RAG frameworks, chunking techniques, and hallucination metrics. In Section 3, we detail our framework and the experimental design. Section 4 showcases the findings related to chunking and retrieval. In Section 5, we examine the findings and constraints, and in Section 6 we outline potential avenues for forthcoming research.

2 Related Work

2.1 Retrieval-Augmented Generation

RAG architectures meld a parametric language model with a retrieval system that is non-parametric. This allows the model to tailor its output based on the documents that are retrieved from an external knowledge source during an inference operation. This architecture improves the factual correctness of an answer and decreases the model's reliance on the knowledge that is internally stored within the model's parameters. This architecture also allows knowledge to be updated without the need to retrain the model. Refinements to the basic retrieve-then-generate architecture have included techniques such as iterative retrieval, query refinement, and retrieval-aware model-generation fine-tuning.

2.2 Chunking Strategies

The partition of source documents into retrievable units has an impact on precision of retrievals and the logical context provided to the generator. The simplicity of fixed-size chunking, which divides documents into equal size windows of tokens or characters, comes at the consequence of breaking documents at logically arbitrary boundaries that may divide sentences or sections. Recursive or hierarchical chunking attempts to retain the structure of documents by first splitting along natural boundaries (i.e., paragraphs, sections) and then applying fixed-size chunking to the remainder. There are also attempts to retain document coherence by splitting based on the embedding similarity of sentences, thus allowing semantically similar sentences to remain in a single chunk. While some of these techniques have been applied to long, domain-specific documents, their use has been limited.

2.3 Hallucination Detection and Mitigation

Hallucination in large language models (LLMs) can be studied at generation time, or can be studied post hoc as a property of LLMs. As a grounding strategy, retrieval grounding mitigates hallucination in LLMs, however grounding methods also cannot ensure the model remains faithful to the context. There are several ways to evaluate LLM responses, including but not limited to, human scoring, natural language inference based-faithfulness scoring, and LLM-as-judge scoring, which has been proposed recently. We also analyze LLM-as-judge based approaches and incorporate them in our claim-level hallucination metric.

2.4 Reranking in Retrieval Pipelines

The computational efficiency of bi-encoder methods for retrieval poses a tradeoff. Their performance on fine-grained relevance tasks can be lacking. Jointly encoding a query and candidate passage is the basis of Cross-encoder methods. Inference cost is larger with cross-encoders, but the benefits with respect to ranking are non-negligible. Cross-encoders are effective in the second stage when a set of candidates is cost-inefficiently retrieved via a first stage method.

3 System Design and Methodology

3.1 Corpus and Task

The domain-specific corpus we have constructed augments publicly available financial filings and transcripts of earnings calls. We have developed a question-answering evaluation set from this corpus that consists of questions with answers that are grounded to particular and verifiable passages within the source documents. The evaluation set consists of the cited passage, the question, and the corresponding answer, which is known to be correct.

3.2 Pipeline Architecture

Our RAG pipeline is composed of four steps: (1) ingesting and decomposing documents, (2) embedding and indexing to a vector database, (3) retrieval with an option to rerank, and (4) context based answer generation with an embedded evaluation of hallucination. The LLM generator is open source and instruction-tuned. The retrieval and embedding components as well as the vector store are unchanged across all experimental variations. Other system alterations are eliminated as a variable. The offline indexing and online querying are differentiated in the architectural overview shown in Figure 1.

Figure 1
Figure 1: System architecture showing the offline indexing path (corpus, chunking, embedding, vector store) and the online query path (retrieve, optional rerank, generate, evaluate).

3.3 Chunking Strategies Evaluated

We analyze three pipeline chunking strategies with three different methods: (i) Fixed-size chunking divides data into fixed-size and partially overlapping frames. (ii) Recursive chunking follows the hierarchical structure of the document (e.g. sections, paragraphs) and resorts to fixed-size chunking only when a document segment is too large. (iii) Semantic chunking uses embedding and identifies boundaries where there is a significant drop in embedding-similarity between segments. Because all three methods will achieve the same target size for each chunk, we use algorithm 1 to describe the most complex method (semantic chunking).

ALGORITHM 1: Semantic Chunking via Embedding Discontinuity Detection
Input: document D split into sentences [s_1, ..., s_n]
      embedding model E, similarity threshold tau, max_chunk_tokens
Output: list of chunks C
  
1: embeddings <- [E(s_i) for s_i in sentences]
2: breakpoints <- []
3: for i in 1 .. n-1:
4:     sim <- cosine_similarity(embeddings[i], embeddings[i+1])
5:     if sim < tau:
6:         breakpoints.append(i)
7: chunks <- split(sentences, breakpoints)
8: for chunk in chunks:
9:     if token_count(chunk) > max_chunk_tokens:
10:        chunks <- chunks - chunk + recursive_split(chunk)
11: return merge_small_chunks(chunks, min_tokens=64)

The similarity threshold tau is set adaptively per document as the 95th percentile of the distribution of inter-sentence distances, following common practice in semantic chunking implementations; this avoids requiring a fixed global threshold across documents with differing writing styles (e.g., narrative MD&A sections versus tabular financial statements). Algorithm 2 shows the core implementation of the chunking dispatcher used in our pipeline, written in Python. The dispatcher exposes a uniform interface so that the retrieval and evaluation stages downstream are agnostic to which chunking strategy produced a given set of chunks.

ALGORITHM 2: Chunking strategy dispatcher (Python)
from dataclasses import dataclass
from typing import Literal
  
@dataclass
class Chunk:
   text: str
   doc_id: str
   start_char: int
   end_char: int
   strategy: str
  
def chunk_document(text: str, doc_id: str,
                   strategy: Literal['fixed', 'recursive', 'semantic'],
                   target_tokens: int = 256, overlap: int = 32) -> list[Chunk]:
   if strategy == 'fixed':
       return _fixed_size_chunk(text, doc_id, target_tokens, overlap)
   elif strategy == 'recursive':
       return _recursive_chunk(text, doc_id, target_tokens,
                                separators=['\ n\ n', '\ n', '. ', ' '])
   elif strategy == 'semantic':
       return _semantic_chunk(text, doc_id, target_tokens,
                               embed_fn=embedding_model.encode)
   raise ValueError(f'Unknown chunking strategy: {strategy}')

3.4 Retrieval Configurations

For each chunking method, we use different retrieval setups: basic, dense retrieval where the most relevant chunks are determined using cosine similarity and second, a two-step retrieval method where an initial dense retrieval is followed by a cross-encoder-based relevance ranking of the top-k chunks prior to the generator. This results in six total experimental setups (three chunking methods x two retrieval methods).

3.5 Evaluation Metrics

We report three categories of metrics. Retrieval precision measures the proportion of retrieved chunks that overlap with the gold reference passage for each question. Answer accuracy measures whether the generated answer matches the gold answer, assessed using a combination of exact-match-style checks for numeric answers and semantic-equivalence judgment for free-text answers. Hallucination rate is computed by decomposing each generated answer into atomic claims and using an LLM-as-judge protocol to determine, for each claim, whether it is supported by the retrieved context. Formally, for an answer A decomposed into claims $\{ {c\_1,{\rm{ }}...,{\rm{ }}c\_m} \}$, the hallucination rate H(A) is defined in Equation 1.

\begin{equation} H\left( A \right) = \left( {1/m} \right)*sum\_\left\{ {i = 1..m} \right\}\left[ {1{\rm{ }} - support\left( {c\_i,{\rm{ }}R} \right)} \right] \end{equation}
((1))
where R is the retrieved context passed to the generator and support(c_i, R) in {0,1} indicates whether claim c_i is judged supported by R. The per-claim support judgment is produced by a separate judge LLM prompted with the claim and the retrieved context, and validated against a manually annotated subset of 150 answers, on which we observe a Cohen's kappa of 0.81 between the judge model and human annotators, indicating substantial agreement.

Algorithm 3 shows the core evaluation loop used to compute hallucination rate, retrieval precision, and answer accuracy across the six experimental conditions.

ALGORITHM 3: Evaluation loop for precision, accuracy, and hallucination rate (Python)
def evaluate_pipeline(questions, pipeline, judge_llm):
   results = []
   for q in questions:
       retrieved = pipeline.retrieve(q.query, top_k=5)
       context = '\ n'.join(c.text for c in retrieved)
       answer = pipeline.generate(q.query, context)
  
       precision = retrieval_precision(retrieved, q.gold_passage_id)
       accuracy = answer_accuracy(answer, q.gold_answer, q.qtype)
  
       claims = decompose_into_claims(answer, llm=judge_llm)
       supported = [judge_llm.is_supported(c, context) for c in claims]
       halluc_rate = 1 - (sum(supported) / max(len(claims), 1))
  
       results.append(dict(qid=q.id, precision=precision,
                            accuracy=accuracy, halluc_rate=halluc_rate))
   return aggregate(results)

4 Results

We present retrieval precision, output answer accuracy, and the answer hallucination rate for each of the six experimental setups (three chunking methods x two retrieval methods) across 412 financial document questions. Table 1 summarizes the results.

Table 1: Retrieval precision@5, answer accuracy, and hallucination rate by configuration (n = 412 questions)
Configuration Precision@5 Answer Accuracy Halluc. Rate Median Latency (ms)
Fixed-size + Plain 0.612 0.701 0.231 184
Fixed-size + Rerank 0.703 0.748 0.176 312
Recursive + Plain 0.701 0.766 0.184 201
Recursive + Rerank 0.781 0.809 0.129 329
Semantic + Plain 0.764 0.812 0.142 238
Semantic + Rerank 0.832 0.861 0.091 367

4.1 Effect of Chunking Strategy

In Figures 2 and 3, we see the hallucination rate and retrieval precision for the six configurations. Among retrieval methods, chunking by semantics achieves the highest retrieval precision for both methods (0.764 plain, 0.832 reranked) and the lowest hallucination rate (0.142 plain, 0.091 reranked). This finding supports the assumption that chunking by semantics minimizes the splitting of the relevant data. Among the three types of chunking, recursive chunking is located between fixed-size and semantic chunking on all metrics. Relative to the fixed-size baseline, recursive chunking achieves about 75% of the reduction in hallucination rate. However, recursive chunking also incurs a significantly lower cost of indexing time, as it doesn't require an embedding pass across every sentence boundary.

Figure 2
Figure 2: Hallucination rate by chunking strategy and retrieval configuration. Lower is better.
Figure 3
Figure 3: Retrieval Precision@5 by chunking strategy and retrieval configuration. Higher is better.

4.2 Effect of Reranking

For all types of chunking, the addition of cross-encoder reranking improves retrieval precision by as much as 0.091 and reduces hallucination rate by as much as 0.055. This is achieved at a cost of increased median query latency (Table 1 and Figure 4) by 70-130 ms. The relative reduction in hallucination rate grows with chunk quality: 23.8% for fixed-size, 29.9% for recursive, and 35.9% for semantic chunking. Semantic chunking therefore benefits most from reranking in relative terms, even though it starts from the lowest hallucination rate. These results support the assumption that chunking and reranking in the cross-encoder are complementary mechanisms rather than substitutes for each other.

Figure 4
Figure 4: Latency vs. hallucination rate tradeoff across all six pipeline configurations. Points toward the bottom-left are preferable.

4.3 Hallucination Rate Analysis by Question Type

Breaking down hallucination rate by question type, we observe that numeric questions (e.g., ”What was reported quarterly revenue?”) show consistently lower hallucination rates (0.071-0.198 across configurations) than free-text synthesis questions requiring aggregation across multiple passages (e.g., ”Summarize the key risk factors discussed in the filing”), which show hallucination rates of 0.134-0.289. This gap is narrowest under the Semantic + Rerank configuration (0.058 vs. 0.117), suggesting that improved retrieval grounding disproportionately benefits multi-passage synthesis questions, which are most vulnerable to incomplete context.

5 Discussion and Limitations

Our findings confirm that chunking strategy is a primary lever for controlling hallucination in domain-specific RAG systems, not a minor implementation detail. Practitioners should treat it as a first-class design decision validated on representative queries, not a default. Limitations include single-domain evaluation (financial documents), potential judge-model bias in the LLM-as-judge metric, and the exclusion of adaptive chunking strategies.

6 Conclusion

This paper demonstrated that chunking strategy and retrieval configuration are primary determinants of hallucination rate in domain-specific RAG pipelines. Across 412 financial-document QA pairs, semantic chunking with cross-encoder reranking reduced hallucination rate by 60.6% relative to the fixed-size plain-retrieval baseline (0.091 vs. 0.231), while recursive chunking offered a practical middle ground. Future work will explore adaptive chunk sizing and retrieval-confidence-aware generation for broader domain applicability.

References

  • Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Rocktaschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 33. 9459-9474.
  • Gautier Izacard and Edouard Grave. 2021. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics (EACL). 874-880.
  • Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6769-6781.
  • Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157-173.
  • Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 9004-9017.
  • Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of Hallucination in Natural Language Generation. Comput. Surveys 55, 12 (2023), Article 248.
  • Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). 3982-3992.
  • Yujia Qin, Zihan Cai, Dian Jin, Lan Yan, Shihao Liang, Kunlun Zhu, Yankai Lin, Xu Han, Ning Ding, Huadong Wang, Ruobing Xie, Fanchao Qi, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023. WebCPM: Interactive Web Search for Chinese Long-form Question Answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL). 8968-8988.
  • Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023. Retrieval-based Language Models and Applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts. 41-46.
  • Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-Based Knowledge Conflicts in Question Answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7052-7063.
  • Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling Large Language Models to Generate Text with Citations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 6465-6488.
  • Zhengbao Jiang, Frank F. Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active Retrieval Augmented Generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 7969-7992.
  • Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-Context Retrieval-Augmented Language Models. Transactions of the Association for Computational Linguistics 11 (2023), 1316-1331.
  • Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2023. Retrieval-Augmented Generation for Large Language Models: A Survey. arXiv preprint arXiv:2312.10997.

CC-BY license image
This work is licensed under a Creative Commons Attribution 4.0 International License.

ICAAI 2026, London, United Kingdom

© 2026 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-2795-5/2026/09
DOI: https://doi.org/10.1145/3859174.3859178