Completeness in RAG Systems

# In ensure-completeness-basic.py
store = DocumentStoreBasic(
    source="custom",
    custom_documents_path=str(custom_docs_path),
    num_docs=1,  # Only one document
    chunk_size=1000  # Larger chunks, less precise
)

The basic implementation suffers from several critical limitations in document retrieval. It’s designed to return only a single document, which significantly limits the breadth of information available for answering questions. The search mechanism relies on basic L2 distance for similarity matching, which is less effective for semantic search compared to more sophisticated approaches. Additionally, the system lacks any form of reranking or relevance scoring, meaning it can’t refine its results based on deeper semantic understanding. The chunk size is also quite large at 1000 characters, which can lead to less precise and potentially less relevant information being included in each chunk.

# In document_store_basic.py
def _load_custom_documents(self, documents_path: str, chunk_size: int):
    # Simple chunking by paragraphs
    paragraphs = [p.strip() for p in text.split('\n\n') if p.strip()]
    # Basic sentence-based chunking
    sentences = re.split(r'(?<=[.!?])\s+', paragraph)

The document processing in the basic implementation is quite rudimentary. It uses a simple paragraph-based splitting approach followed by basic sentence-based chunking. This method doesn’t preserve context effectively, as it doesn’t maintain relationships between chunks or consider the semantic coherence of the text. The metadata stored with each chunk is minimal, containing only basic information like source and chunk ID, without any sophisticated enrichment or relationship tracking. The chunking strategy is also fixed, applying the same approach regardless of the content type or structure.

# In ensure-completeness-basic.py
SYSTEM_PROMPT = "You are a helpful assistant. Answer questions based on the provided document."

class Prompts:
    BASIC = """Question: {query}

Document:
{documents}

Please provide an answer based on the document above."""

The prompting strategy in the basic implementation is minimal and lacks specific guidance for ensuring complete answers. The system prompt is very simple, providing no explicit instructions about completeness or how to handle multiple sources of information. There are no requirements for document synthesis or citation, which can lead to answers that don’t fully utilize the available information or properly attribute their sources.

# In ensure-completeness-enhanced.py
store = DocumentStore(
    source="custom",
    custom_documents_path=str(custom_docs_path),
    num_docs=10,  # More documents
    use_reranking=True,  # Use reranking
    reranking_threshold=0.6
)

The improved implementation significantly improves document retrieval by returning multiple relevant documents (10 or more) instead of just one. It uses cosine similarity for semantic matching, which provides better results for finding conceptually related content. The system implements sophisticated reranking with a threshold of 0.6, allowing it to refine and prioritize the most relevant results. The chunk size is also reduced to 512 characters, making the chunks more precise and focused.

# In document_store.py
def _chunk_text(self, text: str, chunk_size: int) -> List[str]:
    # First split into sentences
    sentences = re.split(r'(?<=[.!?])\s+', text)
    # Maintain context by keeping sentences together
    chunks = []
    current_chunk = []
    current_length = 0

The document processing in the improved implementation is much more sophisticated. It uses context-aware chunking that preserves sentence relationships and maintains semantic coherence. Each chunk is enriched with rich metadata including relevance scores, source tracking, and detailed formatting with relevance indicators. The system also implements dynamic chunking that adapts to the content structure, ensuring better context preservation.

# In ensure-completeness-enhanced.py
SYSTEM_PROMPT = """You are a knowledgeable science historian tasked with providing comprehensive answers...

IMPORTANT INSTRUCTIONS:
1. You MUST use ALL relevant information from the retrieved documents...
2. For each key fact or detail you mention, explicitly cite the source document number...
3. If multiple documents contain related information, synthesize them...
"""

The improved implementation uses a comprehensive system prompt that provides clear guidance for ensuring complete answers. It explicitly instructs the model to use all relevant information from the retrieved documents, requires proper citation of sources, and encourages synthesis of information across multiple documents. The prompt also specifies thoroughness requirements and provides a clear role for the model as a knowledgeable science historian.

# Basic (document_store_basic.py)
def search(self, query: str) -> list:
    # Get only one result
    distances, indices = self.index.search(query_vector.astype('float32'), self.num_docs)

# Enhanced (document_store.py)
def search(self, query: str) -> list:
    # Get more initial results if using reranking
    initial_k = self.k * self.reranking_multiplier if self.use_reranking else self.k
    scores, indices = self.index.search(query_vector.astype('float32'), initial_k)

The basic implementation’s search functionality is quite limited, returning only a single document which can lead to missing key information. In contrast, the improved implementation uses a more sophisticated approach that retrieves multiple documents and implements reranking to ensure comprehensive coverage of the topic. The enhanced version also uses a multiplier for initial results when reranking is enabled, allowing for better selection of the most relevant content.

# Basic (ensure-completeness-basic.py)
SYSTEM_PROMPT = "You are a helpful assistant. Answer questions based on the provided document."

# Enhanced (ensure-completeness-enhanced.py)
SYSTEM_PROMPT = """You are a knowledgeable science historian..."""

The basic implementation’s prompting strategy is minimal and doesn’t encourage synthesis of information across multiple sources. The improved implementation, however, uses a more sophisticated prompt that explicitly requires the model to synthesize information from multiple documents and provide comprehensive answers. This leads to more complete and well-rounded responses that draw from all available relevant information.

# Basic (document_store_basic.py)
def _load_custom_documents(self, documents_path: str, chunk_size: int):
    # Simple chunking by paragraphs
    paragraphs = [p.strip() for p in text.split('\n\n') if p.strip()]

# Enhanced (document_store.py)
def _chunk_text(self, text: str, chunk_size: int) -> List[str]:
    # Context-aware chunking
    sentences = re.split(r'(?<=[.!?])\s+', text)

The basic implementation uses a simple paragraph-based chunking approach that can lead to loss of context, especially with larger chunks. The improved implementation, on the other hand, uses a more sophisticated context-aware chunking strategy that maintains sentence relationships and semantic coherence. This results in better preservation of context and more coherent information retrieval.

# Basic (ensure-completeness-basic.py)
class Prompts:
    BASIC = """Question: {query}
    Document: {documents}
    Please provide an answer based on the document above."""

# Enhanced (ensure-completeness-enhanced.py)
class Prompts:
    ENHANCED = """Question: {query}
    Documents: {documents}
    IMPORTANT: Your response must:
    1. Use ALL relevant information from the documents above
    2. Cite specific document numbers for each fact
    3. Synthesize information from multiple documents..."""

The basic implementation’s prompting strategy often results in answers that miss key details and lack proper context. The improved implementation, with its comprehensive prompting strategy, produces answers that are more thorough, properly cited, and effectively synthesize information from multiple sources. This leads to higher quality, more complete answers that better serve the user’s needs.

# improved approach
store = DocumentStore(
    num_docs=10,
    use_reranking=True,
    reranking_threshold=0.6
)

Effective document retrieval is crucial for ensuring complete answers. The improved approach retrieves multiple relevant documents (typically 10 or more) to ensure comprehensive coverage of the topic. It uses semantic similarity search to find conceptually related content, rather than just keyword matches. The implementation of reranking with a threshold of 0.6 helps refine and prioritize the most relevant results. The system also uses appropriate chunk sizes to ensure that the retrieved information is precise and focused.

# improved approach
def _chunk_text(self, text: str, chunk_size: int) -> List[str]:
    # Context-aware chunking
    sentences = re.split(r'(?<=[.!?])\s+', text)
    chunks = []
    current_chunk = []
    current_length = 0

Sophisticated document processing is essential for maintaining context and relationships between pieces of information. The improved approach uses context-aware chunking that preserves sentence relationships and semantic coherence. Each chunk is enriched with relevant metadata, including source tracking and relevance indicators. The system also implements dynamic chunking that adapts to the content structure, ensuring better context preservation and more coherent information retrieval.

# improved approach
SYSTEM_PROMPT = """You are a knowledgeable science historian...
IMPORTANT INSTRUCTIONS:
1. You MUST use ALL relevant information...
2. For each key fact or detail you mention...
3. If multiple documents contain related information..."""

Effective prompting is crucial for guiding the model to produce complete and well-structured answers. The improved approach uses a comprehensive system prompt that provides clear instructions for ensuring completeness. It requires proper citation of sources and encourages synthesis of information across multiple documents. The prompt also specifies thoroughness requirements and provides a clear role for the model, helping it understand the expectations for the quality and completeness of its responses.