# ADR-0006: Code Indexing Improvements — Comparative Evaluation of code chunking strategies

**Date:** 2026-03-24  
**Status:** Proposed 
**Deciders:** Rafael Ruiz (CTO), MrHouston Engineering

---

## Context
Efficient code indexing is a critical component for enabling high-quality code search, retrieval-augmented generation (RAG), and semantic understanding in developer tooling. The main challenge lies in representing source code in a way that preserves its syntactic and semantic structure while remaining suitable for embedding-based retrieval systems. 

In this context, we explored different strategies to improve the indexing of .avap code files, starting from a naïve approach and progressively moving toward more structured representations based on parsing techniques. 

### Alternatives
- File-level chunking (baseline):

    Each .avap file is treated as a single chunk and indexed directly. This approach is simple and fast but ignores internal structure (functions, classes, blocks). 


- EBNF chunking as metadata:

    Each .avap file is still treated as a single chunk and indexed directly. However, by using the AVAP EBNF syntax, we extract the AST structure and injects it into the chunk metadata.


- Full EBNF chunking:

    Each .avap file is still treated as a single chunk and indexed directly. The difference between this approach and the last 2, is that the AST is indexed instead the code.


- Grammar definition chunking:

    Code is segmented using a language-specific configuration (`avap_config.json`) instead of one-file chunks. The chunker applies a lexer (comments/strings), identifies multi-line blocks (`function`, `if`, `startLoop`, `try`), classifies single-line statements (`registerEndpoint`, `orm_command`, `http_command`, etc.), and enriches every chunk with semantic tags (`uses_orm`, `uses_http`, `uses_async`, `returns_result`, among others).

    This strategy also extracts function signatures as dedicated lightweight chunks and propagates local context between nearby chunks (semantic overlap), improving retrieval precision for both API-level and implementation-level queries.


### Indexed docs
For each strategy, we created a different Elasticsearch Index with their own characteristics. The 3 first approaches have 33 chunks (1 chunk per file), whereas the last approach has 89 chunks.


### How can we evaluate each strategy?

**Evaluation Protocol:**

1. **Golden Dataset**
   - Generate a set of natural language queries paired with their ground-truth context (filename).
   - Each query should be answerable by examining one or more code samples.
   - Example: Query="How do you handle errors in AVAP?" → Context="try_catch_request.avap"

2. **Test Each Strategy**
   - For each of the 4 chunking strategies, run the same set of queries against the respective Elasticsearch index.
   - Record the top-10 retrieved chunks for each query.

3. **Metrics**
    - `NDCG@10`: Normalized discounted cumulative gain at rank 10 (measures ranking quality).
    - `Recall@10`: Fraction of relevant chunks retrieved in top 10.
    - `MRR@10`: Mean reciprocal rank (position of first relevant result).

4. **Relevance Judgment**
   - A chunk is considered relevant if it contains code directly answering the query.
   - For file-level strategies: entire file is relevant or irrelevant.
   - For grammar-definition: specific block/statement chunks are relevant even if the full file is not.

5. **Acceptance Criteria**
   - **Grammar definition must achieve at least a 10% improvement in NDCG@10 over file-level baseline.**
   - **Recall@10 must not drop by more than 5 absolute percentage points vs file-level.**
   - **Index size increase must remain below 50% of baseline.**

## Decision



## Rationale



## Consequences