From 08c5aded35eb554138882ca05c62a16da85b4950 Mon Sep 17 00:00:00 2001 From: izapata Date: Thu, 26 Mar 2026 15:32:23 +0100 Subject: [PATCH] fix(docs): improve formatting and readability in ADR-0005 for embedding model selection --- .../ADR/ADR-0005-embedding-model-selection.md | 148 +++++++++--------- 1 file changed, 77 insertions(+), 71 deletions(-) diff --git a/docs/ADR/ADR-0005-embedding-model-selection.md b/docs/ADR/ADR-0005-embedding-model-selection.md index 226ba3c..fc51d1c 100644 --- a/docs/ADR/ADR-0005-embedding-model-selection.md +++ b/docs/ADR/ADR-0005-embedding-model-selection.md @@ -1,7 +1,7 @@ # ADR-0005: Embedding Model Selection — Comparative Evaluation of BGE-M3 vs Qwen3-Embedding-0.6B -**Date:** 2026-03-19 -**Status:** Under Evaluation +**Date:** 2026-03-19 +**Status:** Under Evaluation **Deciders:** Rafael Ruiz (CTO), MrHouston Engineering --- @@ -14,27 +14,27 @@ The AVAP RAG pipeline requires an embedding model capable of mapping a hybrid co A chunk-level audit was performed on the full indexable corpus: the AVAP Language Reference Manual (`avap.md`) and 40 representative `.avap` code samples. Results (`test_chunks.jsonl`, 190 chunks): -| Metric | Value | -|---|---| -| Total chunks | 190 | -| Total tokens indexed | 11,498 | -| Minimum chunk size | 1 token | -| Maximum chunk size | 833 tokens | -| Mean chunk size | 60.5 tokens | -| Median chunk size | 29 tokens | -| p90 | 117 tokens | -| p95 | 204 tokens | -| p99 | 511 tokens | +| Metric | Value | +| -------------------- | ----------- | +| Total chunks | 190 | +| Total tokens indexed | 11,498 | +| Minimum chunk size | 1 token | +| Maximum chunk size | 833 tokens | +| Mean chunk size | 60.5 tokens | +| Median chunk size | 29 tokens | +| p90 | 117 tokens | +| p95 | 204 tokens | +| p99 | 511 tokens | **Corpus composition by type:** -| Type | Count | Description | -|---|---|---| -| Narrative (Spanish prose) | 79 | LRM explanations, concept descriptions | -| Code chunks | 83 | AVAP `.avap` sample files | -| BNF formal grammar | 9 | Formal language specification in English | -| Code examples | 14 | Inline examples within LRM | -| Function signatures | 2 | Extracted function headers | +| Type | Count | Description | +| ------------------------- | ----- | ---------------------------------------- | +| Narrative (Spanish prose) | 79 | LRM explanations, concept descriptions | +| Code chunks | 83 | AVAP `.avap` sample files | +| BNF formal grammar | 9 | Formal language specification in English | +| Code examples | 14 | Inline examples within LRM | +| Function signatures | 2 | Extracted function headers | **Linguistic composition:** 55% of chunks originate from the LRM (`avap.md`), written in Spanish with embedded English DSL identifiers. 45% are `.avap` code files containing English command names (`addVar`, `addResult`, `registerEndpoint`, `ormDirect`) with Spanish-language string literals and variable names (`"Hola"`, `datos_cliente`, `mi_json_final`, `contraseña`, `fecha`). 18.9% of chunks (36 out of 190) contain both Spanish content and English DSL commands within the same chunk — intra-chunk multilingual mixing. @@ -65,27 +65,27 @@ Benchmark confirmation (BEIR evaluation, three datasets): **CodeXGLUE** (code retrieval from GitHub repositories): -| k | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall | -|---|---|---|---|---| -| 1 | 0.00031 | 0.00031 | **0.9497** | **0.9497** | -| 5 | 0.00086 | 0.00151 | **0.9716** | **0.9876** | -| 10 | 0.00118 | 0.00250 | **0.9734** | **0.9929** | +| k | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall | +| -- | ----------------- | ------------------- | ------------------- | --------------------- | +| 1 | 0.00031 | 0.00031 | **0.9497** | **0.9497** | +| 5 | 0.00086 | 0.00151 | **0.9716** | **0.9876** | +| 10 | 0.00118 | 0.00250 | **0.9734** | **0.9929** | **CoSQA** (natural language queries over code — closest proxy to AVAP retrieval): -| k | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall | -|---|---|---|---|---| -| 1 | 0.00000 | 0.00000 | **0.1740** | **0.1740** | -| 10 | 0.00000 | 0.00000 | **0.3909** | **0.6700** | -| 100 | 0.00210 | 0.01000 | **0.4510** | **0.9520** | +| k | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall | +| --- | ----------------- | ------------------- | ------------------- | --------------------- | +| 1 | 0.00000 | 0.00000 | **0.1740** | **0.1740** | +| 10 | 0.00000 | 0.00000 | **0.3909** | **0.6700** | +| 100 | 0.00210 | 0.01000 | **0.4510** | **0.9520** | **SciFact** (scientific prose — out-of-domain control): -| k | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall | -|---|---|---|---|---| -| 1 | 0.02333 | 0.02083 | **0.5633** | **0.5299** | -| 10 | 0.04619 | 0.07417 | **0.6855** | **0.8161** | -| 100 | 0.07768 | 0.23144 | **0.7129** | **0.9400** | +| k | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall | +| --- | ----------------- | ------------------- | ------------------- | --------------------- | +| 1 | 0.02333 | 0.02083 | **0.5633** | **0.5299** | +| 10 | 0.04619 | 0.07417 | **0.6855** | **0.8161** | +| 100 | 0.07768 | 0.23144 | **0.7129** | **0.9400** | Qwen2.5-1.5B is eliminated. **Qwen3-Embedding-0.6B is the validated baseline.** @@ -110,12 +110,14 @@ The model that demonstrates superior performance under the evaluation criteria d ### Qwen3-Embedding-0.6B **Strengths:** + - Already benchmarked on CodeXGLUE, CoSQA and SciFact — strong results documented - 32,768 token context window — exceeds corpus requirements with large margin - Same model family as the generation model (Qwen) — shared tokenizer vocabulary - Lowest integration risk — already validated in the pipeline **Limitations:** + - Benchmarks are English-only — multilingual performance on AVAP corpus unvalidated - Not a dedicated multilingual model — training distribution weighted towards English and Chinese - No native sparse retrieval support @@ -125,11 +127,13 @@ The model that demonstrates superior performance under the evaluation criteria d ### BGE-M3 **Strengths:** + - Explicit multilingual contrastive training across 100+ languages including programming languages — direct architectural fit for the intra-chunk Spanish/English/DSL mixing observed in the corpus - Supports dense, sparse and multi-vector ColBERT retrieval from a single model inference — future path to consolidating the current BM25+kNN dual-system architecture (ADR-0003) - Higher MTEB retrieval score than Qwen3-Embedding-0.6B in the programming domain **Limitations:** + - Not yet benchmarked on CodeXGLUE, CoSQA or SciFact at the time of candidate selection — no prior empirical results for this corpus - 8,192 token context window — sufficient for current corpus (max chunk: 833 tokens, 10.2% utilization) but lower headroom for future corpus growth - Requires tokenizer alignment: `HF_EMB_MODEL_NAME` must be updated to `BAAI/bge-m3` alongside `OLLAMA_EMB_MODEL_NAME` to keep chunk token counting consistent @@ -185,57 +189,57 @@ BGE-M3 benchmarks were completed on the same three BEIR datasets using identical **CodeXGLUE** (code retrieval from GitHub repositories): -| Metric | k | BGE-M3 | Qwen3-Emb-0.6B | Delta (BGE-M3 − Qwen3) | -|---|---|---|---|---| -| NDCG | 1 | **0.9520** | 0.9497 | +0.23 pp | -| NDCG | 5 | **0.9738** | 0.9717 | +0.21 pp | -| NDCG | 10 | **0.9749** | 0.9734 | +0.15 pp | -| NDCG | 100 | **0.9763** | 0.9745 | +0.18 pp | -| Recall | 1 | **0.9520** | 0.9497 | +0.23 pp | -| Recall | 5 | **0.9892** | 0.9876 | +0.16 pp | -| Recall | 10 | 0.9928 | **0.9930** | −0.02 pp | -| Recall | 100 | **0.9989** | 0.9981 | +0.08 pp | +| Metric | k | BGE-M3 | Qwen3-Emb-0.6B | Delta (BGE-M3 − Qwen3) | +| ------ | --- | ---------------- | ---------------- | ----------------------- | +| NDCG | 1 | **0.9520** | 0.9497 | +0.23 pp | +| NDCG | 5 | **0.9738** | 0.9717 | +0.21 pp | +| NDCG | 10 | **0.9749** | 0.9734 | +0.15 pp | +| NDCG | 100 | **0.9763** | 0.9745 | +0.18 pp | +| Recall | 1 | **0.9520** | 0.9497 | +0.23 pp | +| Recall | 5 | **0.9892** | 0.9876 | +0.16 pp | +| Recall | 10 | 0.9928 | **0.9930** | −0.02 pp | +| Recall | 100 | **0.9989** | 0.9981 | +0.08 pp | Both models perform near-identically on CodeXGLUE. All deltas are below 0.25 absolute percentage points. This dataset does not differentiate the candidates. **CoSQA** (natural language queries over code — most representative proxy for AVAP retrieval): -| Metric | k | BGE-M3 | Qwen3-Emb-0.6B | Delta (BGE-M3 − Qwen3) | -|---|---|---|---|---| -| NDCG | 1 | 0.1160 | **0.1740** | −5.80 pp | -| NDCG | 5 | 0.2383 | **0.3351** | −9.68 pp | -| NDCG | 10 | 0.2878 | **0.3909** | −10.31 pp | -| NDCG | 100 | 0.3631 | **0.4510** | −8.79 pp | -| Recall | 1 | 0.1160 | **0.1740** | −5.80 pp | -| Recall | 5 | 0.3660 | **0.5020** | −13.60 pp | -| Recall | 10 | 0.5160 | **0.6700** | −15.40 pp | -| Recall | 100 | 0.8740 | **0.9520** | −7.80 pp | +| Metric | k | BGE-M3 | Qwen3-Emb-0.6B | Delta (BGE-M3 − Qwen3) | +| ------ | --- | ------ | ---------------- | ----------------------- | +| NDCG | 1 | 0.1160 | **0.1740** | −5.80 pp | +| NDCG | 5 | 0.2383 | **0.3351** | −9.68 pp | +| NDCG | 10 | 0.2878 | **0.3909** | −10.31 pp | +| NDCG | 100 | 0.3631 | **0.4510** | −8.79 pp | +| Recall | 1 | 0.1160 | **0.1740** | −5.80 pp | +| Recall | 5 | 0.3660 | **0.5020** | −13.60 pp | +| Recall | 10 | 0.5160 | **0.6700** | −15.40 pp | +| Recall | 100 | 0.8740 | **0.9520** | −7.80 pp | Qwen3-Embedding-0.6B outperforms BGE-M3 on CoSQA by a wide margin at every k. The NDCG@10 gap is 10.31 absolute percentage points. CoSQA is the most representative proxy for the AVAP retrieval use case — it pairs natural language queries with code snippets — making this the most significant BEIR result. **SciFact** (scientific prose — out-of-domain control): -| Metric | k | BGE-M3 | Qwen3-Emb-0.6B | Delta (BGE-M3 − Qwen3) | -|---|---|---|---|---| -| NDCG | 1 | 0.5100 | **0.5533** | −4.33 pp | -| NDCG | 5 | 0.6190 | **0.6593** | −4.03 pp | -| NDCG | 10 | 0.6431 | **0.6785** | −3.54 pp | -| NDCG | 100 | 0.6705 | **0.7056** | −3.51 pp | -| Recall | 1 | 0.4818 | **0.5243** | −4.25 pp | -| Recall | 5 | 0.7149 | **0.7587** | −4.38 pp | -| Recall | 10 | 0.7834 | **0.8144** | −3.10 pp | -| Recall | 100 | 0.9037 | **0.9367** | −3.30 pp | +| Metric | k | BGE-M3 | Qwen3-Emb-0.6B | Delta (BGE-M3 − Qwen3) | +| ------ | --- | ------ | ---------------- | ----------------------- | +| NDCG | 1 | 0.5100 | **0.5533** | −4.33 pp | +| NDCG | 5 | 0.6190 | **0.6593** | −4.03 pp | +| NDCG | 10 | 0.6431 | **0.6785** | −3.54 pp | +| NDCG | 100 | 0.6705 | **0.7056** | −3.51 pp | +| Recall | 1 | 0.4818 | **0.5243** | −4.25 pp | +| Recall | 5 | 0.7149 | **0.7587** | −4.38 pp | +| Recall | 10 | 0.7834 | **0.8144** | −3.10 pp | +| Recall | 100 | 0.9037 | **0.9367** | −3.30 pp | Qwen3-Embedding-0.6B leads BGE-M3 on SciFact by 3–4 absolute percentage points across all metrics. The gap is consistent but narrower than on CoSQA. ### BEIR summary — NDCG@10 comparison -| Dataset | BGE-M3 | Qwen3-Emb-0.6B | Delta | Leader | -|---|---|---|---|---| -| CodeXGLUE | 0.9749 | 0.9734 | +0.15 pp | BGE-M3 (marginal) | -| CoSQA | 0.2878 | **0.3909** | −10.31 pp | **Qwen3** | -| SciFact | 0.6431 | **0.6785** | −3.54 pp | **Qwen3** | -| **Mean** | **0.6353** | **0.6809** | **−4.56 pp** | **Qwen3** | +| Dataset | BGE-M3 | Qwen3-Emb-0.6B | Delta | Leader | +| -------------- | ---------------- | ---------------- | ------------------- | ----------------- | +| CodeXGLUE | 0.9749 | 0.9734 | +0.15 pp | BGE-M3 (marginal) | +| CoSQA | 0.2878 | **0.3909** | −10.31 pp | **Qwen3** | +| SciFact | 0.6431 | **0.6785** | −3.54 pp | **Qwen3** | +| **Mean** | **0.6353** | **0.6809** | **−4.56 pp** | **Qwen3** | Qwen3-Embedding-0.6B leads on mean NDCG@10 by 4.56 absolute percentage points, driven primarily by a 10.31 pp advantage on CoSQA. @@ -260,6 +264,8 @@ The BEIR benchmarks — the secondary decision signal — favour Qwen3-Embedding The EvaluateRAG evaluation — the primary decision signal — remains pending. It is the only evaluation that directly measures retrieval quality on the actual AVAP corpus with its intra-chunk multilingual mixing. BGE-M3's architectural fit for multilingual content could still produce a measurable advantage on the production corpus that the English-only BEIR benchmarks cannot capture. No final model selection will be made until EvaluateRAG results are available for both candidates. +We have found that Qwen3-embedding is multi-lingual, with good scores in multi-lingual benchmarks. The documentation says so, but the definitive answer will be provided by the scores of the evaluation on the AVAP corpus. + --- ## Consequences