Merge branch 'mrh-online-dev' of github.com:BRUNIX-AI/assistance-engine into mrh-online-dev

changelog

@@ -3,6 +3,18 @@
 All notable changes to the **Brunix Assistance Engine** will be documented in this file. This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ---
+## [1.6.3] - 2026-03-26
+### Changed
+- RESEARCH: updated ADR-0005-embedding-model-selection, BEIR benchmark results (qwen3-emb vs bge-m3) added.
+
+### Added
+- FEATURE: added `research/embeddings/evaluate_embeddings_pipeline.py` in order to facilitate model embedding evaluation with BEIR benchmarks.
+- RESULTS: added qwen3-emb vs bge-m3 scores in BEIR benchmark.
+
+## [1.6.2] - 2026-03-26
+### Changed
+- RESEARCH: updated `embeddings/Embedding model selection.pdf`.
+
 
 ## [1.6.1] - 2026-03-20
 

docs/ADR/ADR-0005-embedding-model-selection.md

@@ -15,7 +15,7 @@ The AVAP RAG pipeline requires an embedding model capable of mapping a hybrid co
 A chunk-level audit was performed on the full indexable corpus: the AVAP Language Reference Manual (`avap.md`) and 40 representative `.avap` code samples. Results (`test_chunks.jsonl`, 190 chunks):
 
 | Metric               | Value       |
-|---|---|
+| -------------------- | ----------- |
 | Total chunks         | 190         |
 | Total tokens indexed | 11,498      |
 | Minimum chunk size   | 1 token     |
@@ -29,7 +29,7 @@ A chunk-level audit was performed on the full indexable corpus: the AVAP Languag
 **Corpus composition by type:**
 
 | Type                      | Count | Description                              |
-|---|---|---|
+| ------------------------- | ----- | ---------------------------------------- |
 | Narrative (Spanish prose) | 79    | LRM explanations, concept descriptions   |
 | Code chunks               | 83    | AVAP `.avap` sample files              |
 | BNF formal grammar        | 9     | Formal language specification in English |
@@ -66,7 +66,7 @@ Benchmark confirmation (BEIR evaluation, three datasets):
 **CodeXGLUE** (code retrieval from GitHub repositories):
 
 | k  | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall |
-|---|---|---|---|---|
+| -- | ----------------- | ------------------- | ------------------- | --------------------- |
 | 1  | 0.00031           | 0.00031             | **0.9497**    | **0.9497**      |
 | 5  | 0.00086           | 0.00151             | **0.9716**    | **0.9876**      |
 | 10 | 0.00118           | 0.00250             | **0.9734**    | **0.9929**      |
@@ -74,7 +74,7 @@ Benchmark confirmation (BEIR evaluation, three datasets):
 **CoSQA** (natural language queries over code — closest proxy to AVAP retrieval):
 
 | k   | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall |
-|---|---|---|---|---|
+| --- | ----------------- | ------------------- | ------------------- | --------------------- |
 | 1   | 0.00000           | 0.00000             | **0.1740**    | **0.1740**      |
 | 10  | 0.00000           | 0.00000             | **0.3909**    | **0.6700**      |
 | 100 | 0.00210           | 0.01000             | **0.4510**    | **0.9520**      |
@@ -82,7 +82,7 @@ Benchmark confirmation (BEIR evaluation, three datasets):
 **SciFact** (scientific prose — out-of-domain control):
 
 | k   | Qwen2.5-1.5B NDCG | Qwen2.5-1.5B Recall | Qwen3-Emb-0.6B NDCG | Qwen3-Emb-0.6B Recall |
-|---|---|---|---|---|
+| --- | ----------------- | ------------------- | ------------------- | --------------------- |
 | 1   | 0.02333           | 0.02083             | **0.5633**    | **0.5299**      |
 | 10  | 0.04619           | 0.07417             | **0.6855**    | **0.8161**      |
 | 100 | 0.07768           | 0.23144             | **0.7129**    | **0.9400**      |
@@ -110,12 +110,14 @@ The model that demonstrates superior performance under the evaluation criteria d
 ### Qwen3-Embedding-0.6B
 
 **Strengths:**
+
 - Already benchmarked on CodeXGLUE, CoSQA and SciFact — strong results documented
 - 32,768 token context window — exceeds corpus requirements with large margin
 - Same model family as the generation model (Qwen) — shared tokenizer vocabulary
 - Lowest integration risk — already validated in the pipeline
 
 **Limitations:**
+
 - Benchmarks are English-only — multilingual performance on AVAP corpus unvalidated
 - Not a dedicated multilingual model — training distribution weighted towards English and Chinese
 - No native sparse retrieval support
@@ -125,11 +127,13 @@ The model that demonstrates superior performance under the evaluation criteria d
 ### BGE-M3
 
 **Strengths:**
+
 - Explicit multilingual contrastive training across 100+ languages including programming languages — direct architectural fit for the intra-chunk Spanish/English/DSL mixing observed in the corpus
 - Supports dense, sparse and multi-vector ColBERT retrieval from a single model inference — future path to consolidating the current BM25+kNN dual-system architecture (ADR-0003)
 - Higher MTEB retrieval score than Qwen3-Embedding-0.6B in the programming domain
 
 **Limitations:**
+
 - Not yet benchmarked on CodeXGLUE, CoSQA or SciFact at the time of candidate selection — no prior empirical results for this corpus
 - 8,192 token context window — sufficient for current corpus (max chunk: 833 tokens, 10.2% utilization) but lower headroom for future corpus growth
 - Requires tokenizer alignment: `HF_EMB_MODEL_NAME` must be updated to `BAAI/bge-m3` alongside `OLLAMA_EMB_MODEL_NAME` to keep chunk token counting consistent
@@ -186,7 +190,7 @@ BGE-M3 benchmarks were completed on the same three BEIR datasets using identical
 **CodeXGLUE** (code retrieval from GitHub repositories):
 
 | Metric | k   | BGE-M3           | Qwen3-Emb-0.6B   | Delta (BGE-M3 − Qwen3) |
-|---|---|---|---|---|
+| ------ | --- | ---------------- | ---------------- | ----------------------- |
 | NDCG   | 1   | **0.9520** | 0.9497           | +0.23 pp                |
 | NDCG   | 5   | **0.9738** | 0.9717           | +0.21 pp                |
 | NDCG   | 10  | **0.9749** | 0.9734           | +0.15 pp                |
@@ -201,7 +205,7 @@ Both models perform near-identically on CodeXGLUE. All deltas are below 0.25 abs
 **CoSQA** (natural language queries over code — most representative proxy for AVAP retrieval):
 
 | Metric | k   | BGE-M3 | Qwen3-Emb-0.6B   | Delta (BGE-M3 − Qwen3) |
-|---|---|---|---|---|
+| ------ | --- | ------ | ---------------- | ----------------------- |
 | NDCG   | 1   | 0.1160 | **0.1740** | −5.80 pp               |
 | NDCG   | 5   | 0.2383 | **0.3351** | −9.68 pp               |
 | NDCG   | 10  | 0.2878 | **0.3909** | −10.31 pp              |
@@ -216,7 +220,7 @@ Qwen3-Embedding-0.6B outperforms BGE-M3 on CoSQA by a wide margin at every k. Th
 **SciFact** (scientific prose — out-of-domain control):
 
 | Metric | k   | BGE-M3 | Qwen3-Emb-0.6B   | Delta (BGE-M3 − Qwen3) |
-|---|---|---|---|---|
+| ------ | --- | ------ | ---------------- | ----------------------- |
 | NDCG   | 1   | 0.5100 | **0.5533** | −4.33 pp               |
 | NDCG   | 5   | 0.6190 | **0.6593** | −4.03 pp               |
 | NDCG   | 10  | 0.6431 | **0.6785** | −3.54 pp               |
@@ -231,7 +235,7 @@ Qwen3-Embedding-0.6B leads BGE-M3 on SciFact by 3–4 absolute percentage points
 ### BEIR summary — NDCG@10 comparison
 
 | Dataset        | BGE-M3           | Qwen3-Emb-0.6B   | Delta               | Leader            |
-|---|---|---|---|---|
+| -------------- | ---------------- | ---------------- | ------------------- | ----------------- |
 | CodeXGLUE      | 0.9749           | 0.9734           | +0.15 pp            | BGE-M3 (marginal) |
 | CoSQA          | 0.2878           | **0.3909** | −10.31 pp          | **Qwen3**   |
 | SciFact        | 0.6431           | **0.6785** | −3.54 pp           | **Qwen3**   |
@@ -260,6 +264,8 @@ The BEIR benchmarks — the secondary decision signal — favour Qwen3-Embedding
 
 The EvaluateRAG evaluation — the primary decision signal — remains pending. It is the only evaluation that directly measures retrieval quality on the actual AVAP corpus with its intra-chunk multilingual mixing. BGE-M3's architectural fit for multilingual content could still produce a measurable advantage on the production corpus that the English-only BEIR benchmarks cannot capture. No final model selection will be made until EvaluateRAG results are available for both candidates.
 
+We have found that Qwen3-embedding is multi-lingual, with good scores in multi-lingual benchmarks. The documentation says so, but the definitive answer will be provided by the scores of the evaluation on the AVAP corpus.
+
 ---
 
 ## Consequences

research/embeddings/cosqa/qrels/test.tsv

@@ -1,501 +0,0 @@
-query-id	corpus-id	score
-q20105	d20105	1
-q20106	d20106	1
-q20107	d20107	1
-q20108	d20108	1
-q20109	d20109	1
-q20110	d20110	1
-q20111	d20111	1
-q20112	d20112	1
-q20113	d20113	1
-q20114	d20114	1
-q20115	d20115	1
-q20116	d20116	1
-q20117	d20117	1
-q20118	d20118	1
-q20119	d20119	1
-q20120	d20120	1
-q20121	d20121	1
-q20122	d20122	1
-q20123	d20123	1
-q20124	d20124	1
-q20125	d20125	1
-q20126	d20126	1
-q20127	d20127	1
-q20128	d20128	1
-q20129	d20129	1
-q20130	d20130	1
-q20131	d20131	1
-q20132	d20132	1
-q20133	d20133	1
-q20134	d20134	1
-q20135	d20135	1
-q20136	d20136	1
-q20137	d20137	1
-q20138	d20138	1
-q20139	d20139	1
-q20140	d20140	1
-q20141	d20141	1
-q20142	d20142	1
-q20143	d20143	1
-q20144	d20144	1
-q20145	d20145	1
-q20146	d20146	1
-q20147	d20147	1
-q20148	d20148	1
-q20149	d20149	1
-q20150	d20150	1
-q20151	d20151	1
-q20152	d20152	1
-q20153	d20153	1
-q20154	d20154	1
-q20155	d20155	1
-q20156	d20156	1
-q20157	d20157	1
-q20158	d20158	1
-q20159	d20159	1
-q20160	d20160	1
-q20161	d20161	1
-q20162	d20162	1
-q20163	d20163	1
-q20164	d20164	1
-q20165	d20165	1
-q20166	d20166	1
-q20167	d20167	1
-q20168	d20168	1
-q20169	d20169	1
-q20170	d20170	1
-q20171	d20171	1
-q20172	d20172	1
-q20173	d20173	1
-q20174	d20174	1
-q20175	d20175	1
-q20176	d20176	1
-q20177	d20177	1
-q20178	d20178	1
-q20179	d20179	1
-q20180	d20180	1
-q20181	d20181	1
-q20182	d20182	1
-q20183	d20183	1
-q20184	d20184	1
-q20185	d20185	1
-q20186	d20186	1
-q20187	d20187	1
-q20188	d20188	1
-q20189	d20189	1
-q20190	d20190	1
-q20191	d20191	1
-q20192	d20192	1
-q20193	d20193	1
-q20194	d20194	1
-q20195	d20195	1
-q20196	d20196	1
-q20197	d20197	1
-q20198	d20198	1
-q20199	d20199	1
-q20200	d20200	1
-q20201	d20201	1
-q20202	d20202	1
-q20203	d20203	1
-q20204	d20204	1
-q20205	d20205	1
-q20206	d20206	1
-q20207	d20207	1
-q20208	d20208	1
-q20209	d20209	1
-q20210	d20210	1
-q20211	d20211	1
-q20212	d20212	1
-q20213	d20213	1
-q20214	d20214	1
-q20215	d20215	1
-q20216	d20216	1
-q20217	d20217	1
-q20218	d20218	1
-q20219	d20219	1
-q20220	d20220	1
-q20221	d20221	1
-q20222	d20222	1
-q20223	d20223	1
-q20224	d20224	1
-q20225	d20225	1
-q20226	d20226	1
-q20227	d20227	1
-q20228	d20228	1
-q20229	d20229	1
-q20230	d20230	1
-q20231	d20231	1
-q20232	d20232	1
-q20233	d20233	1
-q20234	d20234	1
-q20235	d20235	1
-q20236	d20236	1
-q20237	d20237	1
-q20238	d20238	1
-q20239	d20239	1
-q20240	d20240	1
-q20241	d20241	1
-q20242	d20242	1
-q20243	d20243	1
-q20244	d20244	1
-q20245	d20245	1
-q20246	d20246	1
-q20247	d20247	1
-q20248	d20248	1
-q20249	d20249	1
-q20250	d20250	1
-q20251	d20251	1
-q20252	d20252	1
-q20253	d20253	1
-q20254	d20254	1
-q20255	d20255	1
-q20256	d20256	1
-q20257	d20257	1
-q20258	d20258	1
-q20259	d20259	1
-q20260	d20260	1
-q20261	d20261	1
-q20262	d20262	1
-q20263	d20263	1
-q20264	d20264	1
-q20265	d20265	1
-q20266	d20266	1
-q20267	d20267	1
-q20268	d20268	1
-q20269	d20269	1
-q20270	d20270	1
-q20271	d20271	1
-q20272	d20272	1
-q20273	d20273	1
-q20274	d20274	1
-q20275	d20275	1
-q20276	d20276	1
-q20277	d20277	1
-q20278	d20278	1
-q20279	d20279	1
-q20280	d20280	1
-q20281	d20281	1
-q20282	d20282	1
-q20283	d20283	1
-q20284	d20284	1
-q20285	d20285	1
-q20286	d20286	1
-q20287	d20287	1
-q20288	d20288	1
-q20289	d20289	1
-q20290	d20290	1
-q20291	d20291	1
-q20292	d20292	1
-q20293	d20293	1
-q20294	d20294	1
-q20295	d20295	1
-q20296	d20296	1
-q20297	d20297	1
-q20298	d20298	1
-q20299	d20299	1
-q20300	d20300	1
-q20301	d20301	1
-q20302	d20302	1
-q20303	d20303	1
-q20304	d20304	1
-q20305	d20305	1
-q20306	d20306	1
-q20307	d20307	1
-q20308	d20308	1
-q20309	d20309	1
-q20310	d20310	1
-q20311	d20311	1
-q20312	d20312	1
-q20313	d20313	1
-q20314	d20314	1
-q20315	d20315	1
-q20316	d20316	1
-q20317	d20317	1
-q20318	d20318	1
-q20319	d20319	1
-q20320	d20320	1
-q20321	d20321	1
-q20322	d20322	1
-q20323	d20323	1
-q20324	d20324	1
-q20325	d20325	1
-q20326	d20326	1
-q20327	d20327	1
-q20328	d20328	1
-q20329	d20329	1
-q20330	d20330	1
-q20331	d20331	1
-q20332	d20332	1
-q20333	d20333	1
-q20334	d20334	1
-q20335	d20335	1
-q20336	d20336	1
-q20337	d20337	1
-q20338	d20338	1
-q20339	d20339	1
-q20340	d20340	1
-q20341	d20341	1
-q20342	d20342	1
-q20343	d20343	1
-q20344	d20344	1
-q20345	d20345	1
-q20346	d20346	1
-q20347	d20347	1
-q20348	d20348	1
-q20349	d20349	1
-q20350	d20350	1
-q20351	d20351	1
-q20352	d20352	1
-q20353	d20353	1
-q20354	d20354	1
-q20355	d20355	1
-q20356	d20356	1
-q20357	d20357	1
-q20358	d20358	1
-q20359	d20359	1
-q20360	d20360	1
-q20361	d20361	1
-q20362	d20362	1
-q20363	d20363	1
-q20364	d20364	1
-q20365	d20365	1
-q20366	d20366	1
-q20367	d20367	1
-q20368	d20368	1
-q20369	d20369	1
-q20370	d20370	1
-q20371	d20371	1
-q20372	d20372	1
-q20373	d20373	1
-q20374	d20374	1
-q20375	d20375	1
-q20376	d20376	1
-q20377	d20377	1
-q20378	d20378	1
-q20379	d20379	1
-q20380	d20380	1
-q20381	d20381	1
-q20382	d20382	1
-q20383	d20383	1
-q20384	d20384	1
-q20385	d20385	1
-q20386	d20386	1
-q20387	d20387	1
-q20388	d20388	1
-q20389	d20389	1
-q20390	d20390	1
-q20391	d20391	1
-q20392	d20392	1
-q20393	d20393	1
-q20394	d20394	1
-q20395	d20395	1
-q20396	d20396	1
-q20397	d20397	1
-q20398	d20398	1
-q20399	d20399	1
-q20400	d20400	1
-q20401	d20401	1
-q20402	d20402	1
-q20403	d20403	1
-q20404	d20404	1
-q20405	d20405	1
-q20406	d20406	1
-q20407	d20407	1
-q20408	d20408	1
-q20409	d20409	1
-q20410	d20410	1
-q20411	d20411	1
-q20412	d20412	1
-q20413	d20413	1
-q20414	d20414	1
-q20415	d20415	1
-q20416	d20416	1
-q20417	d20417	1
-q20418	d20418	1
-q20419	d20419	1
-q20420	d20420	1
-q20421	d20421	1
-q20422	d20422	1
-q20423	d20423	1
-q20424	d20424	1
-q20425	d20425	1
-q20426	d20426	1
-q20427	d20427	1
-q20428	d20428	1
-q20429	d20429	1
-q20430	d20430	1
-q20431	d20431	1
-q20432	d20432	1
-q20433	d20433	1
-q20434	d20434	1
-q20435	d20435	1
-q20436	d20436	1
-q20437	d20437	1
-q20438	d20438	1
-q20439	d20439	1
-q20440	d20440	1
-q20441	d20441	1
-q20442	d20442	1
-q20443	d20443	1
-q20444	d20444	1
-q20445	d20445	1
-q20446	d20446	1
-q20447	d20447	1
-q20448	d20448	1
-q20449	d20449	1
-q20450	d20450	1
-q20451	d20451	1
-q20452	d20452	1
-q20453	d20453	1
-q20454	d20454	1
-q20455	d20455	1
-q20456	d20456	1
-q20457	d20457	1
-q20458	d20458	1
-q20459	d20459	1
-q20460	d20460	1
-q20461	d20461	1
-q20462	d20462	1
-q20463	d20463	1
-q20464	d20464	1
-q20465	d20465	1
-q20466	d20466	1
-q20467	d20467	1
-q20468	d20468	1
-q20469	d20469	1
-q20470	d20470	1
-q20471	d20471	1
-q20472	d20472	1
-q20473	d20473	1
-q20474	d20474	1
-q20475	d20475	1
-q20476	d20476	1
-q20477	d20477	1
-q20478	d20478	1
-q20479	d20479	1
-q20480	d20480	1
-q20481	d20481	1
-q20482	d20482	1
-q20483	d20483	1
-q20484	d20484	1
-q20485	d20485	1
-q20486	d20486	1
-q20487	d20487	1
-q20488	d20488	1
-q20489	d20489	1
-q20490	d20490	1
-q20491	d20491	1
-q20492	d20492	1
-q20493	d20493	1
-q20494	d20494	1
-q20495	d20495	1
-q20496	d20496	1
-q20497	d20497	1
-q20498	d20498	1
-q20499	d20499	1
-q20500	d20500	1
-q20501	d20501	1
-q20502	d20502	1
-q20503	d20503	1
-q20504	d20504	1
-q20505	d20505	1
-q20506	d20506	1
-q20507	d20507	1
-q20508	d20508	1
-q20509	d20509	1
-q20510	d20510	1
-q20511	d20511	1
-q20512	d20512	1
-q20513	d20513	1
-q20514	d20514	1
-q20515	d20515	1
-q20516	d20516	1
-q20517	d20517	1
-q20518	d20518	1
-q20519	d20519	1
-q20520	d20520	1
-q20521	d20521	1
-q20522	d20522	1
-q20523	d20523	1
-q20524	d20524	1
-q20525	d20525	1
-q20526	d20526	1
-q20527	d20527	1
-q20528	d20528	1
-q20529	d20529	1
-q20530	d20530	1
-q20531	d20531	1
-q20532	d20532	1
-q20533	d20533	1
-q20534	d20534	1
-q20535	d20535	1
-q20536	d20536	1
-q20537	d20537	1
-q20538	d20538	1
-q20539	d20539	1
-q20540	d20540	1
-q20541	d20541	1
-q20542	d20542	1
-q20543	d20543	1
-q20544	d20544	1
-q20545	d20545	1
-q20546	d20546	1
-q20547	d20547	1
-q20548	d20548	1
-q20549	d20549	1
-q20550	d20550	1
-q20551	d20551	1
-q20552	d20552	1
-q20553	d20553	1
-q20554	d20554	1
-q20555	d20555	1
-q20556	d20556	1
-q20557	d20557	1
-q20558	d20558	1
-q20559	d20559	1
-q20560	d20560	1
-q20561	d20561	1
-q20562	d20562	1
-q20563	d20563	1
-q20564	d20564	1
-q20565	d20565	1
-q20566	d20566	1
-q20567	d20567	1
-q20568	d20568	1
-q20569	d20569	1
-q20570	d20570	1
-q20571	d20571	1
-q20572	d20572	1
-q20573	d20573	1
-q20574	d20574	1
-q20575	d20575	1
-q20576	d20576	1
-q20577	d20577	1
-q20578	d20578	1
-q20579	d20579	1
-q20580	d20580	1
-q20581	d20581	1
-q20582	d20582	1
-q20583	d20583	1
-q20584	d20584	1
-q20585	d20585	1
-q20586	d20586	1
-q20587	d20587	1
-q20588	d20588	1
-q20589	d20589	1
-q20590	d20590	1
-q20591	d20591	1
-q20592	d20592	1
-q20593	d20593	1
-q20594	d20594	1
-q20595	d20595	1
-q20596	d20596	1
-q20597	d20597	1
-q20598	d20598	1
-q20599	d20599	1
-q20600	d20600	1
-q20601	d20601	1
-q20602	d20602	1
-q20603	d20603	1
-q20604	d20604	1

research/embeddings/n00 Beir Analysis CodeXGlue.ipynb

@@ -1,333 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "66cbbaf8",
-   "metadata": {},
-   "source": [
-    "# Libraries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "c01c19dc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import Dict, List, Union\n",
-    "import numpy as np\n",
-    "from langchain_ollama import OllamaEmbeddings\n",
-    "from beir.datasets.data_loader import GenericDataLoader\n",
-    "from beir.retrieval.search.dense import DenseRetrievalExactSearch\n",
-    "from beir.retrieval.evaluation import EvaluateRetrieval\n",
-    "from beir import util\n",
-    "import json\n",
-    "from datasets import load_dataset"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ac011c1c",
-   "metadata": {},
-   "source": [
-    "# Utils"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "b83e7900",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class BEIROllamaEmbeddings:\n",
-    "    \"\"\"\n",
-    "    Adapter that makes LangChain's OllamaEmbeddings compatible with BEIR.\n",
-    "    \"\"\"\n",
-    "\n",
-    "    def __init__(\n",
-    "        self,\n",
-    "        base_url: str,\n",
-    "        model: str,\n",
-    "        batch_size: int = 64,\n",
-    "    ) -> None:\n",
-    "        self.batch_size = batch_size\n",
-    "        self.embeddings = OllamaEmbeddings(\n",
-    "            base_url=base_url,\n",
-    "            model=model,\n",
-    "        )\n",
-    "\n",
-    "    def _batch_embed(self, texts: List[str]) -> np.ndarray:\n",
-    "        vectors = []\n",
-    "\n",
-    "        for i in range(0, len(texts), self.batch_size):\n",
-    "            batch = texts[i : i + self.batch_size]\n",
-    "            batch_vectors = self.embeddings.embed_documents(batch)\n",
-    "            vectors.extend(batch_vectors)\n",
-    "\n",
-    "        return np.asarray(vectors, dtype=np.float32)\n",
-    "\n",
-    "    def encode_queries(self, queries: List[str], **kwargs) -> np.ndarray:\n",
-    "        \"\"\"\n",
-    "        BEIR query encoder\n",
-    "        \"\"\"\n",
-    "        return self._batch_embed(queries)\n",
-    "\n",
-    "    def encode_corpus(\n",
-    "        self,\n",
-    "        corpus: Union[List[Dict[str, str]], Dict[str, Dict[str, str]]],\n",
-    "        **kwargs,\n",
-    "    ) -> np.ndarray:\n",
-    "        \"\"\"\n",
-    "        BEIR corpus encoder\n",
-    "        \"\"\"\n",
-    "        if isinstance(corpus, dict):\n",
-    "            corpus = list(corpus.values())\n",
-    "\n",
-    "        texts = []\n",
-    "        for doc in corpus:\n",
-    "            title = (doc.get(\"title\") or \"\").strip()\n",
-    "            text = (doc.get(\"text\") or \"\").strip()\n",
-    "\n",
-    "            if title:\n",
-    "                texts.append(f\"{title}\\n{text}\")\n",
-    "            else:\n",
-    "                texts.append(text)\n",
-    "\n",
-    "        return self._batch_embed(texts)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "af3eb66d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def convert_hf_to_beir(hf_dataset):\n",
-    "    corpus, queries, qrels = {}, {}, {}\n",
-    "    \n",
-    "    for i, data in enumerate(hf_dataset):\n",
-    "        docid = f\"doc_{i}\"\n",
-    "        queryid = f\"q_{i}\"\n",
-    "        \n",
-    "        # El código es el documento (lo que el agente debe recuperar)\n",
-    "        corpus[docid] = {\"title\": data.get(\"func_name\", \"\"), \"text\": data['code']}\n",
-    "        \n",
-    "        # El docstring es la consulta (lo que el usuario pide)\n",
-    "        queries[queryid] = data['docstring']\n",
-    "        \n",
-    "        # Relación 1 a 1: la query i busca el código i\n",
-    "        qrels[queryid] = {docid: 1}\n",
-    "            \n",
-    "    return corpus, queries, qrels"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c9528fb6",
-   "metadata": {},
-   "source": [
-    "# Data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "id": "230aae25",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "raw_dataset = load_dataset(\"google/code_x_glue_tc_nl_code_search_adv\", split=\"test\")\n",
-    "corpus, queries, qrels = convert_hf_to_beir(raw_dataset)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "13050d31",
-   "metadata": {},
-   "source": [
-    "# Test qwen3-0.6B-emb:latest"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "514540af",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model = BEIROllamaEmbeddings(\n",
-    "    base_url=\"http://localhost:11434\",\n",
-    "    model=\"qwen3-0.6B-emb:latest\",\n",
-    "    batch_size=64,\n",
-    ")\n",
-    "\n",
-    "# Inicializar buscador y evaluador\n",
-    "retriever = DenseRetrievalExactSearch(model, batch_size=64)\n",
-    "evaluator = EvaluateRetrieval(retriever, score_function=\"cos_sim\")\n",
-    "\n",
-    "# Ejecutar recuperación\n",
-    "results = evaluator.retrieve(corpus, queries)\n",
-    "\n",
-    "# Evaluar métricas (NDCG, MAP, Recall, Precision)\n",
-    "ndcg, _map, recall, precision = evaluator.evaluate(\n",
-    "    qrels, results, [1, 3, 5, 10]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "5c0f9845",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Resultados para CodeXGLUE:\n",
-      "NDCG: {'NDCG@1': 0.94971, 'NDCG@3': 0.96956, 'NDCG@5': 0.97166, 'NDCG@10': 0.97342}\n",
-      "MAP: {'MAP@1': 0.94971, 'MAP@3': 0.96504, 'MAP@5': 0.9662, 'MAP@10': 0.96694}\n",
-      "Recall: {'Recall@1': 0.94971, 'Recall@3': 0.98251, 'Recall@5': 0.98761, 'Recall@10': 0.99297}\n",
-      "Precision: {'P@1': 0.94971, 'P@3': 0.3275, 'P@5': 0.19752, 'P@10': 0.0993}\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(f\"Resultados para CodeXGLUE:\")\n",
-    "print(\"NDCG:\", ndcg)\n",
-    "print(\"MAP:\", _map)\n",
-    "print(\"Recall:\", recall)\n",
-    "print(\"Precision:\", precision)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c4e643ca",
-   "metadata": {},
-   "source": [
-    "# Test qwen2.5:1.5b"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "5ced1c25",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "model_q2 = BEIROllamaEmbeddings(\n",
-    "    base_url=\"http://localhost:11434\",\n",
-    "    model=\"qwen2.5:1.5b\",\n",
-    "    batch_size=64,\n",
-    ")\n",
-    "\n",
-    "# Inicializar buscador y evaluador\n",
-    "retriever_q2 = DenseRetrievalExactSearch(model_q2, batch_size=64)\n",
-    "evaluator_q2 = EvaluateRetrieval(retriever_q2, score_function=\"cos_sim\")\n",
-    "\n",
-    "# Ejecutar recuperación\n",
-    "results_q2 = evaluator_q2.retrieve(corpus, queries)\n",
-    "\n",
-    "# Evaluar métricas (NDCG, MAP, Recall, Precision)\n",
-    "ndcg_qwen_2, _map_qwen_2, recall_qwen_2, precision_qwen_2 = evaluator_q2.evaluate(\n",
-    "    qrels, results_q2, [1, 3, 5, 10]\n",
-    ")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "id": "6a95189e",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Resultados para CodeXGLUE:\n",
-      "NDCG: {'NDCG@1': 0.00031, 'NDCG@3': 0.00061, 'NDCG@5': 0.00086, 'NDCG@10': 0.00118}\n",
-      "MAP: {'MAP@1': 0.00031, 'MAP@3': 0.00051, 'MAP@5': 0.00065, 'MAP@10': 0.00078}\n",
-      "Recall: {'Recall@1': 0.00031, 'Recall@3': 0.00088, 'Recall@5': 0.00151, 'Recall@10': 0.0025}\n",
-      "Precision: {'P@1': 0.00031, 'P@3': 0.00029, 'P@5': 0.0003, 'P@10': 0.00025}\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(f\"Resultados para CodeXGLUE:\")\n",
-    "print(\"NDCG:\", ndcg_qwen_2)\n",
-    "print(\"MAP:\", _map_qwen_2)\n",
-    "print(\"Recall:\", recall_qwen_2)\n",
-    "print(\"Precision:\", precision_qwen_2)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "3dad9811",
-   "metadata": {},
-   "source": [
-    "# Save data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "f875dd8d",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Resultados guardados en /home/pseco/VsCodeProjects/assistance-engine/data/interim/beir_CodeXGlue_results.json\n"
-     ]
-    }
-   ],
-   "source": [
-    "results_data = {\n",
-    "    \"qwen3-0.6B-emb:latest\": {\n",
-    "        \"NDCG\": ndcg,\n",
-    "        \"MAP\": _map,\n",
-    "        \"Recall\": recall,\n",
-    "        \"Precision\": precision,\n",
-    "    },\n",
-    "    \"qwen2.5:1.5b\": {\n",
-    "        \"NDCG\": ndcg_qwen_2,\n",
-    "        \"MAP\": _map_qwen_2,\n",
-    "        \"Recall\": recall_qwen_2,\n",
-    "        \"Precision\": precision_qwen_2,\n",
-    "    }\n",
-    "}\n",
-    "\n",
-    "output_file = \"/home/pseco/VsCodeProjects/assistance-engine/data/interim/beir_CodeXGlue_results.json\"\n",
-    "with open(output_file, \"w\") as f:\n",
-    "    json.dump(results_data, f, indent=2)\n",
-    "\n",
-    "print(f\"Resultados guardados en {output_file}\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "assistance-engine",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.12.11"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

research/embeddings/n00 Beir Analysis.ipynb

@@ -1,323 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "66cbbaf8",
-   "metadata": {},
-   "source": [
-    "# Libraries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 15,
-   "id": "c01c19dc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from typing import Dict, List, Union\n",
-    "import numpy as np\n",
-    "from langchain_ollama import OllamaEmbeddings\n",
-    "from beir.datasets.data_loader import GenericDataLoader\n",
-    "from beir.retrieval.search.dense import DenseRetrievalExactSearch\n",
-    "from beir.retrieval.evaluation import EvaluateRetrieval\n",
-    "from beir import util\n",
-    "import json"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ac011c1c",
-   "metadata": {},
-   "source": [
-    "# Utils"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "id": "b83e7900",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class BEIROllamaEmbeddings:\n",
-    "    \"\"\"\n",
-    "    Adapter that makes LangChain's OllamaEmbeddings compatible with BEIR.\n",
-    "    \"\"\"\n",
-    "\n",
-    "    def __init__(\n",
-    "        self,\n",
-    "        base_url: str,\n",
-    "        model: str,\n",
-    "        batch_size: int = 64,\n",
-    "    ) -> None:\n",
-    "        self.batch_size = batch_size\n",
-    "        self.embeddings = OllamaEmbeddings(\n",
-    "            base_url=base_url,\n",
-    "            model=model,\n",
-    "        )\n",
-    "\n",
-    "    def _batch_embed(self, texts: List[str]) -> np.ndarray:\n",
-    "        vectors = []\n",
-    "\n",
-    "        for i in range(0, len(texts), self.batch_size):\n",
-    "            batch = texts[i : i + self.batch_size]\n",
-    "            batch_vectors = self.embeddings.embed_documents(batch)\n",
-    "            vectors.extend(batch_vectors)\n",
-    "\n",
-    "        return np.asarray(vectors, dtype=np.float32)\n",
-    "\n",
-    "    def encode_queries(self, queries: List[str], **kwargs) -> np.ndarray:\n",
-    "        \"\"\"\n",
-    "        BEIR query encoder\n",
-    "        \"\"\"\n",
-    "        return self._batch_embed(queries)\n",
-    "\n",
-    "    def encode_corpus(\n",
-    "        self,\n",
-    "        corpus: Union[List[Dict[str, str]], Dict[str, Dict[str, str]]],\n",
-    "        **kwargs,\n",
-    "    ) -> np.ndarray:\n",
-    "        \"\"\"\n",
-    "        BEIR corpus encoder\n",
-    "        \"\"\"\n",
-    "        if isinstance(corpus, dict):\n",
-    "            corpus = list(corpus.values())\n",
-    "\n",
-    "        texts = []\n",
-    "        for doc in corpus:\n",
-    "            title = (doc.get(\"title\") or \"\").strip()\n",
-    "            text = (doc.get(\"text\") or \"\").strip()\n",
-    "\n",
-    "            if title:\n",
-    "                texts.append(f\"{title}\\n{text}\")\n",
-    "            else:\n",
-    "                texts.append(text)\n",
-    "\n",
-    "        return self._batch_embed(texts)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "af3eb66d",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "def convert_codexglue_to_beir(input_file):\n",
-    "    corpus, queries, qrels = {}, {}, {}\n",
-    "    with open(input_file, 'r') as f:\n",
-    "        for i, line in enumerate(f):\n",
-    "            data = json.loads(line)\n",
-    "            docid = f\"doc_{i}\"\n",
-    "            queryid = f\"q_{i}\"\n",
-    "            \n",
-    "            # El código es nuestro documento (Corpus)\n",
-    "            corpus[docid] = {\"title\": \"\", \"text\": data['code']}\n",
-    "            # El docstring es nuestra consulta (Query)\n",
-    "            queries[queryid] = data['docstring']\n",
-    "            # En CodeXGLUE, la consulta i corresponde al código i\n",
-    "            qrels[queryid] = {docid: 1}\n",
-    "            \n",
-    "    return corpus, queries, qrels\n",
-    "\n",
-    "# Carga tus datos (ejemplo con el set de test de AdvTest)\n",
-    "corpus, queries, qrels = convert_codexglue_to_beir(\"test.jsonl\")\n"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c9528fb6",
-   "metadata": {},
-   "source": [
-    "# Data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "id": "230aae25",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "1915c67ec20f4806b30b48eff9a132e2",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "  0%|          | 0/5183 [00:00<?, ?it/s]"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
-   "source": [
-    "dataset=\"scifact\"\n",
-    "url=f\"https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{dataset}.zip\"\n",
-    "data_path=util.download_and_unzip(url, out_dir=\"datasets\")\n",
-    "corpus, queries, qrels=GenericDataLoader(data_path).load(split=\"test\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "13050d31",
-   "metadata": {},
-   "source": [
-    "# Test qwen3-0.6B-emb:latest"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "514540af",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "NDCG: {'NDCG@1': 0.56333, 'NDCG@3': 0.64367, 'NDCG@5': 0.66577, 'NDCG@10': 0.68551, 'NDCG@100': 0.71285}\n",
-      "MAP: {'MAP@1': 0.52994, 'MAP@3': 0.6117, 'MAP@5': 0.62815, 'MAP@10': 0.6383, 'MAP@100': 0.64466}\n",
-      "Recall: {'Recall@1': 0.52994, 'Recall@3': 0.7035, 'Recall@5': 0.75967, 'Recall@10': 0.81611, 'Recall@100': 0.94}\n",
-      "Precision: {'P@1': 0.56333, 'P@3': 0.25889, 'P@5': 0.17067, 'P@10': 0.093, 'P@100': 0.0107}\n"
-     ]
-    }
-   ],
-   "source": [
-    "model = BEIROllamaEmbeddings(\n",
-    "    base_url=\"http://localhost:11434\",\n",
-    "    model=\"qwen3-0.6B-emb:latest\",\n",
-    "    batch_size=64,\n",
-    ")\n",
-    "\n",
-    "retriever = DenseRetrievalExactSearch(model, batch_size=64)\n",
-    "evaluator = EvaluateRetrieval(retriever, score_function=\"cos_sim\")\n",
-    "\n",
-    "results = evaluator.retrieve(corpus, queries)\n",
-    "ndcg, _map, recall, precision = evaluator.evaluate(\n",
-    "    qrels, results, [1, 3, 5, 10, 100]\n",
-    ")\n",
-    "\n",
-    "print(\"NDCG:\", ndcg)\n",
-    "print(\"MAP:\", _map)\n",
-    "print(\"Recall:\", recall)\n",
-    "print(\"Precision:\", precision)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c4e643ca",
-   "metadata": {},
-   "source": [
-    "# Test qwen2.5:1.5b"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "5ced1c25",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "NDCG: {'NDCG@1': 0.02333, 'NDCG@3': 0.03498, 'NDCG@5': 0.0404, 'NDCG@10': 0.04619, 'NDCG@100': 0.07768}\n",
-      "MAP: {'MAP@1': 0.02083, 'MAP@3': 0.03083, 'MAP@5': 0.03375, 'MAP@10': 0.03632, 'MAP@100': 0.04123}\n",
-      "Recall: {'Recall@1': 0.02083, 'Recall@3': 0.04417, 'Recall@5': 0.0575, 'Recall@10': 0.07417, 'Recall@100': 0.23144}\n",
-      "Precision: {'P@1': 0.02333, 'P@3': 0.01556, 'P@5': 0.01267, 'P@10': 0.00833, 'P@100': 0.00277}\n"
-     ]
-    }
-   ],
-   "source": [
-    "model_qwen2 = BEIROllamaEmbeddings(\n",
-    "    base_url=\"http://localhost:11434\",\n",
-    "    model=\"qwen2.5:1.5b\",\n",
-    "    batch_size=64,\n",
-    ")\n",
-    "\n",
-    "retriever_qwen_2 = DenseRetrievalExactSearch(model_qwen2, batch_size=64)\n",
-    "evaluator_qwen_2 = EvaluateRetrieval(retriever_qwen_2, score_function=\"cos_sim\")\n",
-    "\n",
-    "results_qwen_2 = evaluator_qwen_2.retrieve(corpus, queries)\n",
-    "ndcg_qwen_2, _map_qwen_2, recall_qwen_2, precision_qwen_2 = evaluator_qwen_2.evaluate(\n",
-    "    qrels, results_qwen_2, [1, 3, 5, 10, 100]\n",
-    ")\n",
-    "\n",
-    "print(\"NDCG:\", ndcg_qwen_2)\n",
-    "print(\"MAP:\", _map_qwen_2)\n",
-    "print(\"Recall:\", recall_qwen_2)\n",
-    "print(\"Precision:\", precision_qwen_2)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "b9402837",
-   "metadata": {},
-   "source": [
-    "# Save Data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 16,
-   "id": "c281d5e1",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Resultados guardados en /home/pseco/VsCodeProjects/assistance-engine/data/interim/beir_Scifact_results.json\n"
-     ]
-    }
-   ],
-   "source": [
-    "results_data = {\n",
-    "    \"qwen3-0.6B-emb:latest\": {\n",
-    "        \"NDCG\": ndcg,\n",
-    "        \"MAP\": _map,\n",
-    "        \"Recall\": recall,\n",
-    "        \"Precision\": precision,\n",
-    "    },\n",
-    "    \"qwen2.5:1.5b\": {\n",
-    "        \"NDCG\": ndcg_qwen_2,\n",
-    "        \"MAP\": _map_qwen_2,\n",
-    "        \"Recall\": recall_qwen_2,\n",
-    "        \"Precision\": precision_qwen_2,\n",
-    "    }\n",
-    "}\n",
-    "\n",
-    "output_file = \"/home/pseco/VsCodeProjects/assistance-engine/data/interim/beir_Scifact_results.json\"\n",
-    "with open(output_file, \"w\") as f:\n",
-    "    json.dump(results_data, f, indent=2)\n",
-    "\n",
-    "print(f\"Resultados guardados en {output_file}\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "assistance-engine",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.12.11"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

research/embeddings/n00 Beir Analysis_cosqa.ipynb

@@ -1,335 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "66cbbaf8",
-   "metadata": {},
-   "source": [
-    "# Libraries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "id": "c01c19dc",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "import os\n",
-    "import json\n",
-    "from typing import Dict, List, Union\n",
-    "import numpy as np\n",
-    "from datasets import load_dataset\n",
-    "from langchain_ollama import OllamaEmbeddings\n",
-    "from beir.datasets.data_loader import GenericDataLoader\n",
-    "from beir.retrieval.search.dense import DenseRetrievalExactSearch\n",
-    "from beir.retrieval.evaluation import EvaluateRetrieval\n",
-    "from beir import util"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "ac011c1c",
-   "metadata": {},
-   "source": [
-    "# Utils"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "b83e7900",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "class BEIROllamaEmbeddings:\n",
-    "    \"\"\"\n",
-    "    Adapter that makes LangChain's OllamaEmbeddings compatible with BEIR.\n",
-    "    \"\"\"\n",
-    "\n",
-    "    def __init__(\n",
-    "        self,\n",
-    "        base_url: str,\n",
-    "        model: str,\n",
-    "        batch_size: int = 64,\n",
-    "    ) -> None:\n",
-    "        self.batch_size = batch_size\n",
-    "        self.embeddings = OllamaEmbeddings(\n",
-    "            base_url=base_url,\n",
-    "            model=model,\n",
-    "        )\n",
-    "\n",
-    "    def _batch_embed(self, texts: List[str]) -> np.ndarray:\n",
-    "        vectors = []\n",
-    "\n",
-    "        for i in range(0, len(texts), self.batch_size):\n",
-    "            batch = texts[i : i + self.batch_size]\n",
-    "            batch_vectors = self.embeddings.embed_documents(batch)\n",
-    "            vectors.extend(batch_vectors)\n",
-    "\n",
-    "        return np.asarray(vectors, dtype=np.float32)\n",
-    "\n",
-    "    def encode_queries(self, queries: List[str], **kwargs) -> np.ndarray:\n",
-    "        \"\"\"\n",
-    "        BEIR query encoder\n",
-    "        \"\"\"\n",
-    "        return self._batch_embed(queries)\n",
-    "\n",
-    "    def encode_corpus(\n",
-    "        self,\n",
-    "        corpus: Union[List[Dict[str, str]], Dict[str, Dict[str, str]]],\n",
-    "        **kwargs,\n",
-    "    ) -> np.ndarray:\n",
-    "        \"\"\"\n",
-    "        BEIR corpus encoder\n",
-    "        \"\"\"\n",
-    "        if isinstance(corpus, dict):\n",
-    "            corpus = list(corpus.values())\n",
-    "\n",
-    "        texts = []\n",
-    "        for doc in corpus:\n",
-    "            title = (doc.get(\"title\") or \"\").strip()\n",
-    "            text = (doc.get(\"text\") or \"\").strip()\n",
-    "\n",
-    "            if title:\n",
-    "                texts.append(f\"{title}\\n{text}\")\n",
-    "            else:\n",
-    "                texts.append(text)\n",
-    "\n",
-    "        return self._batch_embed(texts)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c9528fb6",
-   "metadata": {},
-   "source": [
-    "# Data"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "230aae25",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Descargando datos de Hugging Face...\n",
-      "Cargando con BEIR GenericDataLoader...\n"
-     ]
-    },
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "0e67479e959248f598db3415efbb13ae",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "  0%|          | 0/20604 [00:00<?, ?it/s]"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
-   "source": [
-    "dataset_name = \"cosqa\"\n",
-    "data_path = f\"/home/acano/PycharmProjects/assistance-engine/data/external/{dataset_name}\"\n",
-    "\n",
-    "os.makedirs(f\"{data_path}/qrels\", exist_ok=True)\n",
-    "\n",
-    "# 1. Cargar desde Hugging Face con los nombres de configuración correctos\n",
-    "print(\"Descargando datos de Hugging Face...\")\n",
-    "hf_corpus = load_dataset(\"CoIR-Retrieval/cosqa\", \"corpus\", split=\"corpus\")\n",
-    "hf_queries = load_dataset(\"CoIR-Retrieval/cosqa\", \"queries\", split=\"queries\")\n",
-    "# Los qrels están en la config 'default'\n",
-    "hf_qrels = load_dataset(\"CoIR-Retrieval/cosqa\", \"default\", split=\"test\")\n",
-    "\n",
-    "# 2. Guardar Corpus\n",
-    "with open(f\"{data_path}/corpus.jsonl\", \"w\") as f:\n",
-    "    for item in hf_corpus:\n",
-    "        f.write(json.dumps({\"_id\": str(item[\"_id\"]), \"text\": item[\"text\"], \"title\": \"\"}) + \"\\n\")\n",
-    "\n",
-    "# 3. Guardar Queries\n",
-    "with open(f\"{data_path}/queries.jsonl\", \"w\") as f:\n",
-    "    for item in hf_queries:\n",
-    "        f.write(json.dumps({\"_id\": str(item[\"_id\"]), \"text\": item[\"text\"]}) + \"\\n\")\n",
-    "\n",
-    "# 4. Guardar Qrels (Formato TSV para BEIR)\n",
-    "with open(f\"{data_path}/qrels/test.tsv\", \"w\") as f:\n",
-    "    f.write(\"query-id\\tcorpus-id\\tscore\\n\")\n",
-    "    for item in hf_qrels:\n",
-    "        # En la config 'default', los campos suelen ser 'query-id' y 'corpus-id'\n",
-    "        f.write(f\"{item['query-id']}\\t{item['corpus-id']}\\t{item['score']}\\n\")\n",
-    "\n",
-    "print(\"Cargando con BEIR GenericDataLoader...\")\n",
-    "corpus, queries, qrels = GenericDataLoader(data_path).load(split=\"test\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "13050d31",
-   "metadata": {},
-   "source": [
-    "# Test qwen3-0.6B-emb:latest"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "id": "514540af",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "NDCG: {'NDCG@1': 0.174, 'NDCG@3': 0.27374, 'NDCG@5': 0.33509, 'NDCG@10': 0.39086, 'NDCG@100': 0.45099}\n",
-      "MAP: {'MAP@1': 0.174, 'MAP@3': 0.247, 'MAP@5': 0.2808, 'MAP@10': 0.30466, 'MAP@100': 0.31702}\n",
-      "Recall: {'Recall@1': 0.174, 'Recall@3': 0.352, 'Recall@5': 0.502, 'Recall@10': 0.67, 'Recall@100': 0.952}\n",
-      "Precision: {'P@1': 0.174, 'P@3': 0.11733, 'P@5': 0.1004, 'P@10': 0.067, 'P@100': 0.00952}\n"
-     ]
-    }
-   ],
-   "source": [
-    "model = BEIROllamaEmbeddings(\n",
-    "    base_url=\"http://localhost:11434\",\n",
-    "    model=\"qwen3-0.6B-emb:latest\",\n",
-    "    batch_size=64,\n",
-    ")\n",
-    "\n",
-    "retriever = DenseRetrievalExactSearch(model, batch_size=64)\n",
-    "evaluator = EvaluateRetrieval(retriever, score_function=\"cos_sim\")\n",
-    "\n",
-    "results = evaluator.retrieve(corpus, queries)\n",
-    "ndcg, _map, recall, precision = evaluator.evaluate(\n",
-    "    qrels, results, [1, 3, 5, 10, 100]\n",
-    ")\n",
-    "\n",
-    "print(\"NDCG:\", ndcg)\n",
-    "print(\"MAP:\", _map)\n",
-    "print(\"Recall:\", recall)\n",
-    "print(\"Precision:\", precision)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "c4e643ca",
-   "metadata": {},
-   "source": [
-    "# Test qwen2.5:1.5b"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "id": "5ced1c25",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "NDCG: {'NDCG@1': 0.0, 'NDCG@3': 0.0, 'NDCG@5': 0.0, 'NDCG@10': 0.0, 'NDCG@100': 0.0021}\n",
-      "MAP: {'MAP@1': 0.0, 'MAP@3': 0.0, 'MAP@5': 0.0, 'MAP@10': 0.0, 'MAP@100': 0.00043}\n",
-      "Recall: {'Recall@1': 0.0, 'Recall@3': 0.0, 'Recall@5': 0.0, 'Recall@10': 0.0, 'Recall@100': 0.01}\n",
-      "Precision: {'P@1': 0.0, 'P@3': 0.0, 'P@5': 0.0, 'P@10': 0.0, 'P@100': 0.0001}\n"
-     ]
-    }
-   ],
-   "source": [
-    "model_qwen2 = BEIROllamaEmbeddings(\n",
-    "    base_url=\"http://localhost:11434\",\n",
-    "    model=\"qwen2.5:1.5b\",\n",
-    "    batch_size=64,\n",
-    ")\n",
-    "\n",
-    "retriever_qwen_2 = DenseRetrievalExactSearch(model_qwen2, batch_size=64)\n",
-    "evaluator_qwen_2 = EvaluateRetrieval(retriever_qwen_2, score_function=\"cos_sim\")\n",
-    "\n",
-    "results_qwen_2 = evaluator_qwen_2.retrieve(corpus, queries)\n",
-    "ndcg_qwen_2, _map_qwen_2, recall_qwen_2, precision_qwen_2 = evaluator_qwen_2.evaluate(\n",
-    "    qrels, results_qwen_2, [1, 3, 5, 10, 100]\n",
-    ")\n",
-    "\n",
-    "print(\"NDCG:\", ndcg_qwen_2)\n",
-    "print(\"MAP:\", _map_qwen_2)\n",
-    "print(\"Recall:\", recall_qwen_2)\n",
-    "print(\"Precision:\", precision_qwen_2)"
-   ]
-<<<<<<< HEAD
-=======
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "id": "1db7d110",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Resultados guardados en /home/acano/PycharmProjects/assistance-engine/data/interim/beir_cosqa_results.json\n"
-     ]
-    }
-   ],
-   "source": [
-    "results_data = {\n",
-    "    \"qwen3-0.6B-emb:latest\": {\n",
-    "        \"NDCG\": ndcg,\n",
-    "        \"MAP\": _map,\n",
-    "        \"Recall\": recall,\n",
-    "        \"Precision\": precision,\n",
-    "    },\n",
-    "    \"qwen2.5:1.5b\": {\n",
-    "        \"NDCG\": ndcg_qwen_2,\n",
-    "        \"MAP\": _map_qwen_2,\n",
-    "        \"Recall\": recall_qwen_2,\n",
-    "        \"Precision\": precision_qwen_2,\n",
-    "    }\n",
-    "}\n",
-    " \n",
-    "output_file = \"/home/acano/PycharmProjects/assistance-engine/data/interim/beir_cosqa_results.json\"\n",
-    "with open(output_file, \"w\") as f:\n",
-    "    json.dump(results_data, f, indent=2)\n",
-    " \n",
-    "print(f\"Resultados guardados en {output_file}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "e4f8d78b",
-   "metadata": {},
-   "outputs": [],
-   "source": []
->>>>>>> 4b5352d93cf89b7562895b550fb5bd62160586c5
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "assistance-engine",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.11.13"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

research/embeddings/n00 first Analysis.ipynb

@@ -1,289 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "id": "096e6224",
-   "metadata": {},
-   "source": [
-    "# Libraries"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "id": "4b0853e9",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/tmp/ipykernel_2931729/1845255288.py:4: DeprecationWarning: Importing SemanticSimilarity from 'ragas.metrics' is deprecated and will be removed in v1.0. Please use 'ragas.metrics.collections' instead. Example: from ragas.metrics.collections import SemanticSimilarity\n",
-      "  from ragas.metrics import SemanticSimilarity\n"
-     ]
-    }
-   ],
-   "source": [
-    "# ...existing code...\n",
-    "from datasets import load_dataset\n",
-    "from ragas import EvaluationDataset, evaluate\n",
-    "from ragas.metrics import SemanticSimilarity\n",
-    "from langchain_community.embeddings import OllamaEmbeddings\n",
-    "import asyncio\n",
-    "from typing import Sequence\n",
-    "from ragas.embeddings.base import BaseRagasEmbedding\n",
-    "import os\n",
-    "from transformers import AutoConfig\n",
-    "import nltk\n",
-    "# ...existing code..."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "id": "6bfe1ca0",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "nltk.download(\"punkt\", quiet=True)\n",
-    "\n",
-    "ES_URL = os.getenv(\"ELASTICSEARCH_LOCAL_URL\")\n",
-    "ES_INDEX_NAME = os.getenv(\"ELASTICSEARCH_INDEX\")\n",
-    "HF_EMBEDDING_MODEL_NAME = os.getenv(\"HF_EMBEDDING_MODEL_NAME\")\n",
-    "BASE_URL = os.getenv(\"LLM_BASE_LOCAL_URL\")\n",
-    "MODEL_NAME = os.getenv(\"OLLAMA_MODEL_NAME\")\n",
-    "\n",
-    "config = AutoConfig.from_pretrained(HF_EMBEDDING_MODEL_NAME)\n",
-    "embedding_dim = config.hidden_size"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "id": "ea41ce0f",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "/tmp/ipykernel_2931729/256987240.py:1: LangChainDeprecationWarning: The class `OllamaEmbeddings` was deprecated in LangChain 0.3.1 and will be removed in 1.0.0. An updated version of the class exists in the `langchain-ollama package and should be used instead. To use it run `pip install -U `langchain-ollama` and import as `from `langchain_ollama import OllamaEmbeddings``.\n",
-      "  embeddings = OllamaEmbeddings(base_url=BASE_URL, model=MODEL_NAME)\n"
-     ]
-    }
-   ],
-   "source": [
-    "embeddings = OllamaEmbeddings(base_url=BASE_URL, model=MODEL_NAME)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "8eee9390",
-   "metadata": {},
-   "source": [
-    "# Similitud Aleatoria"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "d7b150e5",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from datasets import load_dataset\n",
-    "from ragas import EvaluationDataset\n",
-    "\n",
-    "\n",
-    "def _normalize_answer(answer_value: object) -> str:\n",
-    "    \"\"\"\n",
-    "    Normalize answer values to a single string.\n",
-    "    \"\"\"\n",
-    "    if isinstance(answer_value, dict):\n",
-    "        text_value = answer_value.get(\"text\")\n",
-    "        if isinstance(text_value, list):\n",
-    "            return str(text_value[0]) if text_value else \"\"\n",
-    "        if text_value is not None:\n",
-    "            return str(text_value)\n",
-    "\n",
-    "    if isinstance(answer_value, list):\n",
-    "        return str(answer_value[0]) if answer_value else \"\"\n",
-    "\n",
-    "    return str(answer_value)\n",
-    "\n",
-    "\n",
-    "def _first_existing_key(candidates: list[str], keys: set[str]) -> str | None:\n",
-    "    \"\"\"\n",
-    "    Return the first key present in keys from candidates.\n",
-    "    \"\"\"\n",
-    "    for candidate in candidates:\n",
-    "        if candidate in keys:\n",
-    "            return candidate\n",
-    "    return None\n",
-    "\n",
-    "\n",
-    "ds = load_dataset(\"sentence-transformers/natural-questions\")\n",
-    "train_ds = ds[\"train\"]\n",
-    "\n",
-    "max_questions = min(100, len(train_ds))\n",
-    "train_ds = train_ds.select(range(max_questions))\n",
-    "\n",
-    "available_keys = set(train_ds.column_names)\n",
-    "reference_key = _first_existing_key(\n",
-    "    [\"question\", \"query\", \"text\", \"input\"], available_keys\n",
-    ")\n",
-    "response_key = _first_existing_key(\n",
-    "    [\"answer\", \"answers\", \"response\", \"output\"], available_keys\n",
-    ")\n",
-    "\n",
-    "if reference_key is None or response_key is None:\n",
-    "    raise KeyError(\n",
-    "        f\"Expected question/answer-like columns not found. \"\n",
-    "        f\"Available columns: {train_ds.column_names}\"\n",
-    "    )\n",
-    "\n",
-    "rows = []\n",
-    "for row in train_ds:\n",
-    "    rows.append(\n",
-    "        {\n",
-    "            \"reference\": str(row[reference_key]),\n",
-    "            \"response\": _normalize_answer(row[response_key]),\n",
-    "        }\n",
-    "    )\n",
-    "\n",
-    "eval_ds = EvaluationDataset.from_list(rows)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "id": "753aab30",
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "DatasetDict({\n",
-      "    train: Dataset({\n",
-      "        features: ['query', 'answer'],\n",
-      "        num_rows: 100231\n",
-      "    })\n",
-      "})\n",
-      "['query', 'answer']\n",
-      "{'query': 'when did richmond last play in a preliminary final', 'answer': \"Richmond Football Club Richmond began 2017 with 5 straight wins, a feat it had not achieved since 1995. A series of close losses hampered the Tigers throughout the middle of the season, including a 5-point loss to the Western Bulldogs, 2-point loss to Fremantle, and a 3-point loss to the Giants. Richmond ended the season strongly with convincing victories over Fremantle and St Kilda in the final two rounds, elevating the club to 3rd on the ladder. Richmond's first final of the season against the Cats at the MCG attracted a record qualifying final crowd of 95,028; the Tigers won by 51 points. Having advanced to the first preliminary finals for the first time since 2001, Richmond defeated Greater Western Sydney by 36 points in front of a crowd of 94,258 to progress to the Grand Final against Adelaide, their first Grand Final appearance since 1982. The attendance was 100,021, the largest crowd to a grand final since 1986. The Crows led at quarter time and led by as many as 13, but the Tigers took over the game as it progressed and scored seven straight goals at one point. They eventually would win by 48 points – 16.12 (108) to Adelaide's 8.12 (60) – to end their 37-year flag drought.[22] Dustin Martin also became the first player to win a Premiership medal, the Brownlow Medal and the Norm Smith Medal in the same season, while Damien Hardwick was named AFL Coaches Association Coach of the Year. Richmond's jump from 13th to premiers also marked the biggest jump from one AFL season to the next.\"}\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(ds)\n",
-    "print(ds[\"train\"].column_names)\n",
-    "print(ds[\"train\"][0])"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "id": "6c3d4235",
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# ...existing code...\n",
-    "class OllamaRagasEmbeddingAdapter(BaseRagasEmbedding):\n",
-    "    \"\"\"Adaptador de LangChain Ollama a la API moderna de embeddings en Ragas.\"\"\"\n",
-    "\n",
-    "    def __init__(self, base_url: str, model_name: str) -> None:\n",
-    "        self._client = OllamaEmbeddings(base_url=base_url, model=model_name)\n",
-    "\n",
-    "    def embed_text(self, text: str) -> list[float]:\n",
-    "        return self._client.embed_query(text)\n",
-    "\n",
-    "    async def aembed_text(self, text: str) -> list[float]:\n",
-    "        return await asyncio.to_thread(self.embed_text, text)\n",
-    "\n",
-    "    def embed_query(self, text: str) -> list[float]:\n",
-    "        return self.embed_text(text)\n",
-    "\n",
-    "    def embed_documents(self, texts: Sequence[str]) -> list[list[float]]:\n",
-    "        return self._client.embed_documents(list(texts))\n",
-    "\n",
-    "    async def aembed_query(self, text: str) -> list[float]:\n",
-    "        return await self.aembed_text(text)\n",
-    "\n",
-    "    async def aembed_documents(\n",
-    "        self, texts: Sequence[str]\n",
-    "    ) -> list[list[float]]:\n",
-    "        return await asyncio.to_thread(self.embed_documents, texts)\n",
-    "\n",
-    "\n",
-    "if not BASE_URL or not MODEL_NAME:\n",
-    "    raise ValueError(\n",
-    "        \"Faltan variables de entorno: LLM_BASE_LOCAL_URL u OLLAMA_MODEL_NAME.\"\n",
-    "    )\n",
-    "\n",
-    "embeddings = OllamaRagasEmbeddingAdapter(\n",
-    "    base_url=BASE_URL,\n",
-    "    model_name=MODEL_NAME,\n",
-    ")\n",
-    "\n",
-    "semantic_sim = SemanticSimilarity()\n",
-    "# ...existing code..."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "id": "54aacf01",
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "application/vnd.jupyter.widget-view+json": {
-       "model_id": "6a4b6e91c71d4849922f36d45f3e9f7f",
-       "version_major": 2,
-       "version_minor": 0
-      },
-      "text/plain": [
-       "Evaluating:   0%|          | 0/100231 [00:00<?, ?it/s]"
-      ]
-     },
-     "metadata": {},
-     "output_type": "display_data"
-    }
-   ],
-   "source": [
-    "# ...existing code...\n",
-    "result = evaluate(\n",
-    "    dataset=eval_ds,\n",
-    "    metrics=[semantic_sim],\n",
-    "    embeddings=embeddings,\n",
-    ")\n",
-    "\n",
-    "print(result)\n",
-    "# ...existing code...\n",
-    "# ...existing code..."
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "assistance-engine",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.12.11"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}

research/embeddings/scifact/qrels/test.tsv

@@ -1,340 +0,0 @@
-query-id	corpus-id	score
-1	31715818	1
-3	14717500	1
-5	13734012	1
-13	1606628	1
-36	5152028	1
-36	11705328	1
-42	18174210	1
-48	13734012	1
-49	5953485	1
-50	12580014	1
-51	45638119	1
-53	45638119	1
-54	49556906	1
-56	4709641	1
-57	4709641	1
-70	5956380	1
-70	4414547	1
-72	6076903	1
-75	4387784	1
-94	1215116	1
-99	18810195	1
-100	4381486	1
-113	6157837	1
-115	33872649	1
-118	6372244	1
-124	4883040	1
-127	21598000	1
-128	8290953	1
-129	27768226	1
-130	27768226	1
-132	7975937	1
-133	38485364	1
-133	6969753	1
-133	17934082	1
-133	16280642	1
-133	12640810	1
-137	26016929	1
-141	6955746	1
-141	14437255	1
-142	10582939	1
-143	10582939	1
-146	10582939	1
-148	1084345	1
-163	18872233	1
-171	12670680	1
-179	16322674	1
-179	27123743	1
-179	23557241	1
-179	17450673	1
-180	16966326	1
-183	12827098	1
-185	18340282	1
-198	2177022	1
-208	13519661	1
-212	22038539	1
-213	13625993	1
-216	21366394	1
-217	21366394	1
-218	21366394	1
-219	21366394	1
-230	3067015	1
-232	10536636	1
-233	4388470	1
-236	4388470	1
-237	4942718	1
-238	2251426	1
-239	14079881	1
-248	1568684	1
-249	1568684	1
-261	1122279	1
-261	10697096	1
-268	970012	1
-269	970012	1
-274	11614737	1
-275	4961038	1
-275	14241418	1
-275	14819804	1
-279	14376683	1
-294	10874408	1
-295	20310709	1
-298	39381118	1
-300	3553087	1
-303	4388470	1
-312	6173523	1
-314	4347374	1
-324	2014909	1
-327	17997584	1
-338	23349986	1
-343	7873737	1
-343	5884524	1
-350	16927286	1
-354	8774475	1
-362	38587347	1
-380	19005293	1
-384	13770184	1
-385	9955779	1
-385	9767444	1
-386	16495649	1
-388	1148122	1
-399	791050	1
-410	14924526	1
-411	14924526	1
-415	6309659	1
-421	11172205	1
-431	28937856	1
-436	14637235	1
-437	18399038	1
-439	4423559	1
-440	4423559	1
-443	10165258	1
-452	12804937	1
-452	464511	1
-475	18678095	1
-478	14767844	1
-491	56893404	1
-501	17930286	1
-502	13071728	1
-507	30774694	1
-508	13980338	1
-513	13230773	1
-514	16256507	1
-516	29564505	1
-517	15663829	1
-521	34873974	1
-525	13639330	1
-527	3863543	1
-528	5476778	1
-532	12991445	1
-533	12991445	1
-535	39368721	1
-536	16056514	1
-539	13282296	1
-540	11886686	1
-540	25007443	1
-544	24221369	1
-549	9433958	1
-551	33499189	1
-552	1471041	1
-554	1049501	1
-560	40096222	1
-569	23460562	1
-575	10300888	1
-577	5289038	1
-578	8764879	1
-587	16999023	1
-589	10984005	1
-593	19675911	1
-597	12779444	1
-597	36355784	1
-597	25742130	1
-598	25742130	1
-613	9638032	1
-619	20888849	1
-619	2565138	1
-623	17000834	1
-628	24512064	1
-636	24294572	1
-637	25649714	1
-641	5912283	1
-641	31554917	1
-644	13619127	1
-649	12789595	1
-659	1215116	1
-660	1215116	1
-674	2095573	1
-684	4942718	1
-690	18750453	1
-691	10991183	1
-692	24088502	1
-693	24088502	1
-700	4350400	1
-702	4350400	1
-715	18421962	1
-716	18421962	1
-718	17587795	1
-721	1834762	1
-723	5531479	1
-727	7521113	1
-728	7521113	1
-728	36444198	1
-729	26851674	1
-742	32159283	1
-743	32159283	1
-744	8460275	1
-756	2831620	1
-759	1805641	1
-768	6421792	1
-770	15476777	1
-775	32275758	1
-781	24338780	1
-783	40632104	1
-784	2356950	1
-785	12471115	1
-793	8551160	1
-800	22543403	1
-805	22180793	1
-808	36606083	1
-811	19799455	1
-814	33387953	1
-820	8646760	1
-821	8646760	1
-823	15319019	1
-830	1897324	1
-831	1897324	1
-832	30303335	1
-834	5483793	1
-837	15928989	1
-839	1469751	1
-845	17741440	1
-847	16787954	1
-852	13843341	1
-859	1982286	1
-870	195689316	1
-873	1180972	1
-873	19307912	1
-873	27393799	1
-873	29025270	1
-873	3315558	1
-879	8426046	1
-880	8426046	1
-882	14803797	1
-887	18855191	1
-903	10648422	1
-904	7370282	1
-907	6923961	1
-911	11254556	1
-913	3203590	1
-914	3203590	1
-921	1642727	1
-922	17077004	1
-936	5483793	1
-956	12956194	1
-957	123859	1
-960	8780599	1
-967	2119889	1
-967	8997410	1
-971	46695481	1
-971	27873158	1
-971	28617573	1
-971	9764256	1
-975	5304891	1
-982	2988714	1
-985	6828370	1
-993	16472469	1
-1012	9745001	1
-1014	6277638	1
-1019	11603066	1
-1020	9433958	1
-1021	9433958	1
-1024	5373138	1
-1029	13923140	1
-1029	13940200	1
-1029	11899391	1
-1041	25254425	1
-1041	16626264	1
-1049	12486491	1
-1062	20381484	1
-1086	39281140	1
-1088	37549932	1
-1089	17628888	1
-1099	7662206	1
-1100	7662206	1
-1104	3898784	1
-1107	20532591	1
-1110	13770184	1
-1121	4456756	1
-1130	17997584	1
-1132	33499189	1
-1132	9283422	1
-1137	33370	1
-1140	12009265	1
-1144	10071552	1
-1146	13906581	1
-1150	11369420	1
-1163	15305881	1
-1175	31272411	1
-1179	31272411	1
-1180	31272411	1
-1185	16737210	1
-1187	52873726	1
-1191	30655442	1
-1194	11419230	1
-1196	25649714	1
-1197	25649714	1
-1199	16760369	1
-1200	3441524	1
-1202	3475317	1
-1204	31141365	1
-1207	18909530	1
-1213	14407673	1
-1216	24142891	1
-1221	19736671	1
-1225	9650982	1
-1226	13777138	1
-1232	13905670	1
-1241	4427392	1
-1245	7662395	1
-1259	24341590	1
-1262	44172171	1
-1266	37480103	1
-1270	13900610	1
-1271	13768432	1
-1272	17081238	1
-1273	11041152	1
-1274	12428814	1
-1274	27731651	1
-1274	4406819	1
-1278	11335781	1
-1279	11335781	1
-1280	4387784	1
-1281	4387784	1
-1282	23649163	1
-1290	4687948	1
-1292	56893404	1
-1298	11718220	1
-1303	12631697	1
-1316	27910499	1
-1319	16284655	1
-1320	16284655	1
-1332	5304891	1
-1335	27910499	1
-1336	27910499	1
-1337	20231138	1
-1339	15482274	1
-1344	9559146	1
-1352	12885341	1
-1359	11614737	1
-1362	8290953	1
-1363	8290953	1
-1368	2425364	1
-1370	2425364	1
-1379	16322674	1
-1379	27123743	1
-1379	23557241	1
-1379	17450673	1
-1382	17755060	1
-1385	306006	1
-1389	23895668	1
-1395	17717391	1

research/embeddings/scifact/qrels/train.tsv

@@ -1,920 +0,0 @@
-query-id	corpus-id	score
-0	31715818	1
-2	13734012	1
-4	22942787	1
-6	2613775	1
-9	44265107	1
-10	32587939	1
-11	32587939	1
-12	33409100	1
-14	641786	1
-15	22080671	1
-17	1606628	1
-18	22942787	1
-19	3202143	1
-20	3202143	1
-21	41493639	1
-22	6490571	1
-24	3471191	1
-25	2613775	1
-26	32390525	1
-27	32390525	1
-28	12670680	1
-30	24341590	1
-32	12428497	1
-34	11705328	1
-35	5152028	1
-35	11705328	1
-37	5152028	1
-37	11705328	1
-39	13497630	1
-40	13497630	1
-41	18174210	1
-43	7224723	1
-44	56893404	1
-45	56893404	1
-46	380526	1
-47	3512154	1
-47	26996935	1
-52	45638119	1
-55	49556906	1
-58	4709641	1
-60	13899137	1
-60	13901073	1
-61	13899137	1
-61	13901073	1
-62	32587939	1
-63	40349336	1
-64	40349336	1
-66	14806256	1
-67	21295300	1
-68	21295300	1
-69	5956380	1
-69	4414547	1
-71	1127562	1
-73	6076903	1
-74	4387784	1
-76	5531479	1
-77	5531479	1
-78	5099266	1
-79	5099266	1
-80	4920376	1
-81	1797622	1
-82	3619372	1
-85	7521113	1
-85	22406695	1
-86	7521113	1
-86	22406695	1
-88	7521113	1
-88	22406695	1
-89	7521113	1
-89	22406695	1
-90	22406695	1
-91	1084345	1
-92	1084345	1
-93	2692522	1
-95	1215116	1
-96	14500725	1
-98	6540064	1
-104	40164383	1
-105	36606083	1
-106	25515907	1
-106	5151024	1
-108	6191684	1
-108	22995579	1
-108	23865182	1
-109	4319174	1
-111	13513790	1
-112	6157837	1
-114	33872649	1
-116	33872649	1
-119	14606752	1
-120	14606752	1
-121	31460499	1
-122	31460499	1
-123	4883040	1
-126	24512064	1
-134	4695046	1
-138	26016929	1
-139	22080671	1
-144	10582939	1
-149	6227220	1
-152	15488881	1
-153	4702639	1
-154	4702639	1
-155	37549932	1
-156	37549932	1
-157	13439128	1
-159	9394119	1
-160	52874170	1
-161	6903077	1
-164	5824985	1
-165	5824985	1
-166	18872233	1
-167	18872233	1
-168	5824985	1
-169	5824985	1
-172	12670680	1
-173	8126244	1
-174	1710116	1
-175	1710116	1
-176	32587939	1
-177	9669099	1
-178	16322674	1
-178	27123743	1
-178	23557241	1
-178	17450673	1
-181	16966326	1
-182	11369420	1
-184	12827098	1
-186	16855829	1
-187	16855829	1
-189	4421578	1
-196	19313533	1
-197	2177022	1
-199	2177022	1
-200	18231807	1
-201	2462673	1
-203	9558539	1
-204	7898952	1
-205	7898952	1
-205	470625	1
-209	32587939	1
-210	13794374	1
-211	13794374	1
-214	13625993	1
-220	19205437	1
-221	19205437	1
-222	19205437	1
-223	2014909	1
-224	6944800	1
-225	6944800	1
-226	6944800	1
-227	26973393	1
-228	4928057	1
-229	56893404	1
-235	4388470	1
-241	2212067	1
-241	10608822	1
-242	2212067	1
-242	10608822	1
-243	8148122	1
-244	21498497	1
-245	8447873	1
-245	3430789	1
-246	8447873	1
-246	3430789	1
-247	13578199	1
-250	1568684	1
-251	1568684	1
-253	37424881	1
-254	37424881	1
-255	5850219	1
-256	5850219	1
-258	22080671	1
-259	8883846	1
-262	14610165	1
-263	11328820	1
-263	30041340	1
-263	14853989	1
-264	11328820	1
-265	2033917	1
-266	22405338	1
-267	5912283	1
-267	31554917	1
-272	11614737	1
-277	14376683	1
-278	14376683	1
-280	25001628	1
-281	4632921	1
-283	1974176	1
-285	5548081	1
-286	4709641	1
-287	4709641	1
-290	15048300	1
-292	15048300	1
-293	10874408	1
-296	4398832	1
-299	39381118	1
-301	3553087	1
-304	14797520	1
-305	14797520	1
-306	7821634	1
-308	7821634	1
-309	7821634	1
-310	6173523	1
-313	6173523	1
-315	3701541	1
-316	712078	1
-317	4506414	1
-323	2014909	1
-325	40349336	1
-326	40349336	1
-330	9505448	1
-331	9505448	1
-332	29023309	1
-333	29023309	1
-334	25079962	1
-335	1780819	1
-336	2097256	1
-337	2097256	1
-339	23349986	1
-340	7098463	1
-341	7098463	1
-342	7873737	1
-342	5884524	1
-345	4394817	1
-346	11902109	1
-347	11902109	1
-349	13497630	1
-351	14658685	1
-352	14658685	1
-355	12800122	1
-355	38380061	1
-356	6144337	1
-357	18111172	1
-358	18111172	1
-361	38587347	1
-363	5386514	1
-364	1550937	1
-365	600437	1
-366	13956305	1
-367	27099731	1
-368	27099731	1
-369	6826100	1
-370	1550937	1
-371	1550937	1
-372	24922825	1
-375	1522647	1
-376	22401061	1
-377	18810195	1
-378	45154987	1
-378	10534299	1
-378	11886686	1
-378	25007443	1
-378	17150648	1
-379	19005293	1
-381	18340282	1
-382	11659421	1
-383	13770184	1
-389	1148122	1
-390	1148122	1
-391	1148122	1
-392	1148122	1
-393	1148122	1
-394	11360768	1
-396	1456068	1
-397	1456068	1
-398	8883846	1
-400	791050	1
-401	5633876	1
-403	1921218	1
-404	1921218	1
-406	6796297	1
-407	9889151	1
-413	6309659	1
-414	6309659	1
-416	6309659	1
-417	6309659	1
-418	16660256	1
-420	9315213	1
-422	11172205	1
-423	8595678	1
-425	33257464	1
-426	16728949	1
-428	16728949	1
-429	36540079	1
-430	28937856	1
-432	8002887	1
-434	9500590	1
-435	9500590	1
-441	2014909	1
-444	10165258	1
-445	10165258	1
-447	2052720	1
-448	2052720	1
-449	12209494	1
-449	3430789	1
-453	4200695	1
-454	4200695	1
-455	12643937	1
-456	30507607	1
-458	597790	1
-461	40096222	1
-463	19736671	1
-466	22544171	1
-469	1410197	1
-470	12685434	1
-472	7185591	1
-472	26330861	1
-472	4414481	1
-473	4373433	1
-474	4373433	1
-479	6325527	1
-480	6325527	1
-481	14706752	1
-482	10991183	1
-483	22703082	1
-484	14637235	1
-485	14637235	1
-486	14637235	1
-487	14637235	1
-488	1780819	1
-489	6625693	1
-490	56893404	1
-492	19583924	1
-493	19583924	1
-494	34873974	1
-495	17077004	1
-498	17077004	1
-499	26064662	1
-500	17930286	1
-504	10883736	1
-505	22703082	1
-506	7433668	1
-509	13980338	1
-515	29564505	1
-523	14803797	1
-524	14803797	1
-526	3863543	1
-529	10546779	1
-529	25413327	1
-529	36651210	1
-530	10546779	1
-530	25413327	1
-530	36651210	1
-530	87610599	1
-531	10546779	1
-531	25413327	1
-531	36651210	1
-537	16056514	1
-541	45154987	1
-541	11886686	1
-541	25007443	1
-542	19688024	1
-545	24221369	1
-547	10648422	1
-548	18199839	1
-550	33499189	1
-553	1471041	1
-555	1049501	1
-557	1049501	1
-559	3475317	1
-562	20101846	1
-563	2867345	1
-564	2867345	1
-565	16120395	1
-566	16120395	1
-568	23418635	1
-570	20333864	1
-571	20333864	1
-572	4447055	1
-573	10300888	1
-574	10300888	1
-576	4468861	1
-579	34139429	1
-580	23460562	1
-582	14260013	1
-584	14260013	1
-585	42291761	1
-588	16999023	1
-590	10984005	1
-591	14682243	1
-592	14682243	1
-594	19675911	1
-595	4824840	1
-600	12258338	1
-601	12258338	1
-602	3701541	1
-603	6540064	1
-606	712078	1
-607	4506414	1
-609	40096222	1
-610	40096222	1
-611	32408470	1
-612	9638032	1
-614	9638032	1
-615	9638032	1
-616	18670	1
-617	18670	1
-618	6836086	1
-620	2565138	1
-621	1642727	1
-622	17000834	1
-624	20033112	1
-625	20033112	1
-626	16355392	1
-631	5468807	1
-632	5172048	1
-633	5172048	1
-635	1686997	1
-638	25649714	1
-640	6503185	1
-642	13619127	1
-643	15535511	1
-645	12810152	1
-646	12810152	1
-647	15041758	1
-648	15041758	1
-650	12789595	1
-651	9433958	1
-652	9433958	1
-653	24384587	1
-654	57574395	1
-655	57574395	1
-657	8533245	1
-658	5293024	1
-661	37204802	1
-662	37204802	1
-663	22080671	1
-665	12580014	1
-666	4469125	1
-667	6493422	1
-668	6493422	1
-668	25148216	1
-669	6493422	1
-669	25148216	1
-670	5573975	1
-671	5573975	1
-672	15635366	1
-673	2095573	1
-676	857189	1
-677	857189	1
-679	13639330	1
-680	9315213	1
-681	9315213	1
-682	9315213	1
-683	9315213	1
-685	4452659	1
-686	4452659	1
-687	4452659	1
-688	4452659	1
-689	22080671	1
-694	1071991	1
-696	16355392	1
-698	22544171	1
-703	4350400	1
-704	14658685	1
-705	22442133	1
-709	22442133	1
-710	22442133	1
-713	18421962	1
-714	18421962	1
-717	17587795	1
-724	5531479	1
-726	7521113	1
-726	36444198	1
-730	13400643	1
-732	34469966	1
-733	34469966	1
-734	4961038	1
-736	5389095	1
-737	16562534	1
-737	6609935	1
-738	16562534	1
-738	6609935	1
-738	33912020	1
-739	4446814	1
-740	23078022	1
-745	11291348	1
-746	11291348	1
-747	11291348	1
-748	11291348	1
-749	13868795	1
-751	19800147	1
-752	19800147	1
-753	1173667	1
-755	17844478	1
-757	17123657	1
-758	14195528	1
-760	1805641	1
-761	10009203	1
-762	4695046	1
-764	7552215	1
-765	7552215	1
-766	7552215	1
-767	2488880	1
-771	15476777	1
-772	24922825	1
-774	32275758	1
-776	32275758	1
-777	32275758	1
-778	13001323	1
-779	13001323	1
-780	8246922	1
-780	24338780	1
-782	8246922	1
-787	4740447	1
-788	4740447	1
-789	15493354	1
-790	15493354	1
-791	15984735	1
-792	3610080	1
-795	8551160	1
-797	8551160	1
-798	8551160	1
-799	5293024	1
-801	22180793	1
-802	22180793	1
-803	22180793	1
-804	22180793	1
-807	36606083	1
-810	13513790	1
-812	19799455	1
-813	33387953	1
-815	8148304	1
-816	8148304	1
-817	17814815	1
-818	17814815	1
-822	15319019	1
-825	15319019	1
-826	4678846	1
-828	4678846	1
-835	15928989	1
-838	15928989	1
-840	15663829	1
-841	15663829	1
-844	17741440	1
-846	22696649	1
-848	14500725	1
-853	24922825	1
-854	12206390	1
-855	8190282	1
-856	43334921	1
-857	43334921	1
-858	1982286	1
-860	16066726	1
-861	16066726	1
-863	20568364	1
-863	16361581	1
-866	37822406	1
-867	14340571	1
-871	195689316	1
-876	195689316	1
-877	313394	1
-881	14803797	1
-883	14803797	1
-884	14803797	1
-885	6477536	1
-886	6477536	1
-890	2097256	1
-891	2097256	1
-893	13509809	1
-894	14724693	1
-895	18750453	1
-896	14338915	1
-897	14338915	1
-898	13106686	1
-898	5572127	1
-899	13106686	1
-899	5572127	1
-900	18678095	1
-901	6540064	1
-902	10648422	1
-908	6923961	1
-909	11254556	1
-910	11254556	1
-912	11254556	1
-916	18037805	1
-917	34071621	1
-919	16422880	1
-923	17077004	1
-925	17077004	1
-926	16390264	1
-927	16390264	1
-928	18174210	1
-929	18174210	1
-930	16056514	1
-933	14711483	1
-934	8563659	1
-935	5483793	1
-938	26231129	1
-939	26231129	1
-940	12258338	1
-941	12258338	1
-942	11527199	1
-944	1642727	1
-945	8428935	1
-945	26112696	1
-945	4463588	1
-945	13083189	1
-946	8428935	1
-946	26112696	1
-946	4463588	1
-946	13083189	1
-949	13578199	1
-951	21414718	1
-952	3355397	1
-953	3355397	1
-954	3355397	1
-955	2078658	1
-955	30507607	1
-959	8780599	1
-962	13931771	1
-962	935538	1
-962	4306711	1
-963	4162857	1
-963	29828242	1
-964	4162857	1
-964	29828242	1
-965	40817021	1
-969	19356271	1
-969	17368516	1
-970	19356271	1
-970	17368516	1
-972	46695481	1
-972	27873158	1
-972	28617573	1
-972	9764256	1
-973	27446873	1
-973	27873158	1
-973	28617573	1
-973	9764256	1
-976	5304891	1
-977	14075252	1
-977	39264456	1
-978	14075252	1
-979	11659421	1
-980	20128547	1
-984	6828370	1
-988	3033830	1
-989	9988425	1
-990	16472469	1
-992	16472469	1
-994	16472469	1
-996	16472469	1
-997	16472469	1
-998	16472469	1
-999	16472469	1
-1000	16472469	1
-1001	5702790	1
-1002	13639330	1
-1003	14332945	1
-1003	4319844	1
-1003	4899981	1
-1004	301838	1
-1004	2734421	1
-1004	3952288	1
-1005	301838	1
-1005	2734421	1
-1005	3952288	1
-1006	4926049	1
-1008	2547636	1
-1009	1982286	1
-1011	9745001	1
-1015	6277638	1
-1016	6277638	1
-1018	11603066	1
-1023	16927286	1
-1025	32408470	1
-1026	3113630	1
-1027	3113630	1
-1028	13923140	1
-1028	11899391	1
-1030	6441369	1
-1031	12486491	1
-1032	6836086	1
-1033	6836086	1
-1034	4547102	1
-1035	4547102	1
-1036	4547102	1
-1037	16287725	1
-1038	16287725	1
-1040	25254425	1
-1040	16626264	1
-1042	17421851	1
-1043	17671145	1
-1044	22500262	1
-1045	22500262	1
-1046	418246	1
-1046	4324278	1
-1046	16712164	1
-1047	14706752	1
-1048	12486491	1
-1050	19878070	1
-1052	18816720	1
-1053	18816720	1
-1054	10072941	1
-1055	13906581	1
-1056	4200695	1
-1058	13027590	1
-1065	20418809	1
-1067	4429668	1
-1068	4429668	1
-1069	4200695	1
-1070	25649714	1
-1072	4824840	1
-1073	4824840	1
-1074	14658685	1
-1075	14658685	1
-1081	5691302	1
-1084	5691302	1
-1085	5691302	1
-1087	39281140	1
-1090	17628888	1
-1091	2603304	1
-1096	29638116	1
-1097	26851674	1
-1098	13552682	1
-1101	3874000	1
-1102	3874000	1
-1103	3898784	1
-1105	6710713	1
-1106	6710713	1
-1109	13770184	1
-1109	8582337	1
-1111	1686881	1
-1112	1686881	1
-1114	12824568	1
-1115	44048701	1
-1118	23351136	1
-1119	5323845	1
-1119	18997216	1
-1119	13907928	1
-1120	5323845	1
-1120	18997216	1
-1120	13907928	1
-1125	21009874	1
-1126	21009874	1
-1127	27466734	1
-1128	33499189	1
-1128	9283422	1
-1133	24142891	1
-1134	33370	1
-1135	33370	1
-1136	33370	1
-1138	6796297	1
-1139	12009265	1
-1141	12009265	1
-1142	5260382	1
-1145	10071552	1
-1148	4828631	1
-1153	7370282	1
-1156	12584053	1
-1157	12584053	1
-1158	12584053	1
-1159	12584053	1
-1161	13048272	1
-1162	15305881	1
-1164	4455466	1
-1165	4455466	1
-1166	9889151	1
-1168	8563659	1
-1169	4319174	1
-1170	18956141	1
-1171	18956141	1
-1173	7370282	1
-1174	31272411	1
-1176	13910150	1
-1177	13910150	1
-1178	31272411	1
-1181	301838	1
-1181	2734421	1
-1181	39128592	1
-1181	3952288	1
-1182	14541844	1
-1183	1967017	1
-1184	16737210	1
-1186	7485455	1
-1188	4394817	1
-1190	30655442	1
-1193	20532591	1
-1195	26283293	1
-1205	5558754	1
-1206	18909530	1
-1208	10284593	1
-1209	4347374	1
-1210	4928282	1
-1211	4928282	1
-1212	6493422	1
-1212	44724517	1
-1214	6493422	1
-1214	14407673	1
-1215	16355392	1
-1218	15635366	1
-1219	9393969	1
-1219	14864285	1
-1220	13023410	1
-1223	5289038	1
-1224	21932050	1
-1224	34016987	1
-1227	25641414	1
-1228	25641414	1
-1229	1676568	1
-1230	13905670	1
-1231	13905670	1
-1234	13905670	1
-1235	17973161	1
-1236	17973161	1
-1237	3654468	1
-1238	3654468	1
-1239	21387297	1
-1239	4427392	1
-1244	18949516	1
-1246	7662395	1
-1247	5114282	1
-1248	7209559	1
-1249	7209559	1
-1253	3321943	1
-1254	16939583	1
-1255	16939583	1
-1257	581832	1
-1258	12040627	1
-1260	24341590	1
-1261	13023410	1
-1263	3981729	1
-1265	37480103	1
-1268	52072815	1
-1269	13900610	1
-1275	27731651	1
-1276	3475317	1
-1284	3578380	1
-1288	4687948	1
-1289	21239672	1
-1291	56893404	1
-1293	43329366	1
-1294	2078658	1
-1294	30507607	1
-1295	21239672	1
-1297	9167230	1
-1300	6421792	1
-1302	12631697	1
-1304	12631697	1
-1305	12631697	1
-1306	6000423	1
-1306	5836	1
-1307	18231807	1
-1308	18231807	1
-1309	18231807	1
-1310	8042158	1
-1311	13763195	1
-1312	24177706	1
-1314	13072112	1
-1314	16237005	1
-1315	13072112	1
-1315	16237005	1
-1322	16284655	1
-1323	19912367	1
-1324	19912367	1
-1325	40476126	1
-1327	24241932	1
-1327	22194407	1
-1328	3475317	1
-1330	14075252	1
-1331	14075252	1
-1333	1649738	1
-1334	13923140	1
-1334	13940200	1
-1334	11899391	1
-1340	15482274	1
-1341	15482274	1
-1342	8148122	1
-1345	9559146	1
-1346	9505402	1
-1347	19005293	1
-1348	19005293	1
-1349	5377642	1
-1350	5377642	1
-1351	28369117	1
-1353	18816720	1
-1355	5256564	1
-1356	13764090	1
-1360	11614737	1
-1361	15488881	1
-1361	15058155	1
-1364	8290953	1
-1366	4406819	1
-1367	2425364	1
-1371	16256507	1
-1372	21003930	1
-1373	21003930	1
-1374	21993510	1
-1375	21993510	1
-1376	3944632	1
-1378	2488880	1
-1380	16322674	1
-1380	23557241	1
-1380	17450673	1
-1381	13481880	1
-1383	17755060	1
-1386	306006	1
-1387	9669099	1
-1390	2890952	1
-1391	6766459	1
-1392	6766459	1
-1393	2000038	1
-1393	12440953	1
-1394	2251426	1
-1397	17717391	1
-1398	17717391	1
-1400	14706752	1
-1401	5185871	1
-1402	8126244	1
-1403	33370	1
-1403	38355793	1
-1404	33370	1
-1404	38355793	1
-1405	10504681	1
-1406	2617858	1
-1407	8087082	1
-1407	29863668	1

scripts/pipelines/samples_generator/DOCUMENTATION.md

@@ -0,0 +1,341 @@
+# AVAP MAP-Elites Dataset Pipeline — Documentación
+
+> Scripts para la síntesis de datasets de benchmarks AVAP mediante cobertura de gramática garantizada y priors estadísticos extraídos de código real de producción.
+
+---
+
+## Índice
+
+- [Visión general del sistema](#visión-general-del-sistema)
+- [Requisitos e instalación](#requisitos-e-instalación)
+- [construct_prior.py](#construct_priorpy)
+  - [Uso](#uso-construct_prior)
+  - [Cómo funciona](#cómo-funciona-construct_prior)
+- [generate_mbap_v2.py](#generate_mbap_v2py)
+  - [Uso](#uso-generate_mbap_v2)
+  - [Cómo funciona](#cómo-funciona-generate_mbap_v2)
+- [Flujo de trabajo recomendado](#flujo-de-trabajo-recomendado)
+- [Archivos generados](#archivos-generados)
+
+---
+
+## Visión general del sistema
+
+El pipeline consta de dos scripts que trabajan en conjunto:
+
+```
+construct_prior.py          →    construct_map.yaml
+       ↓                                ↓
+generate_mbap_v2.py  ←─────────────────┘
+       ↓
+  dataset .json  +  coverage_stats .json
+```
+
+1. **`construct_prior.py`** analiza codebases reales en GitHub para extraer con qué frecuencia co-ocurren los 38 comandos AVAP en código de producción real. El resultado es un fichero `construct_map.yaml` con pesos estadísticos.
+
+2. **`generate_mbap_v2.py`** usa esos pesos para dirigir un generador MAP-Elites que llama a la API de Claude, garantizando cobertura uniforme de todas las combinaciones de pares y tríos del DSL AVAP.
+
+---
+
+## Requisitos e instalación
+
+```bash
+pip install requests pyyaml anthropic
+pip install datasets huggingface_hub   # solo si usas --prior-sources huggingface
+
+export ANTHROPIC_API_KEY=sk-ant-...
+export GITHUB_TOKEN=ghp_...            # opcional pero recomendado
+```
+
+El parser AVAP debe estar en ejecución si se desea validación AST real (opcional; si no está disponible, se usa keyword scanning como fallback):
+
+```bash
+# El parser debe escuchar en el puerto configurado (por defecto 8080)
+# Ejemplo: http://localhost:8080
+```
+
+---
+
+## `construct_prior.py`
+
+### Uso (construct_prior)
+
+El script tiene dos modos de operación: **generación** del mapa y **verificación** de uno existente.
+
+#### Generar `construct_map.yaml` (ejecutar una vez, o cuando cambie la gramática AVAP)
+
+```bash
+# Ejecución básica (sin token: límite de 10 req/min en GitHub)
+python construct_prior.py --generate-map
+
+# Con token de GitHub (recomendado — 30 req/min)
+python construct_prior.py --generate-map --github-token ghp_...
+
+# Analizar más ficheros para un prior más rico
+python construct_prior.py --generate-map --max-files 200 --github-token ghp_...
+
+# Ruta de salida personalizada
+python construct_prior.py --generate-map --output /ruta/construct_map.yaml
+```
+
+#### Verificar un `construct_map.yaml` existente
+
+```bash
+python construct_prior.py --verify --map construct_map.yaml
+```
+
+#### Uso como módulo desde `generate_mbap_v2.py`
+
+```python
+from construct_prior import ConstructPrior
+
+# Cargar desde YAML generado
+prior = ConstructPrior.from_yaml("construct_map.yaml")
+
+# Consultar el peso de una celda (par o trío de comandos AVAP)
+w = prior.cell_weight(frozenset({"try", "ormAccessSelect"}))
+
+# Fallback estático (sin necesidad de YAML)
+prior = ConstructPrior.static_fallback()
+```
+
+#### Parámetros CLI
+
+| Parámetro | Tipo | Por defecto | Descripción |
+|---|---|---|---|
+| `--generate-map` | flag | — | Activa el modo de extracción desde GitHub |
+| `--verify` | flag | — | Carga y muestra estadísticas de un YAML existente |
+| `--github-token` | str | `$GITHUB_TOKEN` | Token de acceso personal de GitHub |
+| `--max-files` | int | `100` | Número máximo de ficheros a analizar |
+| `--output` | str | `construct_map.yaml` | Ruta de salida del YAML |
+| `--map` | str | `construct_map.yaml` | Ruta del YAML a verificar (solo con `--verify`) |
+
+---
+
+### Cómo funciona (construct_prior)
+
+El objetivo del script es responder a la pregunta: **¿qué combinaciones de comandos AVAP aparecen juntas con más frecuencia en código API real de producción?** Esa información sirve para sesgar el generador de datasets hacia patrones realistas.
+
+#### 1. Vocabulario AVAP (`AVAP_NODE_NAMES`)
+
+Se define una lista canónica de 38 comandos AVAP organizados por categoría (variables, ORM, HTTP, criptografía, concurrencia, etc.). Esta lista es la única fuente de verdad del módulo y es importada también por `generate_mbap_v2.py`.
+
+#### 2. Tabla de equivalencias de lenguaje (`LANGUAGE_MAPPINGS`)
+
+Cada comando AVAP se mapea a sus equivalentes en Python (detección por AST), Go (keywords) y SQL. Esta tabla, verificada manualmente, define *qué buscar* al escanear los codebases. Por ejemplo:
+
+- `ormAccessSelect` → llamadas como `.fetchall()`, `.query()`, `.filter()`
+- `RequestPost` → `requests.post`, `httpx.post`, `session.post`
+- `try` → nodo `ast.Try` en el AST de Python
+
+#### 3. Extracción desde GitHub (`GitHubFetcher`)
+
+Se lanzan 16 queries predefinidas contra la GitHub Code Search API, cada una orientada a un patrón típico de microservicio (ORM + manejo de errores, clientes HTTP, autenticación con crypto, concurrencia async, JSON, fechas, etc.). Los ficheros encontrados se descargan en base64 y se decodifican en memoria.
+
+El fetcher respeta automáticamente el rate limit de la API (10 req/min sin token, 30 req/min con token) con esperas adaptativas entre peticiones.
+
+#### 4. Detección AST de Python (`PythonASTDetector`)
+
+Cada fichero descargado se parsea con el módulo estándar `ast` de Python. El walker del AST recorre todos los nodos y detecta qué comandos AVAP están presentes según su equivalente estructural:
+
+- `ast.Try` → `"try"` + `"exception"`
+- `ast.FunctionDef` → `"function"` + `"addParam"` (si tiene argumentos)
+- `ast.For` / `ast.AsyncFor` → `"startLoop"`
+- `ast.Call` con callee `.fetchall` → `"ormAccessSelect"`
+- etc.
+
+Este enfoque es **AST-level**, no keyword scanning, lo que elimina falsos positivos por nombres de variable, strings o comentarios. Si el fichero tiene errores de sintaxis, cae automáticamente a un keyword fallback.
+
+#### 5. Acumulación de co-ocurrencias (`CooccurrenceExtractor`)
+
+Por cada fichero analizado se obtiene un conjunto de comandos AVAP detectados. Se calculan todas las combinaciones de pares y tríos posibles de ese conjunto y se incrementan sus contadores. Por ejemplo, si un fichero contiene `{try, ormAccessSelect, return}`, se incrementan los contadores de `(try, ormAccessSelect)`, `(try, return)`, `(ormAccessSelect, return)` y el trío `(try, ormAccessSelect, return)`.
+
+#### 6. Normalización y escritura del YAML (`generate_construct_map`)
+
+Los contadores de co-ocurrencia se normalizan a `[0, 1]` dividiéndolos por el máximo observado. El resultado se escribe en `construct_map.yaml` con dos secciones:
+
+- `language_mappings` — la tabla de equivalencias (trazabilidad)
+- `pair_weights` / `trio_weights` — pesos empíricos extraídos
+
+#### 7. Propagación de pesos a subsets (`_propagate_subset_weights`)
+
+Los tríos que contienen pares con alto peso heredan el 60% del peso del par más relevante que contienen. Esto garantiza que los tríos formados por pares comunes sean visitados antes que tríos de combinaciones raras.
+
+#### 8. Fallback estático
+
+Si no se dispone de conexión a GitHub o de un YAML previo, `ConstructPrior.static_fallback()` retorna un conjunto de 40+ pesos hard-coded basados en conocimiento experto (por ejemplo, `(try, exception)` = 1.0, `(function, return)` = 0.98).
+
+---
+
+## `generate_mbap_v2.py`
+
+### Uso (generate_mbap_v2)
+
+#### Invocación básica
+
+```bash
+# Modo por defecto: MAP-Elites + ConstructPrior (Candidato F)
+python generate_mbap_v2.py --lrm avap.md --parser http://localhost:8080
+
+# Generar 5000 ejemplos con celdas de hasta 3 comandos
+python generate_mbap_v2.py --lrm avap.md --parser http://localhost:8080 \
+    --problems 5000 --cell-size 3 --mode map-elites-prior
+```
+
+#### Opciones del prior
+
+```bash
+# Usar datos reales de GitHub para el prior (AST-level, requiere red)
+python generate_mbap_v2.py --lrm avap.md --mode map-elites-prior \
+    --prior-sources github
+
+# Guardar el prior tras la extracción (para reutilizarlo después)
+python generate_mbap_v2.py --lrm avap.md --mode map-elites-prior \
+    --prior-sources github --prior-save prior_weights.json
+
+# Cargar un prior pre-extraído (sin peticiones a GitHub)
+python generate_mbap_v2.py --lrm avap.md --mode map-elites-prior \
+    --prior-load prior_weights.json
+
+# Usar el prior generado por construct_prior.py
+python generate_mbap_v2.py --lrm avap.md --mode map-elites-prior \
+    --prior-map construct_map.yaml
+```
+
+#### Modos de generación
+
+```bash
+# Candidato F: MAP-Elites + ConstructPrior (por defecto, recomendado)
+python generate_mbap_v2.py --lrm avap.md --mode map-elites-prior
+
+# Candidato E: MAP-Elites con pesos uniformes (baseline sin prior)
+python generate_mbap_v2.py --lrm avap.md --mode map-elites
+
+# Candidato A: CW-Reward pool (no implementado en v2, usar generate_mbap.py)
+python generate_mbap_v2.py --lrm avap.md --mode reward
+```
+
+#### Parámetros CLI completos
+
+| Parámetro | Tipo | Por defecto | Descripción |
+|---|---|---|---|
+| `--lrm` | str | `avap.md` | Ruta al Language Reference Manual de AVAP |
+| `--output` | str | `output/mbpp_avap_v2.json` | Ruta del dataset de salida |
+| `--problems` | int | `5000` | Número de ejemplos a generar |
+| `--parser` | str | `http://localhost:8080` | URL del parser AVAP |
+| `--cell-size` | int | `3` | Tamaño máximo de celda: 2=solo pares, 3=pares+tríos |
+| `--quality-threshold` | float | `0.80` | Calidad mínima para considerar una celda "buena" |
+| `--alpha` | float | `0.30` | Peso de los comandos bonus en la calidad |
+| `--beta` | float | `0.20` | Peso de la calidad de los tests en la calidad |
+| `--gamma` | float | `0.10` | Peso de la riqueza del código en la calidad |
+| `--mode` | choice | `map-elites-prior` | Modo de generación (ver arriba) |
+| `--prior-map` | str | `construct_map.yaml` | Ruta al YAML generado por `construct_prior.py` |
+| `--prior-epsilon` | float | `0.05` | Peso mínimo para celdas cola (tail cells) |
+| `--prior-phase3-threshold` | float | `0.70` | Calidad a partir de la cual se activa la Fase 3 (tail) |
+| `--api-key` | str | `$ANTHROPIC_API_KEY` | API key de Anthropic |
+
+---
+
+### Cómo funciona (generate_mbap_v2)
+
+El generador implementa un algoritmo de **Quality-Diversity (QD)** llamado MAP-Elites aplicado a la generación de ejemplos de código AVAP. El objetivo es producir un dataset que cubra de forma garantizada todas las combinaciones de pares y tríos de los 38 comandos del DSL, sin sesgos de distribución.
+
+#### 1. Vocabulario y detección de constructs (`AVAP_NODE_TYPES`, `CellValidator`)
+
+Se define un diccionario que mapea cada comando AVAP a sus patrones de keyword. `CellValidator` detecta qué comandos están presentes en un ejemplo generado usando dos estrategias:
+
+- **Desde el AST del parser AVAP**: se recorre el árbol recursivamente buscando nodos por `type`.
+- **Desde el código fuente** (fallback): se buscan los patrones de keyword del diccionario. `if_mode2` se comprueba antes que `if_mode1` para evitar ambigüedad.
+
+Además calcula una **puntuación de calidad** compuesta:
+
+```
+quality = fidelity
+        + alpha * bonus_constructs_ratio
+        + beta  * test_quality
+        + gamma * code_richness
+```
+
+- `fidelity`: fracción de los constructs requeridos por la celda que están presentes (componente principal).
+- `bonus_ratio`: constructs adicionales más allá de los requeridos.
+- `test_quality`: proporción de tests con patrón `re.match()` y longitud > 10.
+- `richness`: número de líneas normalizadas a 30 (proxy de complejidad).
+
+#### 2. Mapa de cobertura MAP-Elites (`CoverageMap`)
+
+Es la estructura central del algoritmo. Mantiene **una celda por cada combinación posible de 2 o 3 comandos AVAP** (con `cell_size=3`: 703 pares + N tríos de un total de 38 comandos). Cada celda almacena el mejor ejemplo encontrado hasta el momento para esa combinación.
+
+Una celda solo se considera válida si el ejemplo que contiene usa **todos** los constructs de su clave. El mapa expone métricas en tiempo real: tasa de llenado, entropía de distribución (Shannon), y celdas de baja calidad.
+
+#### 3. Selector de celdas — Candidato E (`CellSelector`)
+
+Implementa la estrategia de selección sin prior en tres fases:
+
+- **Fase 1**: Celdas vacías (round-robin aleatorio con semilla fija para reproducibilidad).
+- **Fase 2**: Celdas con calidad por debajo del umbral.
+- **Fase 3**: UCB (Upper Confidence Bound) sobre todas las celdas — equilibrio entre explotar celdas de alta calidad y explorar celdas poco visitadas.
+
+#### 4. Selector con prior — Candidato F (`CellSelectorPrior`)
+
+Extiende `CellSelector` incorporando los pesos de `ConstructPrior`:
+
+- **Fase 1**: Las celdas vacías con peso de prior alto (> 1.5× epsilon) se priorizan mediante muestreo ponderado. Esto hace que el dataset sea útil para RAG desde los primeros ejemplos (los patrones más frecuentes en producción se cubren primero).
+- **Fase 2**: Las celdas de baja calidad se seleccionan con UCB multiplicado por el peso del prior.
+- **Fase 3**: Una vez que las celdas de prior alto alcanzan el umbral de calidad, se activa la cobertura de las celdas cola (tail cells, prior ≈ epsilon).
+
+#### 5. Construcción del prompt y llamada a la API (`Generator`)
+
+Para cada celda seleccionada se construye un prompt estructurado que incluye el LRM completo de AVAP y una especificación precisa: "genera UN problema de benchmark estilo MBPP que use **exactamente** estos constructs: `{cell}`". El generador nunca pide exploración libre — cada llamada tiene una especificación de cobertura forzada.
+
+La respuesta se parsea esperando JSON con campos `prompt`, `code`, y `tests`.
+
+#### 6. Validación y actualización del mapa
+
+Cada ejemplo generado pasa por `CellValidator`:
+
+1. Se intenta parsear con el parser AVAP (si está disponible).
+2. Se detectan los constructs presentes.
+3. Se calcula la puntuación de calidad.
+4. Si la calidad supera el ejemplo actual de la celda (o la celda está vacía), el mapa se actualiza.
+
+#### 7. Checkpoints y métricas
+
+Cada 100 llamadas a la API se imprime un checkpoint con: tamaño del dataset, tasa de éxito, estado del mapa, entropía de distribución, y divergencia KL entre el dataset y el prior (KL = 0 significa alineación perfecta con patrones de producción).
+
+#### 8. Guardado (`_save`)
+
+Al finalizar se guardan dos ficheros:
+
+- `<output>.json` — el dataset completo.
+- `<output>_coverage_stats.json` — estadísticas de cobertura, entropía, frecuencia por nodo, y KL-divergence respecto al prior.
+
+---
+
+## Flujo de trabajo recomendado
+
+```bash
+# Paso 1: Generar el prior estadístico (una sola vez, o al actualizar la gramática)
+python construct_prior.py --generate-map --github-token $GITHUB_TOKEN --max-files 200
+
+# Paso 2: Verificar el prior generado
+python construct_prior.py --verify --map construct_map.yaml
+
+# Paso 3: Generar el dataset
+python generate_mbap_v2.py \
+    --lrm avap.md \
+    --parser http://localhost:8080 \
+    --prior-map construct_map.yaml \
+    --problems 5000 \
+    --output output/dataset_v2.json \
+    --mode map-elites-prior
+```
+
+---
+
+## Archivos generados
+
+| Fichero | Generado por | Descripción |
+|---|---|---|
+| `construct_map.yaml` | `construct_prior.py` | Pesos de co-ocurrencia de pares/tríos AVAP extraídos de GitHub |
+| `<output>.json` | `generate_mbap_v2.py` | Dataset de ejemplos AVAP estilo MBPP |
+| `<output>_coverage_stats.json` | `generate_mbap_v2.py` | Estadísticas de cobertura, entropía y KL-divergence |