- Created a new JSON file containing evaluation results for the AVAP knowledge models, including scores for faithfulness, answer relevancy, context recall, and context precision.
- Updated the evaluation notebook to use a new embedding model and fixed execution counts for code cells.
- Added GoldPool class to manage a top-K pool of high-reward examples.
- Implemented compute_reward function to calculate composite rewards based on execution coverage, novelty, and test quality.
- Introduced call_api_reward function for API calls in the new reward mode.
- Updated main function to support new reward mode with adjustable weights for ECS, novelty, and test quality.
- Enhanced dataset saving functionality to include reward statistics.
- Refactored existing code for improved readability and consistency.
- Created `candidate_F_reward_10_coverage_stats.json` with coverage statistics including total cells, filled cells, fill rate, and node type frequency.
- Added `mbpp_avap.json` containing 14 tasks with descriptions, code implementations, test inputs, and expected test results for various endpoints and functionalities.
- Created `n00 Beir Analysis_cosqa.ipynb` for analyzing CoSQA dataset with BEIR.
- Created `n00 first Analysis.ipynb` for initial analysis using Ragas and Ollama embeddings.
- Implemented `evaluate_embeddings_pipeline.py` to evaluate embedding models across CodexGlue, CoSQA, and SciFact benchmarks.
- Added adapters for Ollama and HuggingFace embeddings to ensure compatibility with BEIR.
- Included functions to load datasets and evaluate models with detailed metrics.
- Created `n00 Beir Analysis_cosqa.ipynb` for analyzing CoSQA dataset with BEIR.
- Created `n00 first Analysis.ipynb` for initial analysis with embeddings.
- Implemented `evaluate_embeddings_pipeline.py` to evaluate embedding models across CodexGlue, CoSQA, and SciFact benchmarks.
- Added adapters for Ollama and HuggingFace embeddings to ensure compatibility with BEIR.
- Enhanced error handling and data normalization in embedding processes.
- Included functionality to load datasets from local cache or download if not present.
- Introduced a new notebook for generating synthetic datasets for AVAP, including loading AVAP and MBPP data, and creating prompts for LLM interactions.
acano
pseco
rafa-ruiz
izapata