Commit Graph

7 Commits

Author SHA1 Message Date
acano fadf813494 Update Elasticsearch index version and enhance document processing
- Changed Elasticsearch index from "avap-docs-test-v3" to "avap-docs-test-v4" in elasticsearch_ingestion.py.
- Added Lark parser for AVAP code processing in chunk.py.
- Enhanced metadata extraction for processed documents, including AST for AVAP files.
- Improved error handling for AVAP code parsing.
2026-03-16 13:21:25 +01:00
pseco acc00adfaa Add AVAP execution and testing scripts
- Implemented parser for executing AVAP files within a Docker container (parser v1.py).
- Created a script to send AVAP code to a local server and handle responses (parser v2.py).
- Introduced a mock MBAP test harness to validate AVAP code against expected outputs (mbap_tester.py).
- Added transformation logic to convert AVAP code into Python-like syntax for testing purposes.
- Enhanced error handling and output formatting in the testing harness.
2026-03-12 15:56:36 +01:00
acano ed25f15542 feat: Enhance Elasticsearch ingestion process with metadata export
- Added output path parameter to elasticsearch_ingestion command for exporting processed documents.
- Implemented ElasticHandshakeWithMetadata class to preserve chunk metadata during ingestion.
- Updated process_documents function to include extra metadata for each chunk.
- Modified ingest_documents function to return Elasticsearch response for each chunk.
- Introduced export_documents function to save processed documents as JSON files.
2026-03-12 12:26:47 +01:00
pseco d04c149e66 workin on scratches bnf and parsing 2026-03-11 12:28:35 +01:00
pseco 3ac432567b BNF extraction pipeline from avap.md 2026-03-11 11:29:19 +01:00
acano a434d34676 Updated docs 2026-03-06 11:38:06 +01:00
acano 1549069f5a feat: Add Elasticsearch ingestion pipeline and document chunking functionality
- Implemented `elasticsearch_ingestion` function to handle document ingestion into Elasticsearch.
- Created `build_chunks_from_folder` function to read and clean text files, generating document chunks.
- Added logging for better traceability during the ingestion process.
- Updated `uv.lock` to include `boto3` as a new dependency.
2026-03-04 18:21:01 +01:00