assistance-engine/scripts/pipelines/flows
acano bf3c7f36d8 feat(chunk): enhance file reading and processing logic
- Updated `read_files` function to return a list of dictionaries containing 'content' and 'title' keys.
- Added logic to handle concatenation of file contents and improved handling of file prefixes.
- Introduced `get_chunk_docs` function to chunk document contents using `SemanticChunker`.
- Added `convert_chunks_to_document` function to convert chunked content into `Document` objects.
- Integrated logging for chunking process.
- Updated dependencies in `uv.lock` to include `chonkie` and other related packages.
2026-03-10 14:36:09 +01:00
..
dataset_synthetic_generation.py working on scrappy 2026-03-09 15:00:07 +01:00
elasticsearch_ingestion.py feat(chunk): enhance file reading and processing logic 2026-03-10 14:36:09 +01:00
elasticsearch_ingestion_from_docs.py feat(chunk): enhance file reading and processing logic 2026-03-10 14:36:09 +01:00
test_chunker.ipynb feat(chunk): enhance file reading and processing logic 2026-03-10 14:36:09 +01:00