4 Commits

Author	SHA1	Message	Date
acano	bf3c7f36d8	feat(chunk): enhance file reading and processing logic ... - Updated `read_files` function to return a list of dictionaries containing 'content' and 'title' keys. - Added logic to handle concatenation of file contents and improved handling of file prefixes. - Introduced `get_chunk_docs` function to chunk document contents using `SemanticChunker`. - Added `convert_chunks_to_document` function to convert chunked content into `Document` objects. - Integrated logging for chunking process. - Updated dependencies in `uv.lock` to include `chonkie` and other related packages.	2026-03-10 14:36:09 +01:00
pseco	4afba7d89d	working on scrappy	2026-03-09 15:00:07 +01:00
acano	d951868200	refactor: Simplify Elasticsearch ingestion by removing chunk management module and integrating document building directly	2026-03-05 16:23:27 +01:00
acano	1549069f5a	feat: Add Elasticsearch ingestion pipeline and document chunking functionality ... - Implemented `elasticsearch_ingestion` function to handle document ingestion into Elasticsearch. - Created `build_chunks_from_folder` function to read and clean text files, generating document chunks. - Added logging for better traceability during the ingestion process. - Updated `uv.lock` to include `boto3` as a new dependency.	2026-03-04 18:21:01 +01:00