Data transformation is the conversion of data into a format, structure, or representation suitable for analysis, indexing, modeling, or processing. In retrieval-augmented generation, it can include cleaning, validation, chunking, formatting, and preparation for embedding or search.
In the context of Retrieval Augmented Generation (RAG), data transformation refers to the conversion of data into a format suitable for indexing and processing by an LLM. This often involves splitting data into smaller chunks to make it manageable for embedding and indexing. Other transformations might include cleaning and validation steps to ensure data quality. For more information, see RAG Engine overview.