Data indexing is the process of organizing data or a knowledge base so that relevant information can be searched, retrieved, and used efficiently. In generative AI and retrieval-augmented generation, indexing may include metadata, chunking, embeddings, or vector search structures that help models access appropriate context.
In the context of generative AI, data indexing is the process of structuring and organizing a knowledge base to optimize search and retrieval. This involves creating an index, often called a corpus, which allows for efficient searching of the data. The process is separate from corpus creation, and the indexed data can be used to enrich the context of large language models (LLMs), reducing hallucinations and improving the accuracy of responses. For example, in a website context, data indexing might involve adding metadata like datePublished and dateModified to improve search functionality. Different methods exist for indexing data, including using vector search for similarity search in applications like retrieving relevant information for LLMs at query time. For more information, see RAG Engine overview.