Geordy, thanks for the reply!
To begin with, I want to emphasize that our solution is only a quick and practical workaround. However, it is still a significant improvement from your solution. The correct approach to RAG with lengthy articles involves breaking the text into smaller segments and embedding these segments separately (Semantic Chunking).
Regarding your points:
The reason is that post content isn’t always clean, readable text. If you’re using page builders, for example, the content is often full of shortcodes and pseudo-code that wouldn’t work well for retrieval.
Numerous tools exist for cleaning HTML or WordPress code, and I believe it’s not a significant issue. I built an add-on for AI Engine that enhances chatbots with Google Search capabilities. This add-on scrapes the complete content from external sites and then inserts a cleaned version into the chatbot’s context. If we can accomplish this with unfamiliar third-party content, you can definitely do it with your own content as well. All model providers and AI apps like BoltAI do this when they integrate search or add context.
Also, articles can be long, repetitive, or contain details that you might not want to expose (like specific names). The goal of context in RAG is to be optimized—concise, relevant, and free of unnecessary repetition. That’s why AI Engine processes the content, cleaning and rewriting it to be shorter and more effective.
The goal for RAG is certainly not to provide concise context. Why do you believe Gemini supports a context window for up to 2 million tokens?
The goal of RAG is to provide as much relevant context as possible, and it is not the responsibility of the embedding model to ensure the relevancy of the content that used during retrieval. You don’t want to lose information by filtering it before it even enters your RAG solution. For example, many of our articles include code that gets entirely lost if you summarize them.
In reality, a repetition of 10-20% is crucial to maintain overlap when employing semantic junking.
The model can handle contexts containing repetitive or irrelevant information by simply disregarding such elements. This is why Perplexity or ChatGPT Search is effective. You need to avoid giving the model concise, incomplete context. It’s the model’s responsibility during inference to sift through the content.
You can simply disable the AI rewriting option in the Embeddings section and use the mwai_post_content filter to customize how content is retrieved
Disabling the AI rewriting option in the Embeddings section isn’t a good idea. We use 1536-dimensional embedding vectors, so the token count should stay between 400–800. Increasing the number of tokens leads to semantic dilution.
So, as I mentioned earlier, the best way to tackle this issue is through semantic junking, but that would mean making substantial changes to your code. A practical workaround is to use the embedding model to generate the optimal embedding vector with minimal semantic loss, as these models are built for this purpose. Then, retrieve the full content for context to ensure no important information is missed during inference.
I am unsure how to use mwai_post_content to retrieve all the embedding IDs and their linked posts. I guess it’s feasible if we combine it with other filters.
To sum up, your embedding solution definitely has a design flaw. I post this because AI Engine is by far the best and most important WordPress plugin I’ve seen in decades. This flaw is an eyesore.
