EY has unveiled a new retrieval‑augmented generation (RAG) framework that places multimodal knowledge graphs at the core of its design, aiming to boost the accuracy of large language model responses in enterprise settings.
Why traditional RAG falls short for many businesses
Standard RAG implementations focus on extracting text from documents and feeding it to a language model. In practice, corporate files often embed key data in charts, tables, engineering drawings, equations and other visual formats. EY’s research notes that overlooking these non‑text elements can leave large language models without critical context, leading to incomplete or misleading answers.
Dipanjan Sengupta, a distinguished technologist with EY’s Global Delivery Services Consulting unit, explained that “RAG works well for textual content, but in many industries, a lot of information is in illustrative content as well.” He cited examples such as an industrial firm storing specifications in engineering schematics and a life‑sciences company relying on complex graphs for experimental results.
How the multimodal approach works
The new framework separates textual and illustrative material into distinct ingestion pipelines. Text is broken into segments and enriched through keyword extraction and named‑entity resolution, linking different references to the same real‑world entity.
Related: Backlash grows, China narrows US lead, IBM falls
Illustrations receive descriptive metadata derived from existing captions, nearby text, bounding‑box analysis, optical character recognition and language‑model‑generated descriptions. Both types of content are indexed in separate vector stores, which narrows the search space and lets a query target text, images, or both. Each element then becomes a node in a knowledge graph, linked by weighted relationships that connect relevant passages and visual assets.
“We establish relationships between the various illustrations and the text segments,” Sengupta said. “We stored the information in a graph database where we had individual nodes for illustrations and for text and rules for establishing the relationships between the two.” The white paper outlines three techniques for building these links: deterministic keyword matching, semantic similarity using embeddings, and machine‑learning inference for implicit associations. A “gleaning” step fills missing connections, resolves ambiguous entities and surfaces related information across documents, producing a heterogeneous graph capable of multi‑hop reasoning without over‑burdening the language model.
Retrieval proceeds in stages. First, a similarity search runs against the modality‑specific index. The resulting identifiers guide a traversal of neighboring nodes in the knowledge graph. Depending on the query, the search may stay local, expand to broader graph communities, or blend both strategies.
A multimodal re‑ranker then orders the retrieved text and visuals before they are inserted into the LLM prompt. “We are not just relying on vector search but also enhancing the content space to other modes of information,” he added. “That gives us a larger narrative.”
Related: Intel touts 130 edge wins and new AI framework
The configuration differs from many existing RAG deployments. Compliance‑focused applications might favor narrow, deterministic retrieval, while research‑oriented uses could benefit from broader semantic exploration. EY suggests that chunking methods, embedding models, relationship‑building techniques, reranking strategies and retrieval scope should be configurable rather than hard‑coded.
From a practical standpoint, the platform approach allows common services—such as ingestion, security, governance and retrieval—to be reused across business units, each of which can tune the system to its specific needs.
One cautious observation: while the architecture promises richer answers by pulling together textual and visual evidence, the lack of publicly available benchmarks means enterprises will need to validate performance gains on their own workloads before fully committing.
EY argues that the framework will become increasingly relevant as organizations deploy AI agents that require up‑to‑date, domain‑specific information to make decisions. Weak retrieval can propagate errors throughout an automated workflow, a risk that multimodal RAG aims to mitigate.
Related: Amazon Web Services launches cost cutting tool
“When RAG is enriched with multimodal capability, we find that the agents are far better equipped to make decisions,” he said. The approach also appears to influence the path an agentic workflow follows, potentially improving overall efficiency.
The paper does not publish comparative benchmarks or quantify the accuracy improvements, relying instead on client experience and the logical premise that combined visual and textual evidence should yield more complete answers than text‑only search. Sengupta dismissed the notion that expanding model context windows would render RAG obsolete, noting that larger windows may reduce the need to split documents but do not solve the core problem of finding the most relevant evidence.
Without focused retrieval, a large language model is still searching for “that proverbial needle in the haystack,” according to the EY technologist.
Validation remains essential.
