Nvidia researchers created a way to reduce the expense and time needed to switch between large language models during extended AI conversations. Their approach, called cross-model KV cache transfer, uses basic linear math to move memory between models without requiring the new model to recompute the entire conversation from the beginning.
The issue arises when an AI system shifts a task from a small model to a larger one or vice versa. Each handoff forces the receiving model to rebuild its memory of the conversation, a step known as the prefill stage. This process scales with model size and input length, making it costly for businesses running multi-turn workflows.
When an LLM receives a prompt, it first completes the prefill stage, calculating keys and values for all input tokens and storing them in a Key-Value cache. Afterward, it enters the decode phase, generating new tokens one by one while reading from the cache. The cache prevents reprocessing the entire conversation history for each new token.
In long agentic sessions, the context expands with each turn. If the system changes models mid-session—sending a complex step to a larger model or switching to a smaller one to save money—the KV cache becomes unusable. Different models expect cache inputs in different formats, so the new model must recompute the prefill stage from scratch.
Related: TrueForge cuts AI task costs by thirds
The Nvidia team discovered that the KV cache follows a highly linear structure. Rather than training a new deep learning model to translate memory between models, they applied simple algebra. Their method avoids expensive gradient-based training or strict architectural requirements. The researchers designed a technique that removes positional encodings before translation, allowing it to work with sequences longer than its training data.
Most AI systems handle memory as a black box, but this technique treats it like a spreadsheet—columns of numbers that can be rearranged with basic math. That change in approach makes the process both fast and affordable.
Tests on compatible model pairs within the Qwen, Llama, or Ministral families showed strong results. For these pairs, the linear mapping process ran 2.7 to 25 times faster than recomputing the conversation while retaining up to 98% of the target model’s standalone accuracy.
Not every pairing succeeded. Some configurations posed challenges, with the linear mapper struggling outside its calibration data.
Cross-model KV cache transfer fits into a broader effort to address memory bottlenecks in enterprise AI. As models process longer documents, codebases, and reasoning tasks, managing the KV cache is becoming as important as the models themselves.
Related: Microsoft rolls out AI defense against cyber threats
Other solutions exist. Dynamic memory sparsification removes less important tokens to cut reasoning costs by up to 8x. Algebraic compression techniques like Attention Matching shrink the cache by 50x without reducing quality. Nvidia’s own KV Cache Transform Coding applies media compression concepts to reduce memory by 20x without changing model weights.
Optimizers such as IndexCache eliminate redundant layer calculations to speed up time-to-first-token in long-context applications. Meanwhile, models like DeepSeek and the GLM series are redesigning KV cache architecture from scratch.
The team sees their method as an early step. Future work may extend it to cross-family transfers or mismatched KV head counts. For now, it provides a practical way to lower inference costs as businesses scale multi-model systems.
Slack has integrated AI coding assistance into group chats, allowing teams to collaborate more efficiently.
