New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.
New NVIDIA paper says AI models do not always need huge amounts of past conversation history (context) to work well. They show that regularly shortening the history (compaction) keeps accuracy almost the same on hard, long tasks.
What have people found the best compaction scheme/harness to be for relatively short context? I’ve been trying to set up an agentic workflow and context management seems like the biggest source of jank and errors. Reassuring to see a paper like this though since the amount of context I can use and still have everything in vram is not that high.