Visual token importance changes as the model processes information deeper—tokens deemed unimportant early on may matter later, so recoverable routing outperforms permanent removal for vision-language tasks.
Vision-language models use thousands of visual tokens, making inference slow. Instead of permanently removing low-scoring tokens, this paper proposes Reroute: tokens can be temporarily skipped and re-evaluated later when they become important. The method works with existing token-reduction techniques and improves performance on grounding tasks without extra computational cost.