RoPE frequency usage is determined by training data's dependency structure—frequencies scale inversely with dependency width. This explains why language models use mid-low frequencies and why frequency scaling enables length generalization when test contexts have similar patterns to training data.
This paper explains why transformer models use certain frequencies in Rotary Position Embeddings (RoPE) non-uniformly. The authors show that frequency selection matches the dependency structure in training data, with optimal frequencies inversely proportional to dependency width.