The authors report that positional encoding fits on-axis 2D sinusoids well but performs worse at off-axis angles, whereas isotropic Gaussian RFF performs similarly well across angles. A separate visual inference from the plotted curves is that positional encoding is better than Gaussian at the on-axis angles, so this is not an unconditional Gaussian advantage. · CiteArk
The authors report that positional encoding fits on-axis 2D sinusoids well but performs worse at off-axis angles, whereas isotropic Gaussian RFF performs similarly well across angles. A separate visual inference from the plotted curves is that positional encoding is better than Gaussian at the on-axis angles, so this is not an unconditional Gaussian advantage.
