Would convolution filters provide better position encoding than the traditional cos/sin functions with k,n,i and d? My thoughts are that the traditional position encoding functions introduce too much noise into the network, with essentially random values at low values of i. I would think that a convolution filter would be more useful for processing contextual information. Are there any resources out there for implementing this sort of idea?
Asked
Active
Viewed 16 times
0
-
Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. – Community Oct 09 '22 at 07:54