The self-attention mechanism of the Transformer can capture dependencies between any two positions in a sequence, and regardless of how far apart the tokens are, the computational cost of calculating attention weights is the same