MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation

Every video token has a ray

Language models use positional encoding to tell words where they occur in a sequence. MeRoPE gives each video token a geometric position: when it occurs, where its camera ray starts, and where the ray points. It encodes relative camera motion without amplifying feature magnitudes, and adds geometric cues for potential matches across views.

The x/y/z axes define 3D space, t indexes time, and H is the prediction horizon. Each token carries its time together with the origin and direction of its camera ray.

Alongside visual features, MeRoPE uses each token's time and calibrated camera ray. These geometric tags help the model relate observed tokens to the tokens it predicts across cameras and time. It does not need a separate 3D reconstruction model such as VGGT [1] or Depth Anything 3 [2] to produce point clouds or depth maps before generation. This keeps the hand-designed interface small: we provide the ray geometry, while correspondence and scene evolution are learned from data.

How geometry enters attention

UCPE [3] assigns a ray-local frame to every calibrated camera ray. The frame begins at the ray origin, and one axis follows the ray direction. MeRoPE uses the same ray-local representation and chooses the remaining frame orientation with a minimum rotation (MinRot). The paper gives the full construction and singularity analysis.

For a query token \(a\) and a key token \(b\), an operator \(\mathcal{U}_{ab}\) inserts their relative geometry into the attention score:

Here \(\mathbf{q}_a\) and \(\mathbf{k}_b\) are the query and key feature vectors, \(d\) is their dimension, and \(\mathcal{U}_{ab}\) encodes the geometric relation between the tokens. The geometric version differs from standard attention only by this operator.

GTA [4], PRoPE [5], and UCPE [3] use the homogeneous relative transform between the two frames as this operator:

The matrix \(R_{ab}\) and vector \(\mathbf{t}_{ab}\) rotate and translate from the key ray-local frame to the query ray-local frame. The vector \(\mathbf{0}\) is zero in three dimensions. Write the corresponding query and key channels as 3D vectors \(\mathbf{u}^{q},\mathbf{u}^{k}\) and scalars \(h_q,h_k\). Their contribution to attention is

The middle term depends directly on metric translation.

Why metric translation can dominate

Ray origins are measured in meters. Scaling the same motion by \(\lambda>0\) changes \(\mathbf{t}_{ab}\) to \(\lambda\mathbf{t}_{ab}\), so the translation term becomes

The term can grow with camera baseline even when visual similarity stays the same. After softmax, attention may favor a distant frame because of its distance rather than its visual content.

MeRoPE retains the metric relative pose but represents it with rotations, whose output norm does not grow with distance.

Encode where rays point and how cameras move

MeRoPE separates relative pose into ray-frame rotation and metric translation. Both become norm-preserving operators in attention: changing camera geometry changes their rotation angles while keeping feature lengths fixed.

Rotation · viewing direction

Translation · camera displacement

Add cues for where a cross-view match may lie

Relative pose describes camera geometry. For a static scene point on a key ray, its possible direction in the query view depends on its unknown depth. These candidate directions trace a purple spherical arc. MeRoPE samples anchors along this arc and encodes each anchor frame relative to the query ray frame, giving attention a geometric cue for potential correspondence.

Turn the key ray · reshape the arc

Move the anchor · change its frame

In these diagrams, u denotes \(\mathbf{u}_\infty\), the key-ray direction at infinite depth; e points from the query camera toward K.

The moving dot illustrates the anchor construction. The model uses a fixed set of angular fractions along each arc, encoding all of their relative rotations together as \(\mathcal{U}^{\mathrm{disp}}\). Visual content determines whether a valid correspondence exists; the anchors supply candidate geometry without requiring a predicted depth map.

Put the geometric cues together

MeRoPE has three camera-geometry blocks: anchor-frame rotations \(\mathcal{U}^{\mathrm{disp}}\), relative ray-frame rotation \(\mathcal{U}^{\mathrm{rot}}\), and metric translation RoPE \(\mathcal{U}^{\mathrm{trans}}\). A fourth block, \(\mathcal{U}^{\mathrm{native}}\), retains the backbone's time and image-coordinate RoPE.

The matrix \(I\) is the identity. Because every block is a rotation, \(\mathcal{U}_{ab}\) changes geometric phase without changing feature length. It encodes ray geometry and metric camera motion without allowing a longer baseline to amplify the query-key score.

Camera-controlled video generation on nuScenes

We test MeRoPE with a Wan2.2 TI2V-5B predictive video model [7] on nuScenes [8]. Each video shows the commanded and recovered camera paths with the generated frames. The predictions follow the requested motion while keeping the roads, vehicles, and surrounding scene coherent. The paper reports quantitative comparisons and component ablations.

Generalization across camera optics

We also test the ray-based encoding on PanShot, the diverse-camera benchmark introduced with UCPE [3], using a Wan2.1 T2V-1.3B backbone [7]. The examples include indoor and outdoor scenes, low light, and very wide fields of view.

Real-to-Sim with retrieved images

MeRoPE also supports a simple form of real-to-sim generation: a calibrated image retrieved from the same location anchors a simulated future rollout. The image may come from another traversal, time, or weather condition, so it anchors the persistent scene layout rather than reproducing the same moving objects.

History SA inserts the retrieved image into temporal self-attention. MeRoPE supplies the geometry between its rays and the rays being predicted, allowing the image to remain a visual anchor over a long rollout. On nuScenes, History SA preserves static structure supported by the retrieval, while History CA fails to retain it over time. The same pattern appears when the retrieved and target traversals have different weather.

BibTeX

@article{qiao2026merope,
  title  = {MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation},
  author = {Qiao, Zhijian and Wang, Xinjiang and Chen, Jiajie and Huang, Haoming and
            Li, Meng and Chou, Chih-Chung and Wang, Jing and Shen, Shaojie},
  journal = {arXiv preprint arXiv:2609.01252},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.01252}
}