MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation

Every video token has a ray

Language models use positional encoding to tell words where they occur in a sequence. MeRoPE gives each video token a geometric position: when it occurs, where its camera ray starts, and where the ray points. It encodes relative camera motion without amplifying feature magnitudes, and adds geometric cues for potential matches across views.

A calibrated camera places each token’s ray in 3D: its origin follows the camera position, and its direction passes through the token’s image-plane location. The highlighted token also moves within the image plane; (u, v) are normalized image coordinates. The synchronized time axis adds the fourth coordinate. Motion is illustrative.

Alongside visual features, MeRoPE uses each token's time and calibrated camera ray. These geometric tags help the model relate observed tokens to the tokens it predicts across cameras and time. It does not need a separate 3D reconstruction model such as VGGT [1] or Depth Anything 3 [2] to produce point clouds or depth maps before generation. This keeps the hand-designed interface small: we provide the ray geometry, while correspondence and scene evolution are learned from data.

How geometry enters attention

UCPE [3] assigns a ray-local frame to every calibrated camera ray. The frame begins at the ray origin, and one axis follows the ray direction. MeRoPE uses the same ray-local representation and chooses the remaining frame orientation with a minimum rotation (MinRot). The paper gives the full construction and singularity analysis.

For a query token \(a\) and a key token \(b\), an operator \(\mathcal{U}_{ab}\) inserts their relative geometry into the attention score:

Here \(\mathbf{q}_a\) and \(\mathbf{k}_b\) are the query and key feature vectors, \(d\) is their dimension, and \(\mathcal{U}_{ab}\) encodes the geometric relation between the tokens. The geometric version differs from standard attention only by this operator.

GTA [4], PRoPE [5], and UCPE [3] use the homogeneous relative transform between the two frames as this operator:

The matrix \(R_{ab}\) and vector \(\mathbf{t}_{ab}\) rotate and translate from the key ray-local frame to the query ray-local frame. The vector \(\mathbf{0}\) is zero in three dimensions. Write the corresponding query and key channels as 3D vectors \(\mathbf{u}^{q},\mathbf{u}^{k}\) and scalars \(h_q,h_k\). Their contribution to attention is

The middle term depends directly on metric translation.

Why metric translation can dominate

Ray origins are measured in meters. Scaling the same motion by \(\lambda>0\) changes \(\mathbf{t}_{ab}\) to \(\lambda\mathbf{t}_{ab}\), so the translation term becomes

The term can grow with camera baseline even when visual similarity stays the same. After softmax, attention may favor a distant frame because of its distance rather than its visual content.

MeRoPE retains the metric relative pose but represents it with rotations, whose output norm does not grow with distance.

Encode where rays point and how cameras move

MeRoPE separates relative pose into ray-frame rotation and metric translation. Both become norm-preserving operators in attention: changing camera geometry changes their rotation angles while keeping feature lengths fixed.

Rotation · viewing direction

Translation · camera displacement

Add cues for where a cross-view match may lie

Relative pose describes camera geometry. For a static scene point on a key ray, its possible direction in the query view depends on its unknown depth. These candidate directions trace a purple spherical arc. MeRoPE samples anchors along this arc and encodes each anchor frame relative to the query ray frame, giving attention a geometric cue for potential correspondence.

Turn the key ray · reshape the arc

Move the anchor · change its frame

In these diagrams, u denotes \(\mathbf{u}_\infty\), the key-ray direction at infinite depth; e points from the query camera toward K.

The moving dot illustrates the anchor construction. The model uses a fixed set of angular fractions along each arc, encoding all of their relative rotations together as \(\mathcal{U}^{\mathrm{disp}}\). Visual content determines whether a valid correspondence exists; the anchors supply candidate geometry without requiring a predicted depth map.

Put the geometric cues together

MeRoPE has three camera-geometry blocks: anchor-frame rotations \(\mathcal{U}^{\mathrm{disp}}\), relative ray-frame rotation \(\mathcal{U}^{\mathrm{rot}}\), and metric translation RoPE \(\mathcal{U}^{\mathrm{trans}}\). A fourth block, \(\mathcal{U}^{\mathrm{native}}\), retains the backbone's time and image-coordinate RoPE.

The matrix \(I\) is the identity. Because every block is a rotation, \(\mathcal{U}_{ab}\) changes geometric phase without changing feature length. It encodes ray geometry and metric camera motion without allowing a longer baseline to amplify the query-key score.

Camera control across driving datasets

One MeRoPE model, trained with Wan2.2 TI2V-5B, controls camera motion on Waymo, Argoverse 2, and nuScenes. These selected held-out examples share the first nine observed frames and follow original, left-offset, or right-offset camera commands. See the paper for results on the full evaluation set.

Commanded pathRecovered camera pathOriginal path (BEV)

Generalization across camera optics

We also test the ray-based encoding on PanShot, the diverse-camera benchmark introduced with UCPE [3], using a Wan2.1 T2V-1.3B backbone [7]. The examples include indoor and outdoor scenes, low light, and very wide fields of view.

Real-to-Sim from historical revisits

Historical images act as visual anchors for generating the same place along a new camera path. Their visual tokens join the video tokens in self-attention, where MeRoPE encodes relative camera pose and ray geometry to help relate historical scene details to the new view.

nuScenes · Nine conditioning frames. Top: reference cameras and two generated paths. Bottom: four representative historical inputs, labeled by capture date; L / R indicates which path uses each view.

BibTeX

@article{qiao2026merope,
  title  = {MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation},
  author = {Qiao, Zhijian and Wang, Xinjiang and Chen, Jiajie and Huang, Haoming and
            Li, Meng and Chou, Chih-Chung and Wang, Jing and Shen, Shaojie},
  journal = {arXiv preprint arXiv:2609.01252},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.01252}
}