MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation

Every video token has a ray

A language model uses positional encoding to tell words where they sit in a sequence. MeRoPE gives video tokens their geometric positions instead: where each camera ray starts and where it points.

The small x/y/z axes describe 3D space, t indexes time, and H is the prediction horizon. Together they form a 4D space-time coordinate. Every token is tagged by its time and by the origin and direction of its camera ray.

Given only calibrated camera rays, attention can learn from data how to relate tokens across cameras and time. No external 3D reconstruction model—such as VGGT [1] or Depth Anything 3 [2]—is needed to first produce point clouds or depth maps as guidance for generation.

How geometry enters attention

UCPE [3] turns every calibrated camera ray into a ray-local frame: its origin is where the ray starts, and one axis follows the ray direction. MeRoPE keeps this useful abstraction but refines how the frame orientation is chosen using a minimum rotation (MinRot). The exact construction and singularity analysis are given in the paper.

Once every token has a local frame, the relative geometry between a query token \(a\) and a key token \(b\) can be inserted directly into their attention comparison:

Here \(\mathbf{q}_a\) and \(\mathbf{k}_b\) are the query and key feature vectors, \(d\) is their dimension, and \(\mathcal{U}_{ab}\) describes the geometric relation between the two tokens. Standard attention uses the left dot product; geometric PE adds only the relative operator shown on the right.

Homogeneous geometric PEs such as GTA [4], PRoPE [5], and UCPE [3] place the relative transform between two frames directly inside this operator:

The matrix \(R_{ab}\) and vector \(\mathbf{t}_{ab}\) are the rotation and translation from the key ray-local frame to the query ray-local frame, and \(\mathbf{0}\) is a three-dimensional zero vector. Splitting the corresponding query and key channels into 3D vectors \(\mathbf{u}^{q},\mathbf{u}^{k}\) and scalars \(h_q,h_k\) exposes what reaches the attention score:

The middle term shows the central issue: metric translation changes the attention score directly.

Why metric translation can dominate

Ray origins are measured in meters. If the same motion is scaled by a factor \(\lambda>0\), then \(\mathbf{t}_{ab}\) becomes \(\lambda\mathbf{t}_{ab}\), and the translation term above becomes

Its magnitude can therefore grow with camera baseline even when visual similarity does not. After the softmax, this unbounded geometric term can pull attention toward a distant frame simply because it is far away.

MeRoPE keeps the same metric relative-pose information, but represents it with rotations whose output norm cannot grow with distance.

Replace metric translation with rotations

The relative rotation \(R_{ab}\) between ray-local frames is already norm preserving, so MeRoPE keeps this rotation PE and changes only the translation part. Let \(C_a\) map the query camera coordinates into the world frame, and let \(\mathbf{o}_a\) and \(\mathbf{o}_b\) be the two ray origins. MeRoPE first expresses their displacement in the query camera:

The factor \(C_a^\top\) depends on the query, so subtracting two independent world-frame phases cannot recover this relative translation. MeRoPE computes it first, then encodes each coordinate with multi-frequency 2D RoPE rotations [6]:

The index \(c\) selects one of the three translation coordinates, \(K\) is the number of frequencies, \(k\) indexes them, and \(\omega_k\) is the corresponding angular frequency. The direct-sum symbol \(\oplus\) places these independent 2D rotations on the diagonal. Distance now changes rotation angle \(\theta\), not feature magnitude.

Geometry suggests where a match may lie

In principle, the pose PE above already achieves our primary goal: every token is represented by a calibrated ray. In practice, multi-view geometry in static scenes provides an additional cue that can further improve camera-pose controllability.

Consider the key ray in the figure below. As the unknown scene depth varies, its possible matching direction in the query camera moves along the purple spherical arc, from \(\mathbf{u}_\infty\) at infinite depth toward the epipole \(\mathbf{e}\). MeRoPE samples a few bounded angular anchors—the purple dots—along this arc. These anchors act like interpolation knots rather than mutually exclusive guesses: their joint responses give the network a coarse coordinate system along the continuous arc, from which it can infer intermediate disparities. Each anchor direction is turned into a local frame in the same way as a token ray, and the rotation from the query ray frame to that anchor frame becomes one positional block.

The rotations induced by all sampled anchors form the disparity component \(\mathcal{U}^{\mathrm{disp}}\).

Put the geometric cues together

We now have three geometric components: anchor-frame rotations \(\mathcal{U}^{\mathrm{disp}}\), the retained relative ray-frame rotation \(\mathcal{U}^{\mathrm{rot}}\), and the new metric translation RoPE \(\mathcal{U}^{\mathrm{trans}}\). MeRoPE places them beside the backbone's native time and image-coordinate RoPE, denoted by \(\mathcal{U}^{\mathrm{native}}\).

The identity matrix \(I\) in the second equation makes the key property explicit: every block is a rotation, so \(\mathcal{U}_{ab}\) changes geometric phase without changing feature length. MeRoPE can therefore encode both ray geometry and metric camera motion without letting a longer baseline amplify the query-key score by construction.

Camera-controlled video generation on nuScenes

We apply MeRoPE to a Wan2.2 TI2V-5B predictive video model [7] on the autonomous-driving dataset nuScenes [8]. Each video places the commanded and recovered camera paths beside the generated frames. The predictions follow the requested motion while preserving coherent roads, vehicles, and surrounding structure. Quantitative comparisons and component ablations are reported in the paper.

Generalization across camera optics

The same ray-based encoding also applies to PanShot, a benchmark for diverse camera optics introduced with UCPE [3], using a Wan2.1 T2V-1.3B backbone [7]. These videos span indoor and outdoor scenes, low light, and very wide fields of view.

Grounding video prediction in the real world

A video model usually sees only the current clip. As prediction moves beyond its observed field of view, the model may lose the identity and layout of the real location.

A historical image here is a calibrated image retrieved from another traversal of the same location, possibly captured by another vehicle, at another time, or under different weather. It is not a record of the current dynamic state: moving objects need not match. Instead, it provides real observations of persistent scene structure, such as roads, buildings, and vegetation.

History SA places the retrieved image directly in temporal self-attention, while MeRoPE tells the model how its rays relate to the rays being predicted. The image can therefore act as a long-horizon visual anchor. On nuScenes, only History SA, rather than History CA, preserves retrieval-supported static structure over time, including when the retrieved and target traversals have different weather.

BibTeX

@article{qiao2026merope,
  title  = {MeRoPE: Metric Rotary Position Embedding for Camera-Controlled Video Generation},
  author = {Qiao, Zhijian and Wang, Xinjiang and Chen, Jiajie and Huang, Haoming and
            Li, Meng and Chou, Chih-Chung and Wang, Jing and Shen, Shaojie},
  year   = {2026}
}