360-degree video catches AI scene models losing track

A convincing AI-generated video can hide a basic failure: the system may discard a person as soon as the camera turns away. That distinction matters if video tools, augmented-reality systems or future robotic components are expected to maintain a stable scene rather than merely produce a plausible next frame.

PersistBench turns the problem into a remarkably clean test. Start with one 360-degree recording. Crop it into two synchronized ordinary-looking videos: an input view that turns away from a target, and a reference view that keeps watching the target. The model receives only the first video, then is asked to render the camera path of the second. Because both crops came from the same recording, the second supplies evidence of what actually happened off-screen—not merely what might have happened. The paper describes this construction, while the project page shows the paired views.

That is more demanding than asking whether the generated background looks reasonable. A person who walked out of view must still exist in the right place, continue along the right path and remain recognizably the same person.

The researchers divide those demands into three scores:

• Object permanence: Does the expected object appear when the model is queried from a view where it should be visible? A tracker and a separate visual judge must both find it.

• Motion continuity: Is the generated object near its recorded position? The score compares the centers of predicted and reference masks, adjusting for the object’s size.

• Appearance preservation: Does the object retain its visual identity? The benchmark compares image features inside the predicted and reference masks rather than demanding identical pixels.

These are operational measurements, not evidence that a model remembers or forgets in the human sense. They test rendered outputs from dynamic-scene systems—three-dimensional environments changing over time—not an inner experience.

Looking good is not staying present

The preprint evaluates 12 models from three families: systems that reconstruct changing 3D scenes, systems that generate video along a requested camera path, and systems that expand a narrow video into a panorama. Every model with comparable results performed worse after the target left the input view, according to the authors’ results page.

The static-object table makes the drop tangible. Argus detected the target in 99.25% of visible-segment frames and 84.06% of invisible-segment frames. CUT3R fell from 66.43% to 3.14%. Those numbers measure permanence, not overall video quality.

Different scores expose different failures. For static objects, 4DGT’s motion-continuity score declined from 93.03% while visible to 70.34% while hidden, yet its permanence plunged from 97.00% to 16.03%. In other words, the surviving tracked outputs could remain reasonably positioned even though the object frequently failed to survive the test at all. GEN3C, meanwhile, retained 82.61% for hidden-segment appearance against 93.26% while visible.

A small calculation helps summarize the table without pretending the models share one master grade. Excluding HyDRA, which lacks visible-segment results, subtract each hidden score from its visible counterpart and take the median across the remaining 11 models. For static objects, the median drops are 25.42 percentage points for permanence, 20.82 for continuity and 24.76 for appearance. Readers can reproduce the calculation directly from the displayed table: compute the 11 differences in a column, sort them and select the sixth.

The authors also report that models generally perform better on static than moving objects. That is unsurprising but important: keeping a parked motorcycle in roughly the same place is a smaller demand than extrapolating a walker’s changing position while the camera faces elsewhere.

A benchmark, not a robot trial

The word “memory” can make this result sound broader than it is. One evaluated model, CUT3R, was introduced in a peer-reviewed paper as a recurrent reconstruction system whose internal state updates with new images and can be queried from virtual viewpoints. Its CVPR description supports treating it as a scene-reconstruction model. PersistBench does not test a complete robot sensing, planning and acting in the physical world.

The benchmark itself is an arXiv preprint and has not yet passed peer review. Its measurements also inherit several uncertainties. Object masks are generated automatically rather than verified by human annotators. Camera-pose estimation, tracking and visual scoring can introduce errors. The source videos are not a uniform sample of the world, and converting spherical footage into flat views can add distortions. Motion presents a deeper conceptual problem: several futures may be plausible, but a reference-based score rewards the one trajectory that happened in the recording.

There is also a version detail worth keeping visible. The reported evaluation uses 2,000 balanced pairs—1,000 static and 1,000 dynamic. The current dataset card lists 2,086 pairs: 1,064 static and 1,022 dynamic. The release count should not silently replace the experiment count when describing the paper’s results.

PersistBench’s useful trick is the second camera crop. Plausibility is easy to admire when the truth is outside the frame; a synchronized 360-degree reference makes it possible to check whether the generated scene kept the right object, in the right place, looking like itself. For systems meant to represent a changing world, seeing once is plainly not enough.