Multimodal / Computer Vision

Consistent Long Video and Audio Generation

Original: Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds

By Nan Duan, Haoyang Huang, Weiyang Jin, Haoran Li, Yaowei Li, Yuming Li, Yijun Liu, Xin Lu, Xiaoxiao Ma, Yanwen Ma, Yaofeng Su, Yilang Sun, Haoyu Wang, Zeyue Xue, Songchun Zhang, Junhao Zhuang Published 2026-08-24

Listen to the summary

Uses a voice available on your device

Speed Voice

On this page 5 sections

Key Takeaways

JoyAI-Echo-1.5 provides two purpose-built variants designed to balance visual quality and narrative persistence.
The world-model variant leads the WBench leaderboard with an average score of 81.7.
The system achieves high cross-shot consistency with a ViCLIP score of 0.8264.
Performance is validated across multiple metrics including Self-CIDS at 0.7937 and Voice scoring at 0.8524.

Summary & Methodology Analysis

JoyAI-Echo-1.5 is an audio-visual generation system that utilizes two distinct variants to manage long-horizon content generation. The world-model variant uses a geometry-aware conditioning pathway to translate heterogeneous navigation inputs into metric six-degree-of-freedom camera trajectories. To maintain stability, the architecture employs task-dependent audio weighting along with audio-specific adversarial and energy constraints, while leveraging short- and long-horizon self-gradient forcing to optimize performance during self-generated rollouts. The system also utilizes stochastic degradation on visual memory during training to enhance robustness against potential generation errors over extended sequences. By applying audio-visual teacher forcing, the system transforms a bidirectional backbone into a causal model, allowing for more stable, iterative output generation. The model achieves leading performance on the SANA-WM-Bench, demonstrating its capacity for maintaining visual quality and persistence throughout long-horizon tasks. It also holds the top position on WBench with a score of 81.7 and provides strong results across other standard metrics, including a ViCLIP score of 0.8264. Despite these results, the system is constrained by accumulated rotational drift during difficult long-horizon trajectories. This issue identifies a requirement for future research into more robust geometric consistency mechanisms and improved state representations to facilitate more reliable open-ended interactions.

Interactive System Flowchart

Click diagram to expand and zoom

Illustrative Implementation

A short sketch of the paper's core idea, not the authors' own code.

# Illustrative sketch (not from the paper)
import torch
import torch.nn as nn

class CrossShotMemory(nn.Module):
    def __init__(self, dim):
        super().__init__()
        # aggregated visual evidence + speaker cues
        self.mem = nn.Parameter(torch.zeros(1, dim))

    def update(self, feats):
        # simple running average over incoming features
        self.mem.data = 0.9 * self.mem.data + 0.1 * feats.mean(0, keepdim=True)

class WorldModel(nn.Module):
    def __init__(self, dim):
        super().__init__()
        # predicts metric 6‑DoF camera trajectory
        self.traj_head = nn.Linear(dim, 6)

    def forward(self, latent, traj):
        # geometry‑aware conditioning: inject trajectory into latent
        return latent + self.traj_head(traj)

def train_step(batch, memory, model, optimizer):
    # causal few‑step training with audio‑visual teacher forcing
    audio, video = batch["audio"], batch["video"]
    latent = model.encode(video)  # bidirectional backbone
    # Self‑Gradient Forcing (SGF) on self‑generated rollout
    rollout = model.decode(latent)
    loss = nn.functional.mse_loss(rollout, video)
    # stochastic degradation of visual memory for robustness
    if torch.rand(1).item() < 0.2:
        memory.mem.data *= 0.9
    loss.backward()
    optimizer.step()

// Illustrative sketch (not from the paper)
const torch = require('torch-js'); // placeholder for a PyTorch‑like lib

class CrossShotMemory extends torch.nn.Module {
  constructor(dim) {
    super();
    // aggregated visual evidence + speaker cues
    this.mem = torch.nn.Parameter(torch.zeros([1, dim]));
  }
  update(feats) {
    // running average over incoming features
    this.mem.data = this.mem.data.mul(0.9).add(feats.mean(0, true).mul(0.1));
  }
}

class WorldModel extends torch.nn.Module {
  constructor(dim) {
    super();
    // predicts metric 6‑DoF camera trajectory
    this.trajHead = torch.nn.Linear(dim, 6);
  }
  forward(latent, traj) {
    // geometry‑aware conditioning: inject trajectory into latent
    return latent.add(this.trajHead.apply(traj));
  }
}

function trainStep(batch, memory, model, optimizer) {
  // causal few‑step training with audio‑visual teacher forcing
  const { audio, video } = batch;
  const latent = model.encode(video); // bidirectional backbone
  // Self‑Gradient Forcing (SGF) on self‑generated rollout
  const rollout = model.decode(latent);
  const loss = torch.nn.functional.mseLoss(rollout, video);
  // stochastic degradation of visual memory for robustness
  if (Math.random() < 0.2) {
    memory.mem.data = memory.mem.data.mul(0.9);
  }
  loss.backward();
  optimizer.step();
}

Cross-Examination & FAQs

A deeper dive clarifying mechanics, constraints, and baseline evaluations.

Q1. What is the primary contribution of this research?

The authors introduced JoyAI-Echo-1.5, a unified system for audio-visual generation that uses two specialized variants to enable long-horizon storytelling.

Q2. How well does the model perform compared to existing benchmarks?

The world-model variant of JoyAI-Echo-1.5 ranks first on WBench with a score of 81.7 and achieves top-tier results on SANA-WM-Bench.

Q3. Is this system suitable for persistent world generation?

Yes, it is designed for persistent stories and interactive worlds, although it currently struggles with rotational drift over long trajectories.

Q4. What specific variants are included in the JoyAI-Echo-1.5 system?

The system features a long-video variant for cross-shot memory and a world-model variant for camera-pathway conditioning.

Q5. How does the model handle visual memory during the training phase?

The authors apply stochastic degradation to visual memory to improve the system robustness against accumulated errors.

Q6. What are the key performance metrics reported for this model?

Key reported metrics include a WBench average score of 81.7, a ViCLIP score of 0.8264, a Self-CIDS score of 0.7937, and a Voice score of 0.8524.

Q7. Does the model use any specific techniques to ensure audio stability?

Yes, it incorporates a task-dependent audio weight and uses audio-specific adversarial and energy constraints.

Q8. What is the main limitation regarding long-horizon performance?

The model faces difficulty with accumulated rotational drift on complex, long-horizon trajectories.

Q9. How were the training rollouts optimized?

The authors used short- and long-horizon self-gradient forcing to train the model on its own generated outputs.

Flag an issue