Segment-Augmented Long Video Assistant for Targeted Retrieval and Routing in Long-Form Video Analysis (2024)

Video Source & Filtering.

For the long and untrimmed video sources, we primarily focus on three key aspects to build densely captioned video dataset: (i) extensive video length with diverse video source categories, (ii) high-quality video contents,(ii) frequent scene transitions within each video. Accordingly, our data collection is mainly sourced from YouTube, ensuring rich dynamic content that better reflects real-world complexities experienced by global users—here, because our main goal for video gathering is on complex scene understanding, we exclude low-quality and user-uploaded aesthetic videos (e.g., WebVid, Pixabay, Pexels, and etc,.) that are rather beneficial for video generation tasks, despite their merits for easy collection. We have collected YouTube urls from[27] and downloaded the whole video in untrimmed states. Among the total $32$ coarse and diverse video categories YouTube API provided, we selectively curated $10$ categories, excluding categories such as News & Politics, Classics, and Documentary, due to their static nature, which provides only sparse temporal information in the videos. We further supplemented the dataset with additional Movie & Drama videos sourced from[53, 21], totaling $87,867$ video sources with $11.87$ Khrs video duration (avg. $486.5$-seconds).

Segmenting Video into Clips.

Next, for the collected long and untrimmed video sources, we cut the lengthy videos into small segments to densely caption the entire video in next phase. Instead of adopting the bottom-up approach used in the ShareGPT4Video dataset[8], which segments videos into fixed time intervals (2-seconds) in advance and then merges adjacent frames based on their CLIP similarity[48], we directly employed PysceneDetect¹¹1We use AdaptiveDetector with default setup in https://github.com/Breakthrough/PySceneDetect to segment the videos, dynamically adjusting the threshold based on the raw-level video information to reliably detect scene changes. At the end, the total number of $1.29$M of video segments with $33.11$-seconds average video length is extracted from the original video sources.

Dense Segment Captioning.

After obtaining the massive video segments, our next goal is to caption each segment with visual details and narrative context to capture the scene-specific explanations, which can enrich scene-level interpretation. To achieve this, we plan to utilize pre-trained LMMs to generate detailed descriptions for the partial video segments. As the captioner, we empirically found that VILA-1.5 (13B)[36] shows competent descriptive quality than other open-sourced models, and used to generate dense captions for each video segment with randomly sampled instructions for detailed descriptions. As a result, we acquire $1.29$M pairs of detailed descriptions corresponding to the video segments, each description with average $137.5$ word length. Please see instruction details and qualitative examples of generated captions in Appendix A.

Scoring Video-Text Correspondence.

Lastly, we score the correspondence between the video segments and the paired dense descriptions, which will later be used as explicit supervision to robustly train our retrieval framework. What we must not overlook here is that the paired video-text relationship is not solely a one-to-one correspondence but is more akin to generalized bipartite matching. That is, within the long and untrimmed video source, each video segment can be connected to other descriptions with additional edges. Therefore, for the $N_{v}$ number of video segments and their paired segments, we can construct a $\{N_{v}\}^{2}$ correspondence matrix between video-text (V2T). To measure each correspondence, we employ LanguageBind[73] due to its competitive alignment capabilities across various modalities. In addition, we build another $\{N_{v}\}^{2}$ matrix to provide a doubly robust measure for the correspondence scores among adjacent descriptions (T2T) by comparing similarity within the textual context using the SBERT model[54].

4.1.1 Spatio-Temporal Connector

The first component of our model, Spatio-Temporal Connector, efficiently handles long and variable-length input video segments by extracting each segment’s visual semantics in a fixed-size latent vector. As illustrated in Fig.2(b), we first sample video frames at 1 FPS from each video segments, then visual features are acquired with $2\times 2$ pooling (thus, $144$ tokens from each frame). After that, the visual features are flattened and fed into the ST-Connector with additional positional and temporal encoding. Here, when the long video is processed, the number of unfolded patch tokens becomes extremely large, leading to exhaustive computations. To address this, we employ a dynamic token drop technique to reduce computational load.

4.1.2 Segment Retrieval Router

Next, the key to conveying the pertinent video information to LLMs is retrieving relevant video segments by querying sentence. To densely cue the similarities between the video and sentence information, we introduce a routing framework, Segment Retrieval Router, which consists of 2-layer Transformer as illustrated in Fig.2(c). After obtaining the routing tokens $R{=}\{R_{i}\}^{N_{v}}\in\mathbb{R}^{N_{v}\times D}$ from entire video segments, we aggregate them and feed into the SR-Router as queries. For the given sentence, we employ the same text encoder used for the vision encoder and project it into the shared embedding space to obtain sentence features $S\in\mathbb{R}^{N_{t}\times D}$, where $N_{t}$ indicates textual length.

Using the cross attention mechanism (q: $R$; k/v: $S$), we can estimate similarity scores between the video segments and given sentence queries (i.e., V-T similarity). The scores enable the SR-Router to prioritize and select the most relevant video segments that align with the sentence query.

4.1.3 FocusFast Pathways: Integration to LLMs

Using the routing tokens, we can calculate the similarities of each video segment for the given query. Leveraging the similarities, SALOVA efficiently retrieves the specific video features that exhibit the highest relevance score to the textual query, where the indexed video features are then directly integrated into the LLM architecture. Here, extending the SlowFast pathways concept[19], we present the FocusFast mechanism to effectively manage the processing pathway for the retrieved video segments: (i) Focus pathway concatenates the top-K most pertinent features to construct a comprehensive video representation, capturing local details across retrieved segments and enabling detailed interactions with textual queries to enhance handling complex video information. (ii) Fast pathway focuses on the more broader-level context by employing segment-wide routing tokens as the condensed global representation. It effectively contains dynamic spatio-temporal changes throughout the video stream, thereby allowing SALOVA to understand the overall video content and scene-level continuity awareness.

Once the most pertinent features are retrieved, they are delivered to the LM backbone for the final processing as in Fig.2(a), integrating video-specific details into the models’ responses. By effectively handling long and untrimmed videos with the proposed retrieval and routing mechanism, SALOVA can maintain the flow of salient information without the processing overhead for less related data, thus generating more context-aware responses.

Stage 1: Cross-modality Alignment.

For the initial step in modality alignment, we utilize $790$K image/video-text paired dataset: (i) $558$K image-text pairs from the CC3M dataset[52], filtered by LLaVA[38] and (ii) video-text pairs sampled from the WebVid $2.5$M subset[4]. We freeze vision encoder and LLMs during the training, and mainly focus on optimizing the connector and router to map the visual information into the textual space.

Stage 1.5: Long Video Knowledge Injection.

As an intermediate training step, we use the SceneWalk dataset to train the SALOVA, unfreezing all trainable parameters except for the vision encoder. During training, we input the long and untrimmed video instances and follow the processing pipeline shown in Fig.2. By training the model with densely captioned video-description pairs, it acquires high-quality parametric knowledge of both spatial and temporal information. In addition, through the aforementioned retrieval process, the model learns to target video segments that are mostly relevant to the video description.

Stage 2: Video Instruction Tuning.

To possess QA capabilities in SALOVA, we use extensive video instruction-tuning data as the final training step. The instruction data are mainly sourced from four different datasets: LLaVA-Video-178K[71], NeXT-QA[58], ActivityNetQA[66], and PerceptionTest[47]—comprising a total of $1.4$M video-instruction QA data, including caption entries, open-ended QA, and multiple-choice QA. Note that we train all the network parameters during this stage and auto-regressively update the instruction-following assistant’s response for the next word prediction.

Implementation.

For the vision encoder and text encoder of the SR-Router, we utilize the CLIP-ViT-L model[48] with a resolution size of $336$. We employ a 2-layer transformer with a head size of 2 for the ST-Connector, which has a latent dimension of $256$. The token drop mechanism is dynamically applied according to video length, with varying maximum drop rates for each training stage—Stage 1 has no token drop, Stage 1.5 up to $0.7$, and Stage 2 up to $0.4$. For the configuration of SR-Router, we set 2-layer of transformers with a single head, and top-K number is set to $5$ during the stage 2. Following[37, 30], the projector layer consists of 2-layer MLP with GELU. Our LLM backbones are (i) 3B: Llama-3.2-3B[18], (ii) 3.8B: Phi-3.5-mini[1], and (iii) 7B: Qwen2.5-7B[63].

Training Details.

For the each training stage, we train SALOVA for $1$ epoch with 1 node of 8 A100 GPUs. The total training hours for 3B/3.8B and 7B models roughly take 5 and 7 days, respectively. We employ FlashAttention-2[15], gradient checkpointing[9], and ZeRO-2[49] to minimize the memory footprint associated with model components (i.e., gradient, activation, and optimizer states). Additionally, we fine-tune the trainable parameters at each step without employing LoRA[24]. For the extended training configuration, we have attached the details in Appendix C.

Evaluation Benchmarks.

We evaluate our model using two types of video analysis benchmarks—long video understanding and general video understanding, categorized based on the video length. For the long video benchmark, we primarily utilize Video-MME[20] and LongVideoBench[57], both of benchmarks includes videos up to two hours long duration. As the general video analysis evaluations, we employ various benchmarks such as ActivityNetQA[66], VideoChatGPT[41], and MVBench[32]. Note that the same pipeline is used to obtain video segments for each benchmark, and all benchmarks are sampled at 1 FPS without token drop during inference. As a comparison baseline, considering academic budget constraints, we evaluate against models that have similar parameter size.

Results on Long Video Understanding.

Video-MME[20] evaluates LMMs with a focus on video analysis across a variety of video types and durations. We primarily compare the benchmark results in settings without subtitles, relying solely on video frames. Therefore it can assess the LMMs’ visual comprehension capabilities rigorously, based purely on visual content. Also, LongVideoBench[57] is designed to assess LMMs’ understanding of long-duration videos up to two hours. It includes a diverse collection of videos, challenging the models’ ability to process and interpret extensive visual and contextual information across a variety of themes. As shown in Tab.1, our model shows competent video understanding performance across all video length distributions in Video-MME and lengthy video instances in LongVideoBench. Notably, we highlight that SALOVA achieved significant performance in the medium (average 562.7 seconds) and long (average 2385.8 seconds) length categories in Video-MME benchmark, even with more smaller size of backbone LM parameters compared with the baseline models.

Such performance gains in long video instances are attributed to our model’s dynamic capability to retrieve and process only the relevant video segments, enabling it to handle lengthy video content efficiently without being constrained by the limited context length. Especially, the routing mechanism in SALOVA strategically prioritizes video segments that are likely to contain crucial visual and contextual cues relevant to the query. This selective routing mechanism reduces the computational load and minimizes the information loss that commonly occurs in current video-LMMs trying to process extensive video data in entirety.

Results on General Video Understanding.

Using benchmarks such as ActivityNetQA[66], VideoChatGPT[41], and MVBench[32], SALOVA was evaluated across various video types to assess its general video understanding capabilities. As shown in Tab.2, SALOVA demonstrated competent performance, comparable to existing video-LMMs, especially in dynamic and shorter video sequences. On ActivityNetQA, the model effectively utilized its segment retrieval strategy to provide focused and contextually appropriate responses, which helped maintain accuracy. This approach was similarly effective in the multi-modal settings of VideoChatGPT and MVBench, where SALOVA showed consistent performance in handling dialogues and visual cues. These outcomes highlight SALOVA’s capability to process general video content efficiently through its dynamic routing mechanism, offering a reliable solution that balances computational resources with output quality.

Ablation Study.

We conduct ablation studies on three components as follows: (i) different video frame sampling strategies, (ii) intermediate training stage for long video knowledge injection, and (iii) the FocusFast mechanism to understand branched local-global representation in videos.

As shown in Table 3, we first observe that using more frames significantly enhances performance, particularly in understanding long-form videos. This aligns with our key insights on managing long videos, suggesting that a higher frame count can provide more spatio-temporal information and improve the model’s response without losing contextual information within the video. Additionally, we compare with a baseline trained with stage 1-2 (skipping stage 1.5). Here, we highlight the effectiveness of the SceneWalk dataset as an intermediate training step to enhance parametric knowledge for the long video analysis by allowing the model to learn from high-quality and densely captioned scene-level information, which is crucial for adapting to various lengths and contexts. Lastly, we conduct an analysis on the FocusFast method and demonstrate its efficacy in analyzing not only local details from relevant video segments but also in understanding the global video context through the simultaneous use of routing tokens, thereby facilitating a more comprehensive understanding of video content.

Analysis of Retrieving Segments.

By retrieving relevant video segments for the given queries, SALOVA can effectively target salient information in the long video and retain long context information. To further demonstrate the model’s targeting capabilities beyond numerical performance in long video analysis, we explore our model’s application in the Visual Needle-In-A-Haystack (V-NIAH) task[70], which extends the Needle-in-a-Haystack (NIAH) evaluation for LLMs to a vision-level benchmark. This task is particularly challenging as it requires models to not only detect but also precisely retrieve the sparse yet crucial visual cues scattered across lengthy videos.

As shown in Fig.3, we compare our model to a baseline trained on sparsely sampled frames (16 frm, without SR-Router). Our framework effectively identifies and extracts relevant video segments from densely packed content, even when handling long context lengths. These results highlight SALOVA’s robustness in managing complex, long-form videos, maintaining contextual continuity and relevance by strategically focusing on critical segments in response to user queries.

Dataset Composition.

The SceneWalk dataset comprises 87,867 long-form video sources, spanning a total of 11.87 Khrs (average video duration: 486.5 seconds). The video sources are collected from a curated selection of 10 diverse categories as inFig.1 sourced primarily from YouTube, with additional contributions from Movie & Drama datasets[53, 21]. This ensures a wide range of real-world scenarios, avoiding static categories.

Video Duration Distribution.

The collected videos can be split into three distinct duration ranges to analyze temporal diversity: (i) 0–240 seconds (short): This range constitutes about 24.4% of all segments, (ii) 240–600 seconds (long): This intermediate range accounts for the largest proportion, approximately 46.1% of the dataset, and (iii) 600–2280 seconds (extreme-long): The longest duration range comprises around 29.5% of the dataset. The distribution of video durations for each video category is illustrated in Fig.1(b) (outer circle), and more detailed duration distributions can be found in Fig.4.

Splitting into Video Segments

To divide untrimmed and long video sources into a massive $1.29$M video segments, we directly utilize PySceneDetect with the AdaptiveDetector using the default adaptive threshold ($3.0$), which compares the difference in content between adjacent frames similar using a rolling average of adjacent frame changes. This can help mitigate false detections in situations such as fast camera motions.

Instructions of Dense Segment Captioning.

To generate detailed descriptions for each video segment obtained from the above process, we mainly use a pre-trained LMM (VILA-1.5-13B[36]). Below Tab.4 includes the instructions for generating those captions. We randomly select one from the list and use it as a query for the model.

Captioning and Scoring.

Each segment is densely captioned, generating highly detailed textual descriptions that average $137.5$ words per segment. Please see the densely captioned video examples in Fig.6 and Fig.7. To ensure alignment quality, a generalized bipartite matching framework is employed: (i) Video-to-Text (V2T) Correspondence: A matrix evaluates the alignment between video segments and their paired captions using LanguageBind[73], and (ii) Text-to-Text (T2T) Context Similarity: The textual coherence among adjacent captions is assessed using SBERT[54], enhancing overall alignment robustness.

Supervision from Correspondence Scores.

To derive the supervision signal $y_{i}$ for training, we leverage the correspondence scores $S_{\text{V2T}}$ (Video-to-Text) and $S_{\text{T2T}}$ (Text-to-Text), as discussed in Sec.4. For each correspondence score matrix, we apply thresholding to extract meaningful relationships. Specifically, we define thresholds $\tau_{\text{V2T}}$ and $\tau_{\text{T2T}}$ for the two matrices, and elements with scores exceeding these thresholds are treated as positive correspondences (th: 0.18 ($\tau_{\text{V2T}}$) and 0.8 ($\tau_{\text{T2T}}$), respectively). These positive correspondences are then one-hot encoded to form binary matrices $Y_{\text{V2T}}$ and $Y_{\text{T2T}}$, where each element indicates whether a specific correspondence is valid. Finally, we compute the union of these binary matrices to produce the final supervision signal (e.g., $y_{i}{=}Y_{\text{V2T}}\cup Y_{\text{T2T}}$). The union operation ensures that any correspondence deemed valid by either of the two modalities contributes to the final supervision. This approach captures both the multi-modal alignment (Video-to-Text) and intra-modal coherence (Text-to-Text), providing a robust supervision signal for the retrieval task.

The resulting $y_{i}$ is then incorporated into the similarity loss function $\mathcal{L}_{\text{sim}}$ as described in Eq.1, ensuring that the model effectively learns the nuanced relationships between video segments and their corresponding textual descriptions. By combining $S_{\text{V2T}}$ and $S_{\text{T2T}}$ in this manner, we account for the complexity of generalized bipartite matching and enhance the model’s ability to align correspondences across and within modalities.