Skip to main content
Moment Group stage merging frame-level intervals into consolidated video moments
The Moment Group stage takes frame-level or chunk-level documents with temporal metadata and merges contiguous intervals into consolidated video moments (time ranges), grouped by parent object. Instead of returning individual frames that matched a query, you get precise start/end time ranges pinpointing where in a video the match occurs.
Stage Category: REDUCE (Merges intervals into moments)Transformation: N frame/chunk documents → M consolidated moments per parent video

​
When to Use

Use CaseDescription
Sub-scene localizationPinpoint exactly where in a video a query matches
Moment extractionReturn time-range clips instead of individual frames
Video search resultsProvide start/end timestamps for player seek
Highlight reelsExtract the top-scoring moments across a video library

​
When NOT to Use

ScenarioRecommended Alternative
Grouping by non-temporal fieldsgroup_by
Time-window aggregations (hour/day/week)temporal
Results without temporal metadataaggregate
Semantic clustering of resultscluster

​
Requirements

  • Input documents must have temporal intervals — either query_chunks with start_ms/end_ms (attached by feature_search preprocessing) or document-level time fields.
  • The parent_field must exist on the documents so results can be grouped by source video/object.
Text query vs file query — pick the right time_field. The default time_field: "query_chunks" is only populated when you search with a file input (via query preprocessing). For a plain text query, the matched video segments carry top-level start_time/end_time — set time_field: "start_time" (the stage derives the end field by swapping start→end). Leaving query_chunks with a text query yields empty moments — the most common silent failure here. See the Video Moment Localization recipe.

​
Parameters

ParameterTypeDefaultDescription
parent_fieldstring"source_object_id"Field to group results by parent video/object. Common values: source_object_id (standard ingest lineage), root_object_id (top-level ancestor), or any custom field.
time_fieldstring"query_chunks"Source of temporal intervals. "query_chunks" reads the array feature_search attaches (each entry has start_ms, end_ms, score). For document-level timestamps, use a dot-path like "metadata.start_ms" — the stage derives the end field by replacing "start" with "end".
merge_tolerance_msinteger2000Maximum gap in milliseconds between intervals before they are split into separate moments. 0 = exact-overlap only. Range: 0–60000.
max_moments_per_parentinteger10Maximum moments returned per parent, sorted by score (highest first). Range: 1–100.
score_strategystring"max"How to aggregate scores across merged intervals: "max", "avg", or "sum".
min_scorefloatnullDrop moments scoring below this threshold. Range: 0.0–1.0.
output_modestring"annotated""annotated" keeps the best-scoring document per parent and attaches a moments array (preserves all original document fields). "moments_only" emits standalone moment documents with IDs like moment_{parent}_{start_ms}.

​
Configuration Examples

{
  "stage_name": "moment_group",
  "stage_type": "reduce",
  "config": {
    "stage_id": "moment_group",
    "parameters": {
      "parent_field": "source_object_id",
      "time_field": "query_chunks",
      "merge_tolerance_ms": 2000,
      "max_moments_per_parent": 5,
      "score_strategy": "max",
      "output_mode": "annotated"
    }
  }
}

​
Output Schema

​
Annotated Mode (default)

In annotated mode, the best-scoring document per parent is preserved with a moments array attached: 
{
  "document_id": "doc_abc123",
  "source_object_id": "video_001",
  "score": 0.92,
  "content": "...",
  "metadata": { "..." : "..." },
  "moments": [
    {
      "start_ms": 12000,
      "end_ms": 18500,
      "score": 0.92,
      "match_count": 4,
      "document_ids": ["doc_abc123", "doc_abc124", "doc_abc125", "doc_abc126"]
    },
    {
      "start_ms": 45000,
      "end_ms": 51000,
      "score": 0.78,
      "match_count": 2,
      "document_ids": ["doc_abc130", "doc_abc131"]
    }
  ]
}

​
Moments-Only Mode

In moments_only mode, standalone moment documents are emitted: 
{
  "document_id": "moment_video_001_12000",
  "start_ms": 12000,
  "end_ms": 18500,
  "score": 0.92,
  "match_count": 4,
  "document_ids": ["doc_abc123", "doc_abc124", "doc_abc125", "doc_abc126"]
}

​
Moment Object Fields

FieldTypeDescription
start_msintegerStart of the moment in milliseconds
end_msintegerEnd of the moment in milliseconds
scorefloatAggregated score (per score_strategy)
match_countintegerNumber of source intervals merged into this moment
document_idsarrayIDs of the original documents that contributed to this moment

​
Performance

MetricValue
Latency5-50ms
MemoryO(N) where N = input documents
CostFree
ScalabilityEfficient — runs in the API layer, no engine call

​
Common Pipeline Patterns

​
Video Search with Moment Localization

The canonical use case: search across video frames, then consolidate matches into seekable moments. 
[
  {
    "stage_name": "feature_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          {
            "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
            "query": "{{INPUT.query}}",
            "top_k": 200
          }
        ],
        "final_top_k": 200
      }
    }
  },
  {
    "stage_name": "moment_group",
    "stage_type": "reduce",
    "config": {
      "stage_id": "moment_group",
      "parameters": {
        "parent_field": "source_object_id",
        "time_field": "query_chunks",
        "merge_tolerance_ms": 1500,
        "max_moments_per_parent": 3,
        "score_strategy": "max",
        "min_score": 0.6,
        "output_mode": "annotated"
      }
    }
  }
]

​
Moment Extraction with Reranking

Search, rerank for precision, then group into moments: 
[
  {
    "stage_name": "feature_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          {
            "feature_uri": "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding",
            "query": "{{INPUT.query}}",
            "top_k": 500
          }
        ],
        "final_top_k": 500
      }
    }
  },
  {
    "stage_name": "rerank",
    "stage_type": "sort",
    "config": {
      "stage_id": "rerank",
      "parameters": {
        "inference_name": "BAAI__bge_reranker_v2_m3",
        "query": "{{INPUT.query}}",
        "top_k": 100
      }
    }
  },
  {
    "stage_name": "moment_group",
    "stage_type": "reduce",
    "config": {
      "stage_id": "moment_group",
      "parameters": {
        "parent_field": "source_object_id",
        "time_field": "query_chunks",
        "merge_tolerance_ms": 2000,
        "max_moments_per_parent": 5,
        "score_strategy": "max",
        "output_mode": "annotated"
      }
    }
  }
]

​
Error Handling

ErrorBehavior
Missing parent_field on documentDocument skipped
Missing temporal metadataDocument skipped
No intervals pass min_scoreParent omitted from output
Empty inputEmpty results returned
  • Temporal - Time-window aggregations with drift detection
  • Group By - Group documents by any field value
  • Aggregate - Statistical aggregations
  • Deduplicate - Remove duplicate documents