What Is Semantic Video Search? How It Works, With Code

What is semantic video search? Semantic video search finds moments in video by meaning, so a query like "a crowd cheering after a goal" returns the matching seconds even when no title, tag or spoken word contains those words. It works by splitting each video into frames or short segments, turning each one into an embedding with a model such as CLIP or SigLIP 2, transcribing the speech, and ranking segments by how close their vectors and text are to the query. Sampling one frame per second gives 3,600 vectors for an hour of video.

Search method	What it matches	Finds	Misses
Filename and metadata	Titles, tags, dates someone typed	"Q3 all-hands 2024"	Anything inside the video nobody tagged
Transcript keyword search	Spoken words, with timestamps	"when the CEO said 'new strategy'"	Silent scenes, actions, visuals
Frame or segment embeddings	What the frames look like	"a red car turning left at night"	Exact names, numbers, quotes
Semantic (hybrid) video search	Embeddings plus transcript and on-screen text, fused	All of the above in one query	Little, if segments are short and models fit the footage

Updated October 2026: the code below uses CLIP ViT-B/32 (512-dimension embeddings) on every 30th frame, which is about one frame per second for 30 fps video. Newer image encoders such as SigLIP 2 retrieve more accurately, and production systems store the vectors in an approximate nearest neighbor index and add transcript search with Whisper. The approach is the same.

What is semantic video search?

Imagine scrubbing through hours of video content, painstakingly trying to locate that one golden moment when the CEO mentioned the company's new strategy, but you don't remember what they said. Just that they were enthusiastic. Your wrist aches from constant rewinding, and you're growing frustrated. Now, what if I told you there's a technology that could pinpoint that exact moment in seconds?

Semantic video search is an emerging technology that has the potential to revolutionize how we interact with videos. By allowing users to search for conceptual content within videos, it offers a way to bypass unnecessary fluff and get straight to the point.

Semantic video search extracts the content and internal features within a video to determine the presence of specific words, objects, themes, topics, sentiments, characters, or entities, rendering media clips easy to query, discover, and retrieve.

With its wide range of applications across various industries, including social media listening, brand insights, podcasts, and user-generated content, semantic video search is a powerful tool for video analysis and understanding video content in context.

How does semantic video search work?

Semantic video search leverages machine learning and natural language processing algorithms trained on extensive datasets of video content. These algorithms analyze the content and internal features within a video to identify specific elements, resulting in a highly accurate search and retrieval of video content.

Firstly, we need to extract frames from a video. Here's a simple way to do it using OpenCV in Python:

# Import the OpenCV library
import cv2

# Define a function to extract frames from a video
def extract_frames(video_path, frame_rate):

    # Create a VideoCapture object
    vidcap = cv2.VideoCapture(video_path)
    
    # Initialize the frame counter
    count = 0

    # Read the first frame from the video
    success, image = vidcap.read()

    # Initialize an empty list to hold the extracted frames
    frames = []

    # Loop over each frame in the video
    while success:
        # If the current frame is a multiple of the frame rate, add it to the frames list
        if count % frame_rate == 0:
            frames.append(image)

        # Read the next frame from the video
        success, image = vidcap.read()

        # Increment the frame counter
        count += 1

    # Return the list of extracted frames
    return frames

Then, we need to use CLIP to embed both the text queries and video frames into a shared semantic space. Here's a simple way to do it using PyTorch:

# Import the necessary libraries
import torch
from PIL import Image
from torchvision.transforms import Compose, Resize, Normalize, ToTensor
from openai_clip import clip

# Load the model
# Use CUDA if available, else use CPU
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load the pretrained CLIP model and the associated preprocessing function
model, preprocess = clip.load("ViT-B/32", device=device)

# Define a preprocessing function to convert video frames into a format suitable for the model
def preprocess_frame(frame):
    # Convert the frame to a PIL Image and apply the preprocessing
    return preprocess(Image.fromarray(frame))

# Define a function to convert text into a format suitable for the model, and get its embedding
def embed_text(text):
    # Run the model in no_grad mode to prevent the computation graph from being built (saves memory)
    with torch.no_grad():
        # Tokenize the text and move it to the appropriate device
        text_encoded = clip.tokenize([text]).to(device)
        
        # Pass the tokenized text through the model to get the text embeddings
        text_embedding = model.encode_text(text_encoded)
    return text_embedding

# Define a function to get the embeddings of a list of frames
def embed_frames(frames):
    # Initialize an empty list to store the frame embeddings
    frame_embeddings = []
    
    for frame in frames:
        # Run the model in no_grad mode to prevent the computation graph from being built (saves memory)
        with torch.no_grad():
            # Preprocess the frame and add a batch dimension
            frame_preprocessed = preprocess_frame(frame).unsqueeze(0).to(device)
            
            # Pass the preprocessed frame through the model to get the frame embeddings
            frame_embedding = model.encode_image(frame_preprocessed)
            
        # Add the frame embeddings to the list
        frame_embeddings.append(frame_embedding)
        
    # Stack the list of frame embeddings into a single tensor
    return torch.stack(frame_embeddings)

Finally, we need to compute the similarity between the text query and each frame, and return the frame with the highest similarity. Here's how we can do it:

def semantic_search(text, video_path):
    # Extract frames from the video
    frames = extract_frames(video_path, frame_rate=30)
    
    # Embed the text and the frames
    text_embedding = embed_text(text)
    frame_embeddings = embed_frames(frames)
    
    # Compute the similarity between the text and each frame
    similarities = (text_embedding @ frame_embeddings.T).squeeze(0)
    
    # Return the frame with the highest similarity
    most_similar_frame_index = similarities.argmax().item()
    return frames[most_similar_frame_index]

Now be sure to run this on a GPU, store it in a vector database, re-train the model, batch update the embeddings and do all so at scale.

How do you run semantic video search without building the pipeline?

Updated October 2026: the two calls below are Mixpeek's 2023 API. Today you upload files to a bucket, process them with a collection whose multimodal extractor splits, embeds, transcribes and reads on-screen text, and query the results with a retriever. The quickstart walks through it, and video search shows the product.

In addition to the contextual search of audio, image, and text, Mixpeek encapsulates everything above in addition to model fine-tuning at scale in a simple-to-use API:

mixpeek.index(["file_1.mp4", "file_2.mp4"])

mixpeek.search("good sportsmanship")

Sample response:

[
  {
    "content_id": "6452f04d4c0c0888bdc6b97c",
    "metadata": {
      "file_ext": "mp4",
      "file_id": "ebc289d7-44e1-4672-bf3c-ccfa490b7k2d",
      "file_url": "https://mixpeek.s3.amazonaws.com/<user>/<file>.mp4",
      "filename": "CR-9146f0.mp4",
    },
    "score": 0.636489987373352,
    "timestamps": [
      2.5035398230088495,
      1.2517699115044247,
      3.755309734513274
    ]
  }
]

And an example of what's possible:

What is semantic video search used for?

Educational Content Search: A student studying for an exam could use the semantic video search to locate specific content within a series of lengthy lecture videos. For instance, they could search for the term "mitochondria functions" to find all relevant discussions within a biology course.

Journalism and Documentary Research: Journalists or documentary filmmakers could use the tool to locate specific quotes or moments within hours of interview footage. For example, a journalist could search for "witness account of the event" to locate and verify statements made by interviewees.

Media Monitoring and Analysis: Businesses could use the tool for media monitoring, analyzing how often their brand or products are mentioned in videos. For example, a company could search for "Brand X reviews" to find relevant content within a vast array of product review videos.

Legal Case Preparation: Lawyers could use the tool to find specific statements or discussions within hours of deposition or courtroom footage. For instance, they might search for "defendant's testimony about the incident" to quickly locate relevant segments.

Film and TV Production: Film and TV producers could use the tool to find specific scenes or lines within a massive database of raw footage or scripts. For example, a director might search for "scenes in the rain" when compiling a mood board or reference reel.

Content Creation and Editing: YouTube content creators or video editors could use the tool to locate specific clips within their video archives. For instance, they could search for "laughing moments" or "funny fails" when compiling a highlight reel.

Law Enforcement and Security: Law enforcement agencies could use the tool to locate specific content within surveillance footage. For example, they might search for "red car entering the parking lot" to find relevant footage quickly.

Healthcare and Medical Research: Medical professionals could use the tool to find specific content within educational or research videos. For example, a surgeon could search for "laparoscopic appendectomy procedure" within a database of surgical procedure videos to review the process before surgery

Frequently asked questions

What is the difference between semantic video search and keyword video search?

Keyword search matches the words in titles, tags or a transcript. Semantic search compares meaning, using embeddings of the frames and the text, so it finds a scene described in different words or one where nothing is said at all. Most systems run both and merge the results, because keyword search is better at exact names and numbers.

How many frames per second should you embed for video search?

One frame per second is a common starting point: 3,600 embeddings for an hour of video. Lower rates miss short events, and higher rates mostly add near-duplicate frames. Many systems embed short segments of 5 to 15 seconds instead of single frames, which gives 240 to 720 vectors an hour and returns a clip with a start and end time.

How much storage does a semantic video index need?

At one frame per second with 512-dimension float32 embeddings, an hour of video takes 3,600 × 512 × 4 bytes, about 7.4 MB of vectors, before index overhead. Larger models with 768 or 1,152 dimensions scale that up in proportion, and quantizing to 8-bit integers cuts it by four.

Which models are used for semantic video search?

Image-text encoders such as CLIP and SigLIP 2 embed frames and queries in one space. Video embedding models embed whole segments with motion. Whisper transcribes speech, and an OCR model reads on-screen text. Mixpeek's multimodal extractor runs transcription, descriptions and OCR on each segment when you set run_transcription, run_video_description and run_ocr.

How does semantic video search return a timestamp?

Each embedding is stored with the start and end time of the frame or segment it came from, so the best-matching vector points straight at a moment. Mixpeek splits video into 10-second segments by default (split_method: "time", time_split_interval: 10) and can split on scene changes or silences instead.

Can semantic video search find what people say?

Yes, through the transcript. Speech is transcribed with timestamps, and the text is indexed for keyword search and embedded for meaning, so "the CEO talking about the new strategy" can match a segment where those exact words never appear.