Knowledge Graph Intelligence

# Ontologies

Turn any detection into a web of connected insights across video, image, audio, and text. Ontologies are warehouse-native knowledge graphs: find relationships your current search cannot see.

[Connect your own content](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=ontologies_hero&utm_campaign=mvs_onramp)[Read Documentation](https://docs.mixpeek.com/retrieval/stages/traverse-edge)

### What You'll Get

Connect related content automatically

Typed edges between entities found across modalities

Rank by relationships, not keywords

Traverse edges at query time with the traverse\_edge stage

4

Link 4 Modalities

Unify video, image, audio, and docs

Best for: Modeling entity relationships

Not for simple categorization (use [Taxonomies](/taxonomies)) or grouping (use [Clusters](/clusters))

## How Ontologies Sharpen Search

Ontologies provide a powerful way to model and traverse relationships between entities in your multimodal content. Unlike simple taxonomies that classify content, ontologies understand how entities relate to each other, enabling sophisticated reasoning about connections, dependencies, and associations.

An ontology holds the relationships between entities that survive across modalities, so a person recognised in video and named in a transcript resolve to the same thing. That job is one layer of [the semantic layer for unstructured and multimodal data](/semantic-layer), the architecture that turns raw objects into named features and serves them through contracts an application can bind to.

### Behind the Scenes: Cross-Modal Reasoning

Entities detected from video, images, and audio connected through relationships

Face

LeBron James

from video

playsFor

Team

LA Lakers

from text

sponsoredBy

Logo

Nike

from image

💡 Cross-Modal Query:"Find Nike ads featuring Lakers players"

Traverse: Face (video) → Team (text) → Sponsor (image)

Multimodal Entity ExtractionExtract

// From video frame
{
  "modality": "video",
  "entity\_type": "face",
  "entity\_id": "lebron\_james"
}

// From audio transcript
{
  "modality": "audio",
  "entity\_type": "team",
  "entity\_id": "la\_lakers"
}

// From image
{
  "modality": "image",
  "entity\_type": "logo",
  "entity\_id": "nike"
}

Cross-Modal QueryQuery

[Try in Mixpeek →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=ontologies_example&utm_campaign=mvs_onramp)

POST /ontologies/expand

{
  "entity": "lebron\_james",
  "source\_modality": "video",
  "relations": \["playsFor", "sponsoredBy"\],
  "return\_modalities": \["image", "audio"\],
  "max\_hops": 2
}

// Returns: Nike logos (images)
//          + Lakers mentions (audio)

### Real-World Application: Multimodal Sports Content

Connect entities across video frames, audio transcripts, and image logos

1

#### Detect face in video

Face recognition identifies "LeBron James" at 00:14:23

🎥 Video

2

#### Traverse relationships automatically

*   • 🖼️ Nike logos in images
*   • 🎧 "Lakers" in audio mentions
*   • 📄 Crypto.com Arena in docs
*   • 🎥 Anthony Davis in videos

✨

#### Result: Rich Multimodal Context

One face detection triggers discovery across all modalities-images, audio, documents, and videos.

## What You Can Achieve

Real outcomes from implementing ontologies in your knowledge infrastructure

### Intelligent Discovery

Find related content across all modalities through relationship chains.

Multimodal Discovery:

"LeBron James" (face in video)

→ Discovers 1,200+ related assets:

*   • 🎥 840 video clips
*   • 🖼️ 215 product images
*   • 🎧 98 audio mentions
*   • 📄 47 documents

### Deeper Insights

Surface patterns across video demos, image catalogs, and document specs.

Cross-Modal Analysis:

"Recalled product X" (image)

→ Finds risk across modalities:

*   • 🎥 31 demo videos with same part
*   • 🖼️ 12 catalog images
*   • 📄 4 spec documents

### Better Recommendations

Recommend related content from any modality based on entity relationships.

Cross-Modal Recommendations:

Viewing product video → Recommends:

*   • 🖼️ Images of compatible accessories
*   • 🎧 Audio reviews from experts
*   • 📄 User manuals & guides

Cross-Modal CTR+2.5x

### Ready to traverse your own relationships?

Attach typed edges to your objects, then follow them at query time across video, images, audio, and documents.

[Read the traverse edge docs →](https://docs.mixpeek.com/retrieval/stages/traverse-edge)[Schedule Demo](/demo)

## Multimodal Questions Ontologies Can Answer

Go beyond single-modality search with cross-modal relationship reasoning

❌

### Without Ontologies

"Find videos of this person"

🎥 Single modality search

✓

### With Ontologies

"Find all content with brands this person endorses"

🎥 Video → 🖼️ Images → 📄 Documents

❌

### Without Ontologies

"Search for this product logo"

🖼️ Image-only results

✓

### With Ontologies

"Find videos where people mention this product"

🖼️ Logo → 🎧 Audio mentions → 🎥 Video

❌

### Without Ontologies

"Show documents about this topic"

📄 Text-only results

✓

### With Ontologies

"Show videos, images, and audio of experts on this topic"

📄 Topic → 👤 Experts → 🎥🖼️🎧 All modalities

Cross-Modal Ontology TripleJSON

{
  "subject\_id": "entity:lebron\_james",
  "subject\_type": "person",
  "subject\_source": "video\_frame\_detection",  // 🎥
  "subject\_modality": "video",

  "relation": "endorses",

  "object\_id": "entity:nike",
  "object\_type": "brand",
  "object\_source": "logo\_detection",  // 🖼️
  "object\_modality": "image",

  "confidence": 0.92,
  "evidence": \[
    { "type": "visual", "modality": "video" },
    { "type": "audio", "modality": "audio" },
    { "type": "contract", "modality": "document" }
  \]
}

Relationships span modalities: Faces from video → Logos from images → Mentions from audio

## Use Cases

Discover how organizations use ontologies to build intelligent systems.

### Sports & Entertainment

[](/solutions/entertainment)

Connect faces in videos, logos in images, and team names in audio to power rich content discovery.

Cross-Modal Relationships:

🎥 Face (video) → playsFor → 🎧 Team (audio)

🎧 Team (audio) → sponsoredBy → 🖼️ Logo (image)

🖼️ Logo (image) → appearsIn → 🎥 Video scenes

Query: Face in video → Find all sponsor logos in images + team mentions in audio

### E-commerce & Retail

[](/solutions/ecommerce)

Link product images, video reviews, and document specifications to discover complete product relationships.

Cross-Modal Relationships:

🖼️ Product (image) → reviewedIn → 🎥 Video

🎥 Video → mentions → 🎧 Audio review

📄 Spec (doc) → describes → 🖼️ Product (image)

Query: Product image → Find video reviews + audio mentions + spec documents

### Media & Publishing

[](/solutions/entertainment)

Connect author faces in videos, voice in podcasts, and bylines in documents for comprehensive content discovery.

Cross-Modal Relationships:

🎥 Author (video) → discusses → 🎧 Podcast

📄 Article (doc) → references → 🖼️ Infographic

🎧 Interview (audio) → mentions → 📄 Research paper

Query: Author face in video → Find their podcasts + articles + cited images

### Enterprise Knowledge

[](/solutions)

Connect employee faces in security footage, voices in meetings, and names in documents across your organization.

Cross-Modal Relationships:

🎥 Face (security) → worksIn → 📄 Department (doc)

🎧 Voice (meeting) → expertIn → 📄 Skill (doc)

📄 Project (doc) → features → 🎥 Demo video

Query: Face in video → Find their presentations + documents + meeting recordings

## One Edge Hop Inside the Retriever

"nylon camping tent" over 118 product images returns five tents, and three of them are polyester or canvas. Material is a fact you already own, so it lives on a typed edge. One traverse\_edge hop along a same\_material edge adds the seven nylon hiking backpacks that no image embedding would rank near a tent, in the same retriever call.

Edges are copied from the object onto its documents at extraction time, so edges written onto an already-extracted corpus take effect after a re-extraction scoped to the linked objects.

[See the full diagram](/diagrams/one-edge-hop)

[![Five tents from semantic search, three of the wrong material, then one same\_material edge hop adds the seven nylon backpacks.](/diagrams/one-edge-hop.webp)](/diagrams/one-edge-hop)

## Choose the Right Approach

Ontologies, Taxonomies, and Clusters serve different purposes

### Ontologies

Model entity relationships. Traverse connections between people, brands, locations across modalities.

e.g., "Player → Team → Sponsor"

### [Taxonomies](/taxonomies)

Classify content into predefined categories. Organize using established systems.

e.g., IAB 3.1, product types

### [Clusters](/clusters)

Automatically group similar content. Discover patterns without predefined structure.

e.g., "Similar scenes"

💡 **Use them together:** Taxonomies classify, Ontologies connect, and Clusters group-making your multimodal data searchable and intelligent.

## Cross-Modal Relationship Intelligence

Connect entities across video frames, audio transcripts, images, and documents

🎥

#### Video

Extract faces, objects, scenes from frames

🖼️

#### Images

Detect logos, products, landmarks

🎧

#### Audio

Extract speakers, topics from transcripts

📄

#### Documents

Parse entities, metadata from text

**The Power:** Entities extracted from any modality can be connected through ontological relationships. A face detected in a video can link to products in images, mentions in audio, and references in documents-all in a single query.

## Ready to Get Started?

Attach typed edges to your objects, then follow them at query time with the traverse edge stage.

[Search your own files](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=ontologies_bottom&utm_campaign=mvs_onramp)[Talk to us](/demo)[Read the traverse edge docs](https://docs.mixpeek.com/retrieval/stages/traverse-edge)

## The pieces this is built from

Relationship traversal ships as typed edges on your objects plus a retriever stage that follows them.

[Traverse edge stageFollows typed, directed edges from a document to the documents they point to.](https://docs.mixpeek.com/retrieval/stages/traverse-edge)[Data modelWhere edges live on an object, and how they reach the retrieval payload.](https://docs.mixpeek.com/platform/data-model)[Multi-stage retrieval deep diveComposing traversal with search, filter, and rerank in one retriever.](https://docs.mixpeek.com/retrieval/multi-stage-deep-dive)