Ontologies
Turn any detection into a web of connected insights across video, image, audio, and text. Ontologies are warehouse-native knowledge graphs: find relationships your current search cannot see.
What You'll Get
Connect related content automatically
Typed edges between entities found across modalities
Rank by relationships, not keywords
Traverse edges at query time with the traverse_edge stage
Link 4 Modalities
Unify video, image, audio, and docs
Best for: Modeling entity relationships
Not for simple categorization (use Taxonomies) or grouping (use Clusters)
How Ontologies Sharpen Search
Ontologies provide a powerful way to model and traverse relationships between entities in your multimodal content. Unlike simple taxonomies that classify content, ontologies understand how entities relate to each other, enabling sophisticated reasoning about connections, dependencies, and associations.
Behind the Scenes: Cross-Modal Reasoning
Entities detected from video, images, and audio connected through relationships
LeBron James
from video
LA Lakers
from text
Nike
from image
Traverse: Face (video) → Team (text) → Sponsor (image)
// From video frame
{
"modality": "video",
"entity_type": "face",
"entity_id": "lebron_james"
}
// From audio transcript
{
"modality": "audio",
"entity_type": "team",
"entity_id": "la_lakers"
}
// From image
{
"modality": "image",
"entity_type": "logo",
"entity_id": "nike"
}POST /ontologies/expand
{
"entity": "lebron_james",
"source_modality": "video",
"relations": ["playsFor", "sponsoredBy"],
"return_modalities": ["image", "audio"],
"max_hops": 2
}
// Returns: Nike logos (images)
// + Lakers mentions (audio)Real-World Application: Multimodal Sports Content
Connect entities across video frames, audio transcripts, and image logos
Detect face in video
Face recognition identifies "LeBron James" at 00:14:23
🎥 VideoTraverse relationships automatically
- • 🖼️ Nike logos in images
- • 🎧 "Lakers" in audio mentions
- • 📄 Crypto.com Arena in docs
- • 🎥 Anthony Davis in videos
Result: Rich Multimodal Context
One face detection triggers discovery across all modalities-images, audio, documents, and videos.
What You Can Achieve
Real outcomes from implementing ontologies in your knowledge infrastructure
Intelligent Discovery
Find related content across all modalities through relationship chains.
Multimodal Discovery:
"LeBron James" (face in video)
→ Discovers 1,200+ related assets:
- • 🎥 840 video clips
- • 🖼️ 215 product images
- • 🎧 98 audio mentions
- • 📄 47 documents
Deeper Insights
Surface patterns across video demos, image catalogs, and document specs.
Cross-Modal Analysis:
"Recalled product X" (image)
→ Finds risk across modalities:
- • 🎥 31 demo videos with same part
- • 🖼️ 12 catalog images
- • 📄 4 spec documents
Better Recommendations
Recommend related content from any modality based on entity relationships.
Cross-Modal Recommendations:
Viewing product video → Recommends:
- • 🖼️ Images of compatible accessories
- • 🎧 Audio reviews from experts
- • 📄 User manuals & guides
Ready to traverse your own relationships?
Attach typed edges to your objects, then follow them at query time across video, images, audio, and documents.
Multimodal Questions Ontologies Can Answer
Go beyond single-modality search with cross-modal relationship reasoning
Without Ontologies
"Find videos of this person"
🎥 Single modality search
With Ontologies
"Find all content with brands this person endorses"
🎥 Video → 🖼️ Images → 📄 Documents
Without Ontologies
"Search for this product logo"
🖼️ Image-only results
With Ontologies
"Find videos where people mention this product"
🖼️ Logo → 🎧 Audio mentions → 🎥 Video
Without Ontologies
"Show documents about this topic"
📄 Text-only results
With Ontologies
"Show videos, images, and audio of experts on this topic"
📄 Topic → 👤 Experts → 🎥🖼️🎧 All modalities
{
"subject_id": "entity:lebron_james",
"subject_type": "person",
"subject_source": "video_frame_detection", // 🎥
"subject_modality": "video",
"relation": "endorses",
"object_id": "entity:nike",
"object_type": "brand",
"object_source": "logo_detection", // 🖼️
"object_modality": "image",
"confidence": 0.92,
"evidence": [
{ "type": "visual", "modality": "video" },
{ "type": "audio", "modality": "audio" },
{ "type": "contract", "modality": "document" }
]
}Relationships span modalities: Faces from video → Logos from images → Mentions from audio
Use Cases
Discover how organizations use ontologies to build intelligent systems.
Connect faces in videos, logos in images, and team names in audio to power rich content discovery.
Cross-Modal Relationships:
🎥 Face (video) → playsFor → 🎧 Team (audio)
🎧 Team (audio) → sponsoredBy → 🖼️ Logo (image)
🖼️ Logo (image) → appearsIn → 🎥 Video scenes
Query: Face in video → Find all sponsor logos in images + team mentions in audio
Link product images, video reviews, and document specifications to discover complete product relationships.
Cross-Modal Relationships:
🖼️ Product (image) → reviewedIn → 🎥 Video
🎥 Video → mentions → 🎧 Audio review
📄 Spec (doc) → describes → 🖼️ Product (image)
Query: Product image → Find video reviews + audio mentions + spec documents
Connect author faces in videos, voice in podcasts, and bylines in documents for comprehensive content discovery.
Cross-Modal Relationships:
🎥 Author (video) → discusses → 🎧 Podcast
📄 Article (doc) → references → 🖼️ Infographic
🎧 Interview (audio) → mentions → 📄 Research paper
Query: Author face in video → Find their podcasts + articles + cited images
Connect employee faces in security footage, voices in meetings, and names in documents across your organization.
Cross-Modal Relationships:
🎥 Face (security) → worksIn → 📄 Department (doc)
🎧 Voice (meeting) → expertIn → 📄 Skill (doc)
📄 Project (doc) → features → 🎥 Demo video
Query: Face in video → Find their presentations + documents + meeting recordings
Choose the Right Approach
Ontologies, Taxonomies, and Clusters serve different purposes
Ontologies
Model entity relationships. Traverse connections between people, brands, locations across modalities.
e.g., "Player → Team → Sponsor"
Taxonomies
Classify content into predefined categories. Organize using established systems.
e.g., IAB 3.0, product types
Clusters
Automatically group similar content. Discover patterns without predefined structure.
e.g., "Similar scenes"
💡 Use them together: Taxonomies classify, Ontologies connect, and Clusters group-making your multimodal data searchable and intelligent.
Cross-Modal Relationship Intelligence
Connect entities across video frames, audio transcripts, images, and documents
Video
Extract faces, objects, scenes from frames
Images
Detect logos, products, landmarks
Audio
Extract speakers, topics from transcripts
Documents
Parse entities, metadata from text
The Power: Entities extracted from any modality can be connected through ontological relationships. A face detected in a video can link to products in images, mentions in audio, and references in documents-all in a single query.
Ready to Get Started?
Attach typed edges to your objects, then follow them at query time with the traverse edge stage.
The pieces this is built from
Relationship traversal ships as typed edges on your objects plus a retriever stage that follows them.