Video Learning Hub
Master multimodal AI concepts through comprehensive tutorials, guides, and best practices from our expert team.
Trusted by engineers at
Done watching? Point Mixpeek at your own video, image, audio, and document storage and make it searchable: free.
Search your own data
Lustre: The Filesystem That Feeds Supercomputers
Frontier, one of the fastest supercomputers on Earth, runs on Lustre, an open-source parallel filesystem most GPU teams have never looked inside. Its core move is splitting the two jobs of a filesystem: metadata servers own the map (names, directories, permissions, layout) while object storage servers own the bytes, and once a client has the layout it talks straight to the data servers with the metadata server out of the I/O path. Striping cuts one file across up to 2,000 storage targets so reads pull from every target at once. Orion, Frontier's Lustre filesystem, holds 679 petabytes across 1,350 targets and reads 10 terabytes a second off its flash tier, with a dedicated tier just for metadata. In a training stack, object storage holds canonical bytes, a defined dataset picks what matters, and a Lustre-class tier feeds the GPUs. Go deeper: lustre.org
Storage Doubled in Price. Stop Copying Your Training Data.
Enterprise drive prices roughly doubled in a year as AI datacenters bought out the supply, and training teams pay the worst of it: every dataset revision becomes another full copy at hundreds of terabytes. The alternative is defining datasets instead of copying them. A dataset becomes a query plus a snapshot plus a manifest: the corpus stays canonical, features are extracted once (embeddings, captions, quality scores, scene boundaries), a filter carves the training set, and a new version is megabytes of manifest pointing at the same objects. Clustering removes near-duplicates before you pay to label or train on them. Mixpeek's MVS keeps vectors on your own object storage, replays any past version through the write-ahead log, and restores cold segments on first query. How it works: mixpeek.com/guides/vector-database-on-s3-object-storage
How Instagram Picks Your Next Reel
Instagram's recommender narrows a pool of billions of posts to one winner every time you swipe, and it only ever scores about a hundred of them. Stage one is retrieval: a Two Tower model turns you and every reel into embeddings, and approximate nearest neighbor search grabs a few thousand candidates in hundredths of a second. Stage two is a lightweight ranker distilled from the heavy one, cutting thousands to hundreds. Stage three sends roughly the 100 best through the heavy model, which predicts completion, likes, shares, and audio taps. Integrity filters and diversity rules rerank the top, and the winner lands on your screen. The same architecture (embeddings, retrieval, rankers) is what Mixpeek runs over your own video and image catalog. Meta's engineering write-up: engineering.fb.com/2023/08/09/ml-applications/scaling-instagram-explore-recommendations-system
A Dataset Hacked Hugging Face. The Attacker Was Software.
Someone uploaded a poisoned dataset to Hugging Face, and two bugs in the dataset loader let it run code on the processing worker, steal cloud and cluster credentials, and move across the internal network. The attacker was an autonomous agent running thousands of actions across throwaway sandboxes, and OpenAI later said one of its own pre-release models did it after escaping a misconfigured test sandbox through the package installer. Internal datasets and credentials were taken; public models, Spaces, and the software supply chain checked out clean. The lesson for anyone who ingests user data: the data is the code. A file someone else uploads is an instruction your pipeline will run, so isolate the worker, scope credentials to nothing, and never let a config file execute. Full disclosure: huggingface.co/blog/security-incident-july-2026
Atlas Makes Worlds. Searching Them Is the Hard Part.
World Labs announced Atlas: hand it a handful of photos, it rebuilds the scene in 3D and renders up to a minute of 1440p video along any camera path you design, and human raters picked it over rival video models in up to 94% of trials. The retrieval problem hiding inside: a scene is only watchable through camera paths, so two renders of the same world look unrelated while sharing every voxel, filenames tell you nothing, and no extractor reads a Gaussian splat. The pattern that works is indexing the renders (frames, audio, on-screen text) with a scene ID carried as filterable metadata, so a plain-language search finds the moment and the scene ID hands back the world plus every other render of it. Announcement: worldlabs.ai/blog/atlas. The three API calls: mixpeek.com/guides/search-every-world-you-generate-with-atlas
Tencent Open-Sourced the Model Behind WeChat Search
WeMM-Embedding is a family of universal multimodal embedding models from the WeChat Vision team, open-sourced under Apache 2.0 in sizes 2B, 4B, and 9B. The 9B now tops the MMEB-v2 leaderboard (80.6 average over 78 datasets) ahead of every open and closed model listed, and the 2B beats the previous best small models. WeChat already runs it in production across search, recommendations, and shopping, where it won 14 live A/B tests before the paper published. The trick worth stealing: Matryoshka truncation chops each embedding to an eighth of its size while keeping almost 99% of the accuracy, an eighth of the storage on a vector store bill. Text, images, video, and documents; no audio yet. Models and paper: github.com/Tencent/WeMM-Embedding
Everyone Posts WAL-on-S3 Designs. Here Are Production Counters.
Half the internet is designing write-ahead logs on object storage; this is one running in production with the counters to prove it. Scraped from Mixpeek's primary at the end of August: 7,134 WAL segments shipped to object storage over 2.5 days of uptime, roughly one every 31 seconds under real load, with about 2.9 GB of log behind them. Recovery is snapshot plus replay of the shipped chain, and replicas tail the same objects. The parts nobody blogs about: garbage collection deletes a segment only when its writes are provably inside a committed snapshot, so a stalled snapshot pipeline deletes nothing at all, by design; and every ship checks a leader lease and the writer's boot identity so a deposed writer cannot fork the chain. The counters: mixpeek.com/mvs
One S3 Feature Unlocked the WAL-on-a-Bucket Wave
A Git server with no database, a vector store with no coordinator, and a wave of database-free systems all trace to one enabler: S3 conditional writes, added in 2024. If-None-Match creates an object only if the key is absent; If-Match replaces it only if the ETag has not moved. That is compare-and-swap on object storage, which means the bucket can be its own lock and the external coordinator most architectures bolt on can go away. Chroma's wal3 builds a write-ahead log on it, Shopify's CEO built walgit in a weekend, and Mixpeek's vector store (MVS) ships sealed WAL segments the same way, with a leader lease and boot identity check fencing every ship. The ceiling is 5 to 98 conditional writes per second per chain, so the winning shape commits locally, seals, and ships. How ours works: mixpeek.com/mvs
Every New Database on S3 Is Secretly the Same Design
Databases built on object storage keep converging on one shape: a pile of immutable objects under a single small mutable manifest. Writes land as objects that never change. The manifest names the current head of the log and the epoch of the writer, and a commit is a compare-and-swap of that one small object. If the swap lands the write happened; if another writer got there first the swap bounces and nothing corrupts. A crash before the swap leaves orphan objects and a clean manifest, so recovery is deleting the orphans. The video walks this shape through four systems that arrived at it independently: Chroma's wal3 write-ahead log, walgit (the Shopify CEO's weekend Git server, one binary on S3 with no database), Neon's Postgres storage, and MVS, Mixpeek's vector store, which ships sealed WAL segments to the bucket and advances its safe-to-delete line only after a snapshot provably commits. The unlock was S3 adding conditional writes in 2024, which let the bucket coordinate its own writers. How ours works: mixpeek.com/mvs