Search & Discovery

8 min read

Updated 2026-10-13

# Why Is My Image Embedding Model So Slow on CPU?

An image embedding model that crawls on CPU is usually running in bfloat16 on a processor with no hardware for it, or leaving most of its cores idle. PyTorch autocast on CPU defaults to bfloat16, and without AVX512-BF16 or AMX the math runs in software: we measured 13.0 s per image under autocast against 0.555 s in fp32 for SigLIP base. Run CPU inference in fp32, give each process one thread, and start one model copy per core.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

Embeddings

CPU Inference

PyTorch

Performance

Images

Ray

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## Why is my image embedding model so slow on CPU?

  
  
Usually one of two things. The model is running in bfloat16 on a processor that has no hardware for bfloat16, so every matrix multiply is emulated in software, or the job is using one core while the rest sit idle. PyTorch's autocast picks bfloat16 on CPU by default, and most cloud CPUs without AVX512-BF16 or AMX pay heavily for it. On our own production batch, each image took about 20 seconds and the batch wrote about 185 images an hour. Running in fp32 with one model copy per core brought the same workload to about 10,250 images an hour.  
  
The fix is three settings: run CPU inference in fp32, give each process the number of threads it has cores for, and run several single-threaded copies of the model, one per core.  
  

## How much slower is bfloat16 on a CPU without bfloat16 hardware?

  
  
About 23 times slower, in our measurement. We timed the SigLIP base vision encoder ([google/siglip-base-patch16-224](/models/siglip-base-patch16-224)) on 16 images of 1024 by 1024 pixels on a Google Cloud node with an AMD EPYC 7B12, which has AVX2 and no AVX512.  
  

Setting

Seconds per image

fp32, 1 thread

0.555

fp32, 2 threads

0.289

fp32, 4 threads

0.256

autocast bfloat16, 1 thread

13.016

autocast bfloat16, 2 threads

13.052

  
  
Measured by Mixpeek on 2026-10-12. Adding a second thread halved the fp32 time and did nothing for bfloat16, which is what emulated arithmetic looks like: the time goes into converting numbers, which more threads do not help.  
  

## Why does PyTorch use bfloat16 on CPU?

  
  
Because that is autocast's default for the CPU device. The [PyTorch automatic mixed precision documentation](https://docs.pytorch.org/docs/stable/amp.html) says autocast uses `torch.float16` for CUDA and `torch.bfloat16` for CPU when no dtype is given, and lists `linear`, `matmul`, `conv2d` and `bmm` among the CPU ops that autocast to bfloat16. Those are most of an image encoder's work.  
  
Code written for a GPU often wraps inference in `torch.autocast` to save memory and time there. Moved to a CPU worker, the same line keeps running and quietly switches the model to bfloat16. Nothing errors and the embeddings come out about the same, so the only symptom is speed.  
  

## How do I check whether my CPU supports bfloat16?

  
  
On Linux, look for the instruction set flags:

grep -o -w -E 'avx512\_bf16|amx\_bf16' /proc/cpuinfo | sort -u

No output means no bfloat16 hardware, and you should run in fp32. Intel Xeon processors from Cooper Lake onward and AMD EPYC processors from Zen 4 onward report `avx512_bf16`, and Intel Sapphire Rapids and later also report `amx_bf16`. Cloud instance families mix processor generations, so check on the machine the job actually runs on. Google Cloud lists the processors behind each machine series on its [CPU platforms page](https://cloud.google.com/compute/docs/cpu-platforms).  
  

## How do I run an image embedding model fast on CPU?

  
  
Turn autocast off for the CPU path, run without gradients, and set the thread count yourself:

import torch
from transformers import SiglipImageProcessor, SiglipVisionModel

torch.set\_num\_threads(1)  # one thread per process; run one process per core

model\_id = "google/siglip-base-patch16-224"
processor = SiglipImageProcessor.from\_pretrained(model\_id)
model = SiglipVisionModel.from\_pretrained(model\_id, torch\_dtype=torch.float32).eval()

def embed(images):
    pixels = processor(images=images, return\_tensors="pt").pixel\_values
    with torch.inference\_mode():  # fp32, no autocast on CPU
        return model(pixel\_values=pixels).pooler\_output

Keep autocast for the GPU path, where it helps. A simple guard is to enter `torch.autocast` only when the device is CUDA.  
  

## Why does my embedding job use one core when the machine has eight?

  
  
Two common causes. The thread count is wrong: PyTorch sets its own thread pool size ([torch.set\_num\_threads](https://docs.pytorch.org/docs/stable/generated/torch.set_num_threads.html)), and several processes each using every core fight over the same cores. Or the pipeline never starts more than one copy of the model.  
  
Threads stop paying off quickly for a small encoder. In the table above, going from 2 threads to 4 saved 11%. For throughput across many images, several single-threaded copies of the model beat one copy with many threads, as long as memory holds them.  
  
The second cause hit our own pipeline. Our image batches run on [Ray Data](https://docs.ray.io/en/latest/data/transforming-data.html), and the embedding step was an actor pool that started at one actor with room to grow to eight. Ray Data split the job's 4 CPUs across the pipeline's five steps, and the embedding step's share sat at 0.2 to 0.3 CPU, under the 1 CPU a second actor needed. The pool never grew. One worker ran at about 1.1 cores and the other at about 0.1.  
  
We fixed it with a pool whose minimum equals its maximum, so every copy starts at once: one single-threaded actor per worker CPU, each reserving 0.75 CPU and leaving the rest for decoding, thumbnails and writes. In Ray this is `min_size` and `max_size` on [ActorPoolStrategy](https://docs.ray.io/en/latest/data/api/doc/ray.data.ActorPoolStrategy.html).  
  

## What changed when we fixed both?

  
  
Here is the same pipeline on a GKE node with an Intel Xeon at 2.20 GHz, two workers of 2 CPUs each, embedding 3,414 images at the sizes of a real product catalog.  
  

Version

Model copies

Seconds per image in a copy

Images an hour

All 3,414 images

Before

1 copy, 1 CPU, batches of 64

19.7 to 21.0

180

about 19 hours

More copies only, still bfloat16

4 copies, 0.75 CPU each, batches of 16

20.0

720

about 4.7 hours

More copies and fp32

4 copies, 0.75 CPU each, batches of 16

1.32 median

10,253

20.0 minutes

  
  
Measured by Mixpeek on 2026-10-12. Starting more copies multiplied throughput by 4, and fp32 multiplied it by about 14 on top of that.  
  
Batch size mattered for memory. With 64 large images decoded at once, the single copy held 6.8 GB. At 16 per batch, two copies and their share of the other steps peaked at 5.5 GB per worker.  
  

## Should I use a GPU instead?

  
  
For large models or millions of images, often yes. A GPU runs the same encoder far faster per image, and autocast in float16 is the right setting there. A CPU worker is cheaper to keep running, scales to zero more easily and is enough for many catalogs: about 10,000 images an hour on 4 CPUs covers a 100,000-image catalog in roughly ten hours. Measure on your own model and images before choosing, since per-image cost on our Intel node was 2.4 times the AMD node's.  
  

## How does Mixpeek run image embedding on CPU?

  
  
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. Its image extractor runs these settings on CPU batch workers since October 13, 2026: fp32 on CPU, autocast only on GPU, one model copy per worker CPU and batches of 16. You upload or sync a bucket and the batch embeds every image with no pipeline of your own; the change is in the [changelog](/changelog). Extractors are described in the [feature extractor docs](/docs/processing/feature-extractors), and prices are on the [pricing page](/pricing).  
  
To choose a model, see the [best multimodal embedding models](/curated-lists/best-multimodal-embedding-models) and the [best self-hosted embedding models](/curated-lists/best-self-hosted-embedding-models). To estimate what a whole library costs to index, see [how much it costs to make a media library searchable](/guides/cost-to-make-a-media-library-searchable).  
  

## Frequently Asked Questions

  
  

### Why is my image embedding model so slow on CPU?

  
  
Most often it runs in bfloat16 on a CPU with no bfloat16 hardware, so the math is emulated, or it uses one core of many. Run CPU inference in fp32, set one thread per process and run one model copy per core.  
  

### Does torch.autocast speed up CPU inference?

  
  
It can on CPUs with bfloat16 hardware, such as Intel Xeon with AVX512-BF16 or AMX. On CPUs without it, autocast's default bfloat16 is emulated, and in our test SigLIP base took 13.0 seconds per image against 0.555 seconds in fp32.  
  

### How many threads should PyTorch use for inference?

  
  
For throughput over many images, one thread per model copy and one copy per core. Extra threads help a single request finish sooner, but in our test going from 2 threads to 4 saved only 11%.  
  

### How fast can a CPU embed images?

  
  
With SigLIP base in fp32 on our test nodes, 0.26 to 0.56 seconds per image on an AMD EPYC node with 1 to 4 threads, and about 1.3 seconds per copy on a shared Intel Xeon hyperthread. Four CPUs embedded about 10,250 catalog images an hour, including download and decoding.

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Search & Discovery

### IAB Content Taxonomy 3.1 classification benchmark: Mixpeek on IAB Tech Lab's test set

Mixpeek's IAB Content Taxonomy 3.1 text classifier run on IAB Tech Lab's open 704-page benchmark, next to IAB Tech Lab's published runs of EmbeddingGemma, Qwen3 and BGE-M3. Top-1 and top-5 accuracy, accuracy by category tier, a coverage-versus-precision threshold table, the full method and its limits.

Read guide →](/guides/iab-content-taxonomy-classification-benchmark)[

Search & Discovery

### What Is AI Video Analysis? How It Works, and How to Find a Moment in a Video

AI video analysis turns video into data you can query: what is on screen, what is said, what text appears and when each happens. How it works in five steps, a request you can run against a public footage library with no key, and how Mixpeek, Twelve Labs, Google Video Intelligence, Amazon Rekognition and an open-source CLIP pipeline compare on price and on what they return.

Read guide →](/guides/ai-video-analysis)[

Search & Discovery

### AdCP, AAMP, ARTF and Agentic Audiences: what each one does

AdCP and AAMP are two separate efforts to standardize AI agents in advertising. AdCP is one protocol from AgenticAdvertising.org, carried over MCP or A2A, with tasks like get\_signals and create\_media\_buy. AAMP is IAB Tech Lab's umbrella, which includes ARTF (agent containers inside an exchange that return bid request mutations) and Agentic Audiences (embeddings carried in OpenRTB). A dated comparison table, spec links, and what to do when a contextual embedding model changes.

Read guide →](/guides/adcp-vs-aamp-agentic-advertising-standards)