Why is my image embedding model so slow on CPU?
Usually one of two things. The model is running in bfloat16 on a processor that has no hardware for bfloat16, so every matrix multiply is emulated in software, or the job is using one core while the rest sit idle. PyTorch's autocast picks bfloat16 on CPU by default, and most cloud CPUs without AVX512-BF16 or AMX pay heavily for it. On our own production batch, each image took about 20 seconds and the batch wrote about 185 images an hour. Running in fp32 with one model copy per core brought the same workload to about 10,250 images an hour.
The fix is three settings: run CPU inference in fp32, give each process the number of threads it has cores for, and run several single-threaded copies of the model, one per core.
How much slower is bfloat16 on a CPU without bfloat16 hardware?
About 23 times slower, in our measurement. We timed the SigLIP base vision encoder (google/siglip-base-patch16-224) on 16 images of 1024 by 1024 pixels on a Google Cloud node with an AMD EPYC 7B12, which has AVX2 and no AVX512.
| Setting | Seconds per image |
| --- | ---: |
|---|---|
| fp32, 1 thread | 0.555 |
| fp32, 2 threads | 0.289 |
| fp32, 4 threads | 0.256 |
| autocast bfloat16, 1 thread | 13.016 |
| autocast bfloat16, 2 threads | 13.052 |
Why does PyTorch use bfloat16 on CPU?
Because that is autocast's default for the CPU device. The PyTorch automatic mixed precision documentation says autocast uses
torch.float16 for CUDA and torch.bfloat16 for CPU when no dtype is given, and lists linear, matmul,
conv2d and bmm among the CPU ops that autocast to bfloat16. Those are most of an image encoder's work.Code written for a GPU often wraps inference in
torch.autocast to save memory and time there. Moved to a CPU
worker, the same line keeps running and quietly switches the model to bfloat16. Nothing errors and the embeddings
come out about the same, so the only symptom is speed.How do I check whether my CPU supports bfloat16?
On Linux, look for the instruction set flags:
grep -o -w -E 'avx512_bf16|amx_bf16' /proc/cpuinfo | sort -u
No output means no bfloat16 hardware, and you should run in fp32. Intel Xeon processors from Cooper Lake onward
and AMD EPYC processors from Zen 4 onward report
Turn autocast off for the CPU path, run without gradients, and set the thread count yourself:
avx512_bf16, and Intel Sapphire Rapids and later also report
amx_bf16. Cloud instance families mix
processor generations, so check on the machine the job actually runs on. Google Cloud lists the processors behind
each machine series on its CPU platforms page.How do I run an image embedding model fast on CPU?
Turn autocast off for the CPU path, run without gradients, and set the thread count yourself:
import torch
from transformers import SiglipImageProcessor, SiglipVisionModel
torch.set_num_threads(1) # one thread per process; run one process per core
model_id = "google/siglip-base-patch16-224"
processor = SiglipImageProcessor.from_pretrained(model_id)
model = SiglipVisionModel.from_pretrained(model_id, torch_dtype=torch.float32).eval()
def embed(images):
pixels = processor(images=images, return_tensors="pt").pixel_values
with torch.inference_mode(): # fp32, no autocast on CPU
return model(pixel_values=pixels).pooler_outputKeep autocast for the GPU path, where it helps. A simple guard is to enter
Two common causes. The thread count is wrong: PyTorch sets its own thread pool size (torch.set_num_threads), and several processes each using every core fight over the same cores. Or the pipeline never starts more than one copy of the model.
Threads stop paying off quickly for a small encoder. In the table above, going from 2 threads to 4 saved 11%. For throughput across many images, several single-threaded copies of the model beat one copy with many threads, as long as memory holds them.
The second cause hit our own pipeline. Our image batches run on Ray Data, and the embedding step was an actor pool that started at one actor with room to grow to eight. Ray Data split the job's 4 CPUs across the pipeline's five steps, and the embedding step's share sat at 0.2 to 0.3 CPU, under the 1 CPU a second actor needed. The pool never grew. One worker ran at about 1.1 cores and the other at about 0.1.
We fixed it with a pool whose minimum equals its maximum, so every copy starts at once: one single-threaded actor per worker CPU, each reserving 0.75 CPU and leaving the rest for decoding, thumbnails and writes. In Ray this is
Here is the same pipeline on a GKE node with an Intel Xeon at 2.20 GHz, two workers of 2 CPUs each, embedding 3,414 images at the sizes of a real product catalog.
Measured by Mixpeek on 2026-10-12. Starting more copies multiplied throughput by 4, and fp32 multiplied it by about 14
on top of that.
Batch size mattered for memory. With 64 large images decoded at once, the single copy held 6.8 GB. At 16 per batch, two copies and their share of the other steps peaked at 5.5 GB per worker.
For large models or millions of images, often yes. A GPU runs the same encoder far faster per image, and autocast in float16 is the right setting there. A CPU worker is cheaper to keep running, scales to zero more easily and is enough for many catalogs: about 10,000 images an hour on 4 CPUs covers a 100,000-image catalog in roughly ten hours. Measure on your own model and images before choosing, since per-image cost on our Intel node was 2.4 times the AMD node's.
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. Its image extractor runs these settings on CPU batch workers since October 13, 2026: fp32 on CPU, autocast only on GPU, one model copy per worker CPU and batches of 16. You upload or sync a bucket and the batch embeds every image with no pipeline of your own; the change is in the changelog. Extractors are described in the feature extractor docs, and prices are on the pricing page.
To choose a model, see the best multimodal embedding models and the best self-hosted embedding models. To estimate what a whole library costs to index, see how much it costs to make a media library searchable.
Most often it runs in bfloat16 on a CPU with no bfloat16 hardware, so the math is emulated, or it uses one core of many. Run CPU inference in fp32, set one thread per process and run one model copy per core.
It can on CPUs with bfloat16 hardware, such as Intel Xeon with AVX512-BF16 or AMX. On CPUs without it, autocast's default bfloat16 is emulated, and in our test SigLIP base took 13.0 seconds per image against 0.555 seconds in fp32.
For throughput over many images, one thread per model copy and one copy per core. Extra threads help a single request finish sooner, but in our test going from 2 threads to 4 saved only 11%.
With SigLIP base in fp32 on our test nodes, 0.26 to 0.56 seconds per image on an AMD EPYC node with 1 to 4 threads, and about 1.3 seconds per copy on a shared Intel Xeon hyperthread. Four CPUs embedded about 10,250 catalog images an hour, including download and decoding.
torch.autocast only when the device
is CUDA.Why does my embedding job use one core when the machine has eight?
Two common causes. The thread count is wrong: PyTorch sets its own thread pool size (torch.set_num_threads), and several processes each using every core fight over the same cores. Or the pipeline never starts more than one copy of the model.
Threads stop paying off quickly for a small encoder. In the table above, going from 2 threads to 4 saved 11%. For throughput across many images, several single-threaded copies of the model beat one copy with many threads, as long as memory holds them.
The second cause hit our own pipeline. Our image batches run on Ray Data, and the embedding step was an actor pool that started at one actor with room to grow to eight. Ray Data split the job's 4 CPUs across the pipeline's five steps, and the embedding step's share sat at 0.2 to 0.3 CPU, under the 1 CPU a second actor needed. The pool never grew. One worker ran at about 1.1 cores and the other at about 0.1.
We fixed it with a pool whose minimum equals its maximum, so every copy starts at once: one single-threaded actor per worker CPU, each reserving 0.75 CPU and leaving the rest for decoding, thumbnails and writes. In Ray this is
min_size and max_size on ActorPoolStrategy.What changed when we fixed both?
Here is the same pipeline on a GKE node with an Intel Xeon at 2.20 GHz, two workers of 2 CPUs each, embedding 3,414 images at the sizes of a real product catalog.
| Version | Model copies | Seconds per image in a copy | Images an hour | All 3,414 images |
| --- | --- | ---: | ---: | --- |
|---|---|---|---|---|
| Before | 1 copy, 1 CPU, batches of 64 | 19.7 to 21.0 | 180 | about 19 hours |
| More copies only, still bfloat16 | 4 copies, 0.75 CPU each, batches of 16 | 20.0 | 720 | about 4.7 hours |
| More copies and fp32 | 4 copies, 0.75 CPU each, batches of 16 | 1.32 median | 10,253 | 20.0 minutes |
Batch size mattered for memory. With 64 large images decoded at once, the single copy held 6.8 GB. At 16 per batch, two copies and their share of the other steps peaked at 5.5 GB per worker.
Should I use a GPU instead?
For large models or millions of images, often yes. A GPU runs the same encoder far faster per image, and autocast in float16 is the right setting there. A CPU worker is cheaper to keep running, scales to zero more easily and is enough for many catalogs: about 10,000 images an hour on 4 CPUs covers a 100,000-image catalog in roughly ten hours. Measure on your own model and images before choosing, since per-image cost on our Intel node was 2.4 times the AMD node's.
How does Mixpeek run image embedding on CPU?
Mixpeek reads your files where they already live, in S3, GCS or any bucket, and turns what's in them into data your software can search, classify and moderate. Its image extractor runs these settings on CPU batch workers since October 13, 2026: fp32 on CPU, autocast only on GPU, one model copy per worker CPU and batches of 16. You upload or sync a bucket and the batch embeds every image with no pipeline of your own; the change is in the changelog. Extractors are described in the feature extractor docs, and prices are on the pricing page.
To choose a model, see the best multimodal embedding models and the best self-hosted embedding models. To estimate what a whole library costs to index, see how much it costs to make a media library searchable.
Frequently Asked Questions
Why is my image embedding model so slow on CPU?
Most often it runs in bfloat16 on a CPU with no bfloat16 hardware, so the math is emulated, or it uses one core of many. Run CPU inference in fp32, set one thread per process and run one model copy per core.
Does torch.autocast speed up CPU inference?
It can on CPUs with bfloat16 hardware, such as Intel Xeon with AVX512-BF16 or AMX. On CPUs without it, autocast's default bfloat16 is emulated, and in our test SigLIP base took 13.0 seconds per image against 0.555 seconds in fp32.
How many threads should PyTorch use for inference?
For throughput over many images, one thread per model copy and one copy per core. Extra threads help a single request finish sooner, but in our test going from 2 threads to 4 saved only 11%.
How fast can a CPU embed images?
With SigLIP base in fp32 on our test nodes, 0.26 to 0.56 seconds per image on an AMD EPYC node with 1 to 4 threads, and about 1.3 seconds per copy on a shared Intel Xeon hyperthread. Four CPUs embedded about 10,250 catalog images an hour, including download and decoding.