What is Keypoint Detection
Keypoint Detection - Locating specific points on an object, such as joints or facial landmarks
Keypoint detection finds the pixel location of specific, named points on an object: the joints of a body for pose estimation, the eyes, nose and mouth corners of a face, or the corners of a product or document. Each keypoint comes with x and y coordinates and usually a confidence score. Keypoints add an object's shape and pose to the position a bounding box records.
How It Works
- Top-down: detect each object with a bounding box first, then find the keypoints inside each box; accurate, and slower when there are many objects
- Bottom-up: find every keypoint in the image first, then group them into objects; speed stays flat as the crowd grows
- Most models predict a heatmap per keypoint type and take the peak as the location
- Common layouts: 17 body keypoints in the COCO format, 5 facial landmarks for face alignment, 68 or more for detailed face meshes
Uses
- Pose estimation for sports, fitness and ergonomics analysis
- Face alignment: rotating and scaling a face to a standard template before recognition
- Action recognition in video, from how keypoints move between frames
- Augmented reality and motion capture
Keypoints in Mixpeek
The face identity extractor uses SCRFD to detect each face and its 5 facial landmarks (eyes, nose and mouth corners), aligns the face to a canonical template from those points, and then computes a 512-dimensional ArcFace embedding. The landmarks are stored with each face, next to its bounding box and quality scores. Full-body pose keypoints are not a built-in extractor; a custom extractor can add them.
Common Pitfalls
- Occluded or out-of-frame keypoints predicted with false confidence; filter by score
- Left and right swapped on mirrored images
- Evaluating with plain pixel distance across objects of different sizes; use a scale-aware metric such as COCO's object keypoint similarity (OKS)
Put multimodal search to work
Connect a bucket and Mixpeek runs the whole multimodal search pipeline for you: extraction, indexing, and search over your own objects. No models to wire up, nothing to host.
Start with ManagedAlready have vectors?
Keep your embeddings on your own cloud and run dense, sparse, and BM25 search directly on object storage. From $25/mo.
Start with MVS