Skip to main content
Mixpeek provides multiple observability surfaces: health endpoints, task metadata, Ray dashboards, analytics APIs, and webhook histories. Combine them to detect regressions early and debug production issues quickly.

​
Health & Status

Four health routes, and only one of them looks at your services.
GET /v1/health is a liveness constant. It does not check any service. It returns the same ok whether or not MongoDB, MVS, Redis, Celery or the engine are reachable, by design and in under 10ms.GET /v1/health?deep=true does nothing. That route declares no query parameters, so the flag is dropped and you get the same constant back. A monitor built on it reports green through a total outage. The deep check is a different path: GET /v1/health/deep.If you built a dashboard on ?deep=true, it is measuring the event loop rather than the services behind it. Point it at GET /v1/health/deep.
/v1/health/deep needs an organization API key. It is the only health route with auth. An unauthenticated call gets 403, and the platform private token gets 401 with “API key not found”, so neither an anonymous probe nor the private token can read component health. A 401 or 403 here is the auth requirement, not a broken route.On that path, ?deep=true is a real parameter: it adds data-plane verification on top of the service pings, reading from collections, probing MVS namespaces and inspecting Celery workers.
  • Tasks API – /v1/tasks/{task_id} and /v1/tasks/list expose status for batches, clustering jobs, taxonomy materialization, and migrations. All tasks use TaskStatusEnum.

​
Who owns this alert

GET /v1/ops/ownership maps every alert class to the team that owns it and the procedure for fixing it. Org-scoped, needs a bearer token.
Filter by alert_class to resolve one alert, or by subsystem to pull one entry. The response is {subsystems: [...], total}. Resolving the responder for one alert: use alert_escalations[alert_class] when that key is present, otherwise escalation. Reach for an override when a single alert inside a subsystem pulls in someone other than the subsystem default. A key that names an alert the subsystem does not own is rejected at load time, because an override on an unowned alert never fires and reads like routing that works. owner_confirmed records that the owner agreed to the entry, and where they said so. Absent means nobody has confirmed it. An unconfirmed entry can still be correct, so do not read the absence as a dispute.
Every entry names an owner and a procedure. The loader rejects an entry carrying one without the other, so runbook or runbook_text is always populated even when the written doc does not exist yet. Read both: either can be the one that is filled.CI pins that every declared alert class is covered by exactly one subsystem, so an alert cannot be owned twice or by nobody.
An unknown alert_class returns 404, and that is the useful answer. It means the alert has no owner in the registry — a gap to fill, not an empty result to shrug at.Class names are the CamelCase names from the alerting rules, so MvsShardDown, not mvs_shard_down. A snake_case name 404s the same way, and the error names the registry file so you can go and look.
  • Webhooks – webhook events recorded in MongoDB provide a durable log of ingestion and enrichment milestones (collection.documents.written, etc.).

​
Engine Monitoring

  • Ray Dashboard (port 8265) – view worker health, task timelines, Serve deployments, resource utilization, and logs.
  • Ray logs – pod logs (Kubernetes) or Ray CLI provide detailed extractor and clustering output (ray logs <job_id>).
  • Serve metrics – per-model latency and request counts; scrape via Prometheus or Ray metrics endpoint.

​
Analytics APIs

Enable analytics (ENABLE_ANALYTICS=true) to populate ClickHouse-backed metrics: Use these APIs to populate dashboards or feed alerting systems.

​
Logging & Tracing

  • API layer – structured JSON logs include request IDs, namespace, HTTP status, error codes, and downstream latency.
  • Celery workers – log task execution, retries, and webhook dispatch results.
  • Ray workers – include extractor metrics, batch IDs, and queue stats; aggregate logs centrally for long-term retention.
  • Correlation – propagate x-request-id from API to Engine jobs via additional_data.request_id to stitch traces together.

​
Metrics to Track

Integrate with Prometheus, Datadog, or your preferred metrics stack via existing exporters or custom scrapers.

​
Alerting Playbook

  1. Latency spike → check retriever analytics, stage statistics, and Ray Serve load.
  2. Task backlog → inspect Celery queue length, Redis health, and Ray worker availability.
  3. Failed enrichment → query /v1/tasks/list for FAILED, inspect error_message, review webhook events.
  4. Storage saturation → monitor MVS storage usage and MongoDB disk consumption; scale storage or shard by namespace.
  5. Cache regression → view cache hit-rate endpoint; adjust TTLs or stage cache configuration.

​
Dashboards to Build

  • API dashboard – health endpoint status, request latency, error breakdown, rate-limit counters.
  • Engine dashboard – Ray worker utilization, job runtime percentiles, extractor throughput, Serve queue depth.
  • Retrieval performance – retriever analytics charts (latency, cache hits, slow queries).
  • Storage dashboard – MongoDB/Redis/MVS metrics for capacity planning.
  • Task tracker – open tasks by status, median processing times, failure rates.

​
Incident Response Tips

  • Keep runbooks for common failures (e.g., extractor timeouts, MVS restarts).
  • Use webhook history to confirm whether ingestion completed or stalled.
  • Capture Ray job IDs from task metadata to replay logs quickly.
  • Snapshot retriever and collection configurations when debugging to ensure you’re reproducing the same pipeline.
With health checks, task metadata, analytics APIs, and Ray observability, you can confidently operate Mixpeek in production and catch issues before users notice.