Skip to main content
Your platform team will delete things. Node pools get recreated on a schedule, operators get patched, clusters get upgraded, and none of that will pause for Mixpeek. This page says what each of those does to a running install, and links the fixture run that measured it. Everything below was measured on a throwaway Kubernetes cluster with KubeRay on it, running the same install kit and chart a customer applies. Where something has not been measured, this page says so rather than reasoning about it.

​
The short version

​
Deleting the RayCluster is a non-event

Delete the RayCluster under a live RayService and KubeRay’s controller builds a new one. Measured at about one second of no Ray capacity, unattended, with a running head at the end. The rebuilt cluster carries the same name and a new UID. That matters if you are writing your own health check: a check keyed on the name cannot tell “this survived” from “this was destroyed and replaced”, and those are different answers. Key yours on .metadata.uid.

​
Deleting the RayService needs a redeploy

This one goes the other way, so do not generalise from the case above. Nothing reconciles a deleted RayService back. Only the Helm release owns it, so removing it is a redeploy rather than something that heals. Measured: the object was deleted, nothing recreated it, and a helm upgrade --install is what brings it back. The cascade is clean, which is the part worth stating positively. The RayCluster carries an owner reference to the RayService, so deleting the parent collects the child. Removing Mixpeek’s RayService does not leave Ray pods running and billing on your nodes with no object left to explain them.

​
Draining the head’s node recovers on its own

Cordon the node the Ray head is on and delete the head. While the node is cordoned the head cannot schedule. Uncordon it and the head returns to Running by itself, with no redeploy and no manual step. That is the answer to the node-pool question: a routine drain costs you the time the capacity is gone. It does not wedge the engine. One caveat about how this was measured. The fixture has two nodes and the second carries a NoSchedule taint for an unrelated test, so the head had nowhere to move while its node was cordoned. On your cluster, with spare schedulable capacity, the head reschedules instead of waiting. The transferable half is the recovery, and that is what was asserted.

​
Upgrading KubeRay does not restart your engine

KubeRay 1.5.1 upgraded to 1.6.2 underneath a live RayService. The running RayCluster was untouched: same UID before and after, head still Running. So you can patch KubeRay on your own schedule without coordinating a Mixpeek window. If a future KubeRay release does recreate the cluster, the qualification suite reports it as a compatibility finding rather than passing quietly.

​
What is not measured yet

In-flight work. Every result on this page was measured on an idle cluster. Seconds of missing Ray capacity is a non-event when nothing is running, and it may not be when a batch is mid-flight. What happens to a batch that was executing across a deletion, a drain or an upgrade has not been measured, and nothing here should be read as a claim about it. Kubernetes control-plane upgrades. The KubeRay half of the upgrade question is answered above. The Kubernetes half is not. The fixture cannot answer it, because throwaway local clusters do not upgrade their control plane in place, so a pass there would describe the fixture rather than your cluster. Measuring it needs a real managed cluster and it is scheduled work.

​
Where your data actually lives

None of the above touches your index. Ray holds no durable state in a Customer-Hosted install:
  • Your vectors are durable in your object storage. MVS keeps an unreplayed write-ahead window on node-local disk, so losing a pod costs a rebuild of that window rather than data.
  • Metadata is in Mongo, queues are in Redis, and source media stays in your own bucket by reference.
The stateful thing a node drain actually threatens is Redis, which uses a persistent volume. Size and back it up the way you would any other stateful workload in your cluster. See also what you provide, the qualification suite, and the compatibility table.