> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Ray infrastructure lifecycle

> What happens to a Customer-Hosted install when your platform team deletes, drains or upgrades underneath it

Your platform team will delete things. Node pools get recreated on a schedule,
operators get patched, clusters get upgraded, and none of that will pause for
Mixpeek. This page says what each of those does to a running install, and links
the fixture run that measured it.

Everything below was measured on a throwaway Kubernetes cluster with KubeRay on
it, running the same install kit and chart a customer applies. Where something
has not been measured, this page says so rather than reasoning about it.

## The short version

| What you do                          | What happens                             | Recovery                         |
| ------------------------------------ | ---------------------------------------- | -------------------------------- |
| Delete the `RayCluster`              | KubeRay rebuilds it under the same name  | Automatic, about a second        |
| Delete the `RayService`              | The cluster is garbage collected with it | Redeploy the chart               |
| Drain the node the head is on        | Head goes unschedulable, then returns    | Automatic, once capacity is back |
| Upgrade KubeRay underneath           | The running cluster is untouched         | Nothing to do                    |
| Upgrade the Kubernetes control plane | Not yet measured                         | See below                        |

## Deleting the RayCluster is a non-event

Delete the `RayCluster` under a live `RayService` and KubeRay's controller builds
a new one. Measured at about one second of no Ray capacity, unattended, with a
running head at the end.

The rebuilt cluster carries the same name and a new UID. That matters if you are
writing your own health check: a check keyed on the name cannot tell "this
survived" from "this was destroyed and replaced", and those are different answers.
Key yours on `.metadata.uid`.

## Deleting the RayService needs a redeploy

This one goes the other way, so do not generalise from the case above.

Nothing reconciles a deleted `RayService` back. Only the Helm release owns it, so
removing it is a redeploy rather than something that heals. Measured: the object
was deleted, nothing recreated it, and a `helm upgrade --install` is what brings
it back.

The cascade is clean, which is the part worth stating positively. The `RayCluster`
carries an owner reference to the `RayService`, so deleting the parent collects
the child. Removing Mixpeek's `RayService` does not leave Ray pods running and
billing on your nodes with no object left to explain them.

## Draining the head's node recovers on its own

Cordon the node the Ray head is on and delete the head. While the node is
cordoned the head cannot schedule. Uncordon it and the head returns to `Running`
by itself, with no redeploy and no manual step.

That is the answer to the node-pool question: a routine drain costs you the time
the capacity is gone. It does not wedge the engine.

One caveat about how this was measured. The fixture has two nodes and the second
carries a `NoSchedule` taint for an unrelated test, so the head had nowhere to
move while its node was cordoned. On your cluster, with spare schedulable
capacity, the head reschedules instead of waiting. The transferable half is the
recovery, and that is what was asserted.

## Upgrading KubeRay does not restart your engine

KubeRay 1.5.1 upgraded to 1.6.2 underneath a live `RayService`. The running
`RayCluster` was untouched: same UID before and after, head still `Running`.

So you can patch KubeRay on your own schedule without coordinating a Mixpeek
window. If a future KubeRay release does recreate the cluster, the qualification
suite reports it as a compatibility finding rather than passing quietly.

## What is not measured yet

**In-flight work.** Every result on this page was measured on an idle cluster.
Seconds of missing Ray capacity is a non-event when nothing is running, and it may
not be when a batch is mid-flight. What happens to a batch that was executing
across a deletion, a drain or an upgrade has not been measured, and nothing here
should be read as a claim about it.

**Kubernetes control-plane upgrades.** The KubeRay half of the upgrade question is
answered above. The Kubernetes half is not. The fixture cannot answer it, because
throwaway local clusters do not upgrade their control plane in place, so a pass
there would describe the fixture rather than your cluster. Measuring it needs a
real managed cluster and it is scheduled work.

## Where your data actually lives

None of the above touches your index. Ray holds no durable state in a
Customer-Hosted install:

* Your vectors are durable in your object storage. MVS keeps an unreplayed
  write-ahead window on node-local disk, so losing a pod costs a rebuild of that
  window rather than data.
* Metadata is in Mongo, queues are in Redis, and source media stays in your own
  bucket by reference.

The stateful thing a node drain actually threatens is Redis, which uses a
persistent volume. Size and back it up the way you would any other stateful
workload in your cluster.

See also [what you provide](/docs/customer-hosted/kuberay),
[the qualification suite](/docs/customer-hosted/qualification), and
[the compatibility table](/docs/customer-hosted/compatibility).