> ## Documentation Index > Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Troubleshooting > Every failure the qualification suite reports, what it means, and what to do One section per verdict either suite can produce. If you are holding a report with a `FAIL` or a `WARN`, find the row here. ## Substrate Rows from `substrate_e2e.py`, which runs before anything of ours is installed. ### `pvc-durability`: the volume was empty on a second pod Writes do not survive the pod that made them. MVS keeps its index on a PersistentVolume, so this is disqualifying rather than inconvenient. Usually the claim bound to something ephemeral, or the StorageClass provisioner is not actually provisioning. ### `pvc-durability`: the volume returned a marker that is not this run's The claim came back carrying data the suite did not write. Durability was not demonstrated, and separately, a volume being reused across workloads is worth understanding before anything of ours stores an index on it. Check whether the StorageClass is handing out a pre-existing volume. ### `pvc-durability`: SKIP, this identity cannot create PVCs The row needs to create one claim and two pods. Re-run as an identity with `create` on `persistentvolumeclaims` in the namespace, or accept that the durability claim is untested. ### `storage-class`: no StorageClass exists Nothing can provision a PersistentVolume, so MVS cannot run on this cluster as it stands. ### `storage-class`: WARN, no class is marked default A PVC without an explicit `storageClassName` will never bind. Tell us which class to use and we render it into the chart rather than assuming. ### `storage-class`: WARN, `allowVolumeExpansion` is false An MVS shard that outgrows its volume has to be replaced rather than expanded. Workable, and worth knowing before it happens at 3am. ### `storage-throughput`: WARN, no floor given The row measured a number and asserted nothing about it, which is what it does without `--require-write-mbps`. Ask us for the floor your configuration needs. ### `storage-throughput`: FAIL, below the stated floor Worth reading the caveat with the number: this is a sequential write to a freshly provisioned volume, measured once. Cloud block storage sells burst credits, so sustained indexing will be slower than the figure that already failed. If the class is sized by capacity (GKE `pd-balanced`, `gp3` without provisioned IOPS), a larger volume is often the whole fix. ### `workload-identity`: WARN, no ServiceAccount is bound to a cloud identity Mixpeek would need static credentials in a Secret to reach object storage. That is workable and is a posture worth choosing on purpose rather than by default. If you have wired an identity some other way, this row cannot see it: it reads ServiceAccount annotations only and does not attempt a bucket call. The Ray suite's `object-storage` row does that end to end, after install. ### `crd-install-grant`: WARN, this identity cannot create ClusterRoles Not a failure, and both answers are supported. It means your platform team installs and upgrades KubeRay, and the cost is that a Mixpeek release requiring a newer KubeRay needs a change on your side first. See [Security and RBAC](/docs/customer-hosted/security-rbac). On GKE, note that `container.customResourceDefinitions.create` and `container.clusterRoles.create` are separate permissions. Holding one does not imply the other, which is why the row checks three verbs and not one. ### `namespace-grant`: FAIL, cannot create some resource in the namespace The install creates it, so an install would fail partway through and leave objects behind. The row names each missing verb. ### `pod-security`: WARN, the namespace enforces `restricted` Our pods must run non-root with a `seccompProfile` and no added capabilities. Tell us before the install so the chart renders a compliant `securityContext`. Otherwise the first failure is an admission webhook message that reads like a Mixpeek bug. ### `node-capacity`: FAIL, not enough untainted allocatable capacity Tainted and cordoned nodes are excluded and listed in the evidence. If Mixpeek should be scheduling onto them, tell us the tolerations and that capacity counts. A cluster that is merely fenced should not read as too small. ### `node-capacity`: SKIP, no requirement given The row will not infer its own bound. Ask us for the CPU and memory your configuration needs and pass them. ### `kuberay`: WARN, not installed Expected on a cluster where KubeRay is not installed. Somebody has to install it; `crd-install-grant` says who can. ## Cluster ### `ray-reachable`: UNREACHABLE, or the dashboard does not answer The suite could not reach the Ray dashboard at the address it was given. Nothing downstream ran, so the rest of the report is empty rather than clean. Usually one of: the dashboard is not exposed outside the cluster, a port-forward died, or the address points at a different cluster. Confirm with `curl /api/version` from wherever the suite runs. ### `fingerprint`: the cluster is not the one you think The suite ran a job on the target and compared what came back against the address it was given. A mismatch means two different clusters are involved, which makes every other row describe something you did not intend to test. ### `kuberay`: CRDs not served, or below 1.1 KubeRay **1.1 or newer is required**. Below it, `submissionMode: HTTPMode` does not exist and batch cannot be submitted at all. Check with `kubectl get crd rayjobs.ray.io`. ## Security ### `rbac-minimal-sa`: a forbidden verb answered yes The submitter ServiceAccount can do something it must not: create a `ClusterRole`, create namespaces, delete nodes, or create RayJobs outside its own namespace. The row names which. This usually means the kit was applied on top of a pre-existing binding that granted more, or the ServiceAccount was reused. It is the one row where a `FAIL` means *you have given us too much* rather than too little. ### `rbac-minimal-sa`: every answer is `no` `kubectl auth can-i --as` needs impersonation rights. Run as an identity that cannot impersonate, everything answers `no`, and the row reads like a clean bill of health. The suite calls this out; re-run as an identity that can impersonate. ### `rayjob-cr-secrets`: FAIL Expected today, and it is our defect rather than yours. Batch RayJob CRs carry credential-shaped environment variables inline, so anyone with `get`/`list` on `rayjobs` can read them. See [Security and RBAC](/docs/customer-hosted/security-rbac) for the fix in flight and the two-minute TTL that bounds it. The row stays red until it lands, deliberately. ### `network-policy`: WARN, no NetworkPolicy exists Not a failure. Mixpeek's workloads in that namespace have unrestricted egress, which is a legitimate choice on a flat internal cluster. The row exists so that choosing it and forgetting stop looking the same. ### `network-policy`: WARN, every policy carries a podSelector A NetworkPolicy exists but the suite cannot resolve whether it selects *our* pods. A policy that selects something else leaves our workloads as open as none at all while still counting as one. Confirm the selector matches the pods in the Mixpeek namespace. ## Connectivity ### `registry-pull`: the nodes could not pull the image Your **nodes**, not the machine running the suite. Almost always a missing `imagePullSecret` in the namespace, or a registry your nodes cannot authenticate to. Mirror the image into a registry they already trust, or add the pull secret. ### `registry-pull` / `scheduling` / `failure-recovery`: SKIP, cannot create pods Not a defect in your cluster. The identity running the suite lacks `create` on pods, which the submitter Role does not grant by design. Re-run those rows as an identity that can create pods in the namespace. ### `object-storage`: WARN, static keys in the environment The bucket was reachable, but the job carried static credentials, so the run does not demonstrate the cluster's own workload identity. If you intend to use Workload Identity, this is the row that tells you it is not actually in use yet. ### `control-plane`: probe never reported from inside the cluster The job that was supposed to reach your control-plane endpoints never came back. Usually egress: a NetworkPolicy, a firewall rule, or a private endpoint without a route. The suite runs this from inside the cluster on purpose, because reachable from your laptop and reachable from a Ray worker are different questions. ### `engine-on-target`: the engine claims services not on this dashboard The Mixpeek engine you reached is running somewhere other than the cluster under test. The report would otherwise describe two machines at once. ## Workloads ### `scheduling`: a pod without a toleration still landed Your taint is not being honoured. The suite submitted the same pod twice, differing only by a toleration, and both scheduled. ### `scheduling`: did not schedule, but the reason does not mention a taint The pod was blocked by something else, most often insufficient CPU. The row fails because a pod failing to schedule is not evidence your taint did it, and crediting it to the taint would report a working policy that was never exercised. ### `scheduling`: a pod requesting 999 CPU was accepted There is no effective `ResourceQuota` in the namespace. Nothing bounds what Mixpeek can ask for. ### `autoscaling`: SKIP, no maximum given The row needs to be told your `maxReplicas`. It will not infer it: an assertion that the count never exceeded the highest value observed is true by construction and cannot fail. ### `autoscaling`: went past maxReplicas The autoscaler exceeded the ceiling in your own configuration. Worth escalating; this is your policy not being honoured by Ray, not by Mixpeek. ### `autoscaling`: scaled up and never came back down Workers remained after the load finished. You are paying for idle capacity. Check `idleTimeoutSeconds` in the cluster's autoscaler options. ### `failure-recovery`: two faults are indistinguishable Two different infrastructure failures produced the same diagnosis through the API, so a customer hitting one cannot tell which they hit. The row names the pair. ### `failure-recovery`: a fault never reached a settled state The suite read a transition rather than an outcome, and says so rather than reporting the transition as a diagnosis. ### `serve-lifecycle`: the customer's app moved Deploying, scaling or removing a Mixpeek Serve application changed one of yours: its status, its replica count, or its deploy timestamp. This is the row that must never fail on a shared cluster. See [Existing RayCluster](/docs/customer-hosted/existing-raycluster) for the deployment path and why it is per-application. ### `serve-lifecycle`: INCONCLUSIVE, cluster unstable The suite watched your cluster before writing anything and something moved on its own. Nothing can be attributed to Mixpeek in either direction on that run. Usually memory pressure or a cluster still settling; re-run once it is quiet. ## Isolation ### `non-interference`: a foreign app, job or actor changed Something of yours is no longer as it was after Mixpeek ran. The row names it. If the cluster was also unstable, the row says INCONCLUSIVE instead, because interference and a cluster falling over produce the same observation. ### `cleanup`: Mixpeek residue after uninstall Objects we created are still present after teardown. The row lists them. ## After install: the plane is up and a batch will not run ### `Engine submission failed ... [Errno 111] Connection refused` The API accepted the batch and could not hand it to the engine. That errno means a hostname resolved and nothing answered on the port, so the address is usually right in form and wrong in destination. The message names the address the API tried; start there. Three things produce it, in the order worth checking. **No Serve endpoint yet.** `kubectl get rayservice -n ` shows a blank `SERVICE STATUS` and `NUM SERVE ENDPOINTS`. The engine is still importing, or it cannot schedule. Check the next two. **No Ray worker.** `DESIRED WORKERS` above `AVAILABLE WORKERS`, and the reason is in the RayCluster's conditions rather than on any pod: ```bash theme={null} kubectl get raycluster -n \ -o jsonpath='{.items[0].status.conditions[?(@.type=="ReplicaFailure")].message}' ``` A worker whose resource requests exceed its limits is refused by the Kubernetes API server, so no pod is ever created and there is nothing to describe. `RayClusterProvisioned` still reads `True`, because it counts the pods that exist. **The applications have nowhere to go.** The engine's Serve deployments request a custom Ray resource named `serve_node`. If you supplied your own cluster config and dropped the `rayStartParams` that declare it, every application stays unschedulable and no status field says so. ### The address itself `kubectl exec -n deploy/api -- env | grep ENGINE_API` shows what the API resolved. The chart sets it to the head-only Serve service it installs alongside the `RayService`. Point it somewhere else only if you are running the engine outside this release. The service is head-only on purpose. KubeRay labels every pod `ray.io/serve=true`, workers included, and the engine runs its HTTP proxy on the head alone. A service selecting on that label by itself sends a share of requests to pods with nothing listening. ## Verdicts ### `INCOMPLETE` Nothing failed and not everything ran. Look for `SKIP` rows: a missing credential, a permission the running identity lacks, or a check that could not run. It is not a pass and not a failure, and treating it as either is the mistake it exists to prevent. ### `UNREACHABLE` The cluster did not answer. No other row in the report means anything.