> ## Documentation Index
> Fetch the complete documentation index at: https://docs.mixpeek.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Rollout & Safety

> Safely deploy learned fusion with traffic splitting, shadow mode, and kill switches

Learned fusion changes how search results are ranked — per user, per query. That power requires operational controls. This guide covers how to roll out Auto-Tune safely: start with shadow mode, ramp traffic gradually, monitor, and know how to revert instantly if something goes wrong.

## Traffic Splitting

The `rollout_pct` field controls what percentage of requests use learned fusion weights. The rest fall back to static fusion (RRF by default).

```json theme={null}
{
  "learning_config": {
    "rollout_pct": 10.0
  }
}
```

Bucketing is **deterministic by user ID** — the same user always gets the same treatment (learned or static) on consecutive requests. This prevents a user from seeing different ranking behavior on every search.

```
rollout_pct: 0    → all requests use static fusion
rollout_pct: 10   → ~10% of users get learned fusion
rollout_pct: 50   → ~50% of users get learned fusion
rollout_pct: 100  → all requests use learned fusion (default)
```

<Info>
  Bucketing uses a hash of the `user_id`. If no `user_id` is provided in the query, the request always uses static fusion regardless of `rollout_pct`.
</Info>

### Live rollout override

`rollout_pct` in `learning_config` is the *durable* setting. To **ramp rollout up or down without editing the retriever config** — for a gradual canary, or to dial back instantly during an incident — set a live override:

```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/rollout
{ "rollout_pct": 25 }
```

The override is stored in Redis and takes effect on the next request, just like the kill switch — no config mutation, no redeploy. Setting `rollout_pct: 100` clears the override so the configured value applies again. In the Studio, the **Learning tab → Rollout Controls** slider drives this endpoint. The `learned-fusion/stats` response reports the `rollout_pct` actually in effect plus a `rollout_override` flag and the underlying `config_rollout_pct`.

## Shadow Mode

Shadow mode computes learned fusion weights and logs what the results *would* have been, but serves static fusion results to the user. This lets you evaluate the quality of learned fusion before it affects real users.

```json theme={null}
{
  "learning_config": {
    "shadow_mode": true,
    "rollout_pct": 100.0
  }
}
```

In shadow mode:

1. Both fusion strategies execute in parallel
2. Static fusion results are served to the user
3. Learned fusion weights and re-ranked results are logged as `shadow_execution` metadata
4. The [evaluation system](/retrieval/evaluations) can compare shadow vs. served results offline

The execution response includes a `learned_fusion_context` field (when shadow mode is active) so you can inspect what weights *would* have been used:

```json theme={null}
{
  "results": [ ... ],
  "learned_fusion_context": {
    "shadow_mode": true,
    "served_fusion": "rrf",
    "sampled_weights": {
      "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1": 0.68,
      "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding": 0.32
    },
    "context_level": "personal",
    "effective_exploration": 0.42,
    "feature_uris": [
      "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1",
      "mixpeek://multimodal_extractor@v1/vertex_multimodal_embedding"
    ]
  }
}
```

<Tip>
  Run shadow mode for at least one week before enabling live traffic. Compare NDCG and click-through rates between shadow and served results using [evaluations](/retrieval/evaluations).
</Tip>

## Kill Switch

Instantly disable learned fusion for all users on a retriever:

```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/disable
```

This sets a separate kill-switch key in Redis — it does not modify `rollout_pct`. All requests fall back to static fusion within the next request cycle. Re-enabling removes the kill-switch key, restoring the previous rollout percentage.

To re-enable:

```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/enable
```

The kill switch does not modify the retriever configuration permanently. It is an operational toggle — the `learning_config` and accumulated user weights remain intact, ready to resume when re-enabled.

<Warning>
  The kill switch is an emergency control. For planned rollbacks, use `rollout_pct: 0` in the retriever configuration instead — that is the durable setting.
</Warning>

## Per-User Opt-Out

Exclude a specific user from learned fusion. Their requests will always use static fusion regardless of `rollout_pct`.

```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/opt-out/{user_id}
```

Use cases:

* Internal test accounts that would skew weights
* Users who reported unexpected result changes
* Debugging — isolate a user to static fusion while investigating

To re-include:

```bash theme={null}
POST /v1/retrievers/{retriever_id}/learned-fusion/opt-in/{user_id}
```

## Preference Reset

Delete all learned personalization for a specific user, returning them to global-level weights:

```bash theme={null}
DELETE /v1/retrievers/{retriever_id}/learned-fusion/user/{user_id}
```

This:

1. Flushes their session cache entries
2. Marks pre-reset interactions so the aggregation query ignores them
3. Returns the user to global fallback weights immediately

The user can begin building new personal weights from scratch through future interactions.

## Weight Bounds

Even with strong personalization data, individual feature weights are clamped to prevent degenerate rankings:

```json theme={null}
{
  "learning_config": {
    "min_weight": 0.05,
    "max_weight": 0.95
  }
}
```

| Setting      | Default | Effect                                                                        |
| ------------ | :-----: | ----------------------------------------------------------------------------- |
| `min_weight` |  `0.05` | No feature drops below 5% weight — it always contributes something to results |
| `max_weight` |  `0.95` | No feature exceeds 95% weight — a single feature cannot completely dominate   |

After Thompson Sampling produces raw weights, they are clamped to `[min_weight, max_weight]` and re-normalized to sum to 1.0. This means even a user with extreme interaction patterns will always see results influenced by all configured features.

## Circuit Breaker

If learned fusion weight resolution takes too long (due to ClickHouse latency, Redis issues, or high load), the system automatically falls back to static fusion for that individual request:

```
learned fusion resolution > circuit_breaker_timeout_ms (default 1000ms) → fall back to static fusion
```

The timeout is configurable via the `circuit_breaker_timeout_ms` field in `learning_config` (range 1–30000ms).

The circuit breaker is per-request, not per-retriever. A single slow request does not disable learned fusion for everyone. The response's `learned_fusion_context` metadata indicates when a fallback occurred:

```json theme={null}
{
  "learned_fusion_context": {
    "context_level": "none",
    "sampled_weights": { ... },
    "effective_exploration": 1.0,
    "circuit_breaker_triggered": true,
    "weight_resolution_ms": 1023.4
  }
}
```

A `context_level` of `"none"` with `circuit_breaker_triggered: true` indicates the circuit breaker fired and uniform priors were used. The `weight_resolution_ms` field shows the actual resolution time. Monitor the `mxp_learned_fusion_circuit_breaker_total` Prometheus metric to detect systemic latency issues.

## Recommended Rollout Plan

<Steps>
  <Step title="Shadow mode (1-2 weeks)">
    Enable `shadow_mode: true` with `rollout_pct: 100`. All users get static results, but learned fusion runs in parallel and logs what it would have returned.

    **Check:** Run evaluations comparing shadow vs. served results. Learned fusion NDCG should be equal to or better than static.
  </Step>

  <Step title="1% canary (1 week)">
    Set `shadow_mode: false`, `rollout_pct: 1.0`. A small fraction of users get learned fusion.

    **Check:** Monitor click-through rates and circuit breaker triggers. No degradation vs. control group.
  </Step>

  <Step title="10% ramp (1 week)">
    Set `rollout_pct: 10.0`. Enough traffic to produce statistically significant comparisons.

    **Check:** Run a [benchmark](/retrieval/evaluations) comparing learned vs. static. Expect learned fusion to show improvement for users with sufficient interaction history.
  </Step>

  <Step title="50% ramp (1 week)">
    Set `rollout_pct: 50.0`. Half of users get personalized results.

    **Check:** Monitor weight distributions across users. Look for convergence patterns (weights stabilizing) rather than oscillation.
  </Step>

  <Step title="100% GA">
    Set `rollout_pct: 100.0`. All users get personalized fusion weights, with hierarchical fallback for new users.

    **Check:** Keep evaluations running on a schedule to catch regressions. Set up alerts for high global fallback rates and circuit breaker storms.
  </Step>
</Steps>

At any step, if you see regression:

1. Use the **kill switch** for immediate revert
2. Investigate using the `/learned-fusion/weights/{user_id}` endpoint to inspect specific users
3. Check whether the issue is global (bad reward map) or user-specific (outlier behavior)
4. Adjust the `learning_config` and restart from the previous step

## Python SDK

All rollout controls are available via the Python SDK:

<CodeGroup>
  ```python Kill Switch theme={null}
  from mixpeek import Mixpeek

  client = Mixpeek(api_key="sk_...")

  # Disable learned fusion (emergency kill switch)
  client.retrievers.disable_learned_fusion("ret_abc123")

  # Re-enable
  client.retrievers.enable_learned_fusion("ret_abc123")
  ```

  ```python Rollout & Opt-Out theme={null}
  # Set rollout percentage (live override, no config change)
  client.retrievers.set_rollout("ret_abc123", rollout_pct=25.0)

  # Opt out a specific user
  client.retrievers.opt_out_user("ret_abc123", user_id="user_456")

  # Opt them back in
  client.retrievers.opt_in_user("ret_abc123", user_id="user_456")

  # Reset all personalization for a user
  client.retrievers.reset_user("ret_abc123", user_id="user_456")
  ```

  ```python Monitoring theme={null}
  # Global weight distribution
  weights = client.retrievers.get_weights("ret_abc123")
  print(weights["feature_weights"])

  # Per-user weights and context level
  user_weights = client.retrievers.get_user_weights("ret_abc123", user_id="user_456")
  print(user_weights["context_level"])  # "personal", "demographic", or "global"

  # Aggregate stats (learner count, interaction count, rollout info)
  stats = client.retrievers.get_stats("ret_abc123")

  # Recent activity feed
  activity = client.retrievers.get_activity("ret_abc123")
  ```
</CodeGroup>

## Related

* [Auto-Tune](/retrieval/auto-tune) — overview of the full feedback loop
* [Reward Signals](/retrieval/reward-signals) — tuning how interactions influence weights
* [Evaluations](/retrieval/evaluations) — measuring retrieval quality