---
url: https://docs.fluxify.rest/deployments/kubernetes.md
description: >-
  What Fluxify creates in your Kubernetes cluster for each claim, how workers
  are sized and autoscaled, and how to change claims with kubectl or GitOps.
---

# How it works on Kubernetes

On Kubernetes, Fluxify runs its workers as pods spread over your cluster. You
describe what you want the same way as anywhere else: a **claim** says which
project a set of workers serves, what they run, and how many of them. The
orchestrator turns each claim into ordinary Kubernetes objects and keeps them
in line with what you asked for.

This page explains what happens in your cluster once Fluxify runs.

> \[!TIP]
> **To install Fluxify, follow [Install on Kubernetes](./install)**. It goes
> step by step from an empty cluster to signing in.

## Docker or Kubernetes?

| | **Docker** | **Kubernetes** |
| :--- | :--- | :--- |
| **Where workers run** | Containers on one Docker host | Pods spread over your cluster's machines |
| **Scaling** | Fixed number of workers per claim | Grows and shrinks between a minimum and a maximum, on load |
| **Credentials** | Passed to each container as settings | Kept in Kubernetes Secrets |
| **Install guide** | [Production Setup →](../production) | [Install on Kubernetes →](./install) |

Your claims, projects and data are the same on both. Moving from one to the
other is a change to the orchestrator's settings, not to your database.

## Changing workers by hand

The orchestrator keeps every object it created matching your claims. If you
edit one of them directly (with `kubectl edit`, for example), your change is
put back within a few seconds, and deleting one makes it come back. Change the
claim instead, in the portal or through its `NodeClaim` (below). A manual
`kubectl scale` is undone by the autoscaler.

## Changing claims from Kubernetes {#nodeclaim}

If you manage your cluster with `kubectl` or a GitOps tool such as Argo CD or
Flux, you can change a claim there instead of in the portal. Each claim shows up
as a `NodeClaim` resource, named after the claim's id:

```bash
kubectl get fluxclaim
```

```text
NAME                                   TYPE       REPLICAS   MAX   READY   ACCEPTED   AGE
0192b1c4-1111-7000-8000-000000000001   workflow   2          6     2       True       3d
```

```yaml
apiVersion: fluxify.rest/v1alpha1
kind: NodeClaim
metadata:
  name: 0192b1c4-1111-7000-8000-000000000001
spec:
  project: shop
  type: workflow
  env: production
  groups: [shop/orders]
  replicas: 2
  maxReplicas: 6
  resources: { cpu: 1, memoryMb: 1024 }
```

| Field | Can you change it here? |
| :--- | :--- |
| `replicas`, `maxReplicas`, `resources` | Yes. The change is rolled out within a few seconds, exactly as if it was made in the portal. Remove `maxReplicas` to stop autoscaling. |
| `project`, `type`, `env`, `groups` | No. Change these in the project's settings, or in the instance settings for a claim that serves every project. |

`env` is `production` or `development` — a [development
claim](../production#dev-claims) serves the development paths only, takes no
license slot and never autoscales, so it has no `maxReplicas` and no
`ScaledObject` beside its Deployment.

Projects and groups are named by readable keys, not ids:

* `project` is the project's **slug**, chosen when the project is created (it
  defaults to the project name, lowercased with hyphens). A claim that serves
  every project shows `*`.
* Each group is `<project slug>/<group name>`, for example `shop/orders`. The
  copy button next to each group in the project's trigger groups copies it.

Neither a slug nor a group name can be changed once created, so a manifest you
write by hand keeps working.

What to expect:

* **The same rules as the portal.** A change your license or pool does not
  allow is refused. The resource goes back to the claim's current values, and
  `status.conditions` says why:

  ```bash
  kubectl get fluxclaim <claim id> -o jsonpath='{.status.conditions[0].message}'
  ```

* **You cannot create or delete claims here.** The orchestrator creates one
  `NodeClaim` per claim and removes it when the claim is released in the portal.
  Deleting a `NodeClaim` by hand releases nothing: it comes back on the next pass.

* **The last change wins.** A change made in the portal is copied onto the
  `NodeClaim`, and a change made on the `NodeClaim` is copied into the portal.
  The claim's history says which side each change came from.

* **`status` shows what the portal shows:** how many workers are placed, how
  many are ready, and how many are waiting and why.

> \[!WARNING]
> **GitOps tools put their own values back.** If a GitOps tool manages a
> `NodeClaim`, your git repository wins: a change made in the portal is undone at
> the tool's next sync. For now we recommend managing a project's claims in one
> place only, the portal or git. Fluxify does not yet keep a versioned history of
> claims in git for you.

This needs the `NodeClaim` definition in your cluster. The chart and
`fluxify.yaml` install it. Without it, everything else works and claims are
changed in the portal only. The orchestration page in the portal lists what
your cluster is missing. To install it by hand:

```bash
kubectl apply -f https://raw.githubusercontent.com/Fluxify-rest/Fluxify/main/deploy/helm/fluxify/crds/nodeclaim.crd.yaml
```

The orchestrator's role also needs access to `nodeclaims` and
`nodeclaims/status`. The chart's role has both.

> \[!TIP]
> **Running Karpenter?** It has its own `NodeClaim`. Use `fluxclaim` (or
> `nodeclaims.fluxify.rest`) with `kubectl` so you always get Fluxify's.

## Sizing a worker {#sizing}

Every claim says how much CPU and memory **each** of its workers may use.
Workloads differ — an API serving small requests and a queue crunching reports
need different amounts — so this is set per claim, not once for everything.

| | Steps of | Range | Default |
| :--- | :--- | :--- | :--- |
| **CPU** | 0.5 core | 0.5 – 16 | 1 core |
| **Memory** | 256 MB | 256 – 65 536 MB | 1024 MB |

Set it on the claim: **CPU per pod** and **Memory per pod** in the claim form,
in a project's settings or on the instance's Orchestration page.

On Kubernetes a worker is guaranteed exactly this much and never uses more.
The autoscaler measures CPU and memory use against these numbers, so they also
decide when a claim grows. The same setting applies on Docker, as the
container's limit.

## What a claim becomes in your cluster {#objects}

Each claim becomes a handful of objects, all named after the claim —
`fluxify-worker-<claim id>` — and all labelled
`fluxify.managed-by=orchestrator`. Nothing without that label is ever read,
changed or deleted.

| Object | When | What it does |
| :--- | :--- | :--- |
| **Deployment** | Always | Runs the claim's workers. Every worker is identical; its pod name is its identity. |
| **Service** | Claims that serve APIs | One stable address in front of the claim's ready workers. |
| **IngressRoute** (Traefik) | Claims that serve APIs | Sends traffic in: a project's own domain for a project's claim, and every other request to the claim that serves all projects. |
| **ScaledObject** (KEDA) | Always, when KEDA is installed | Grows and shrinks the Deployment between the claim's minimum and maximum. |

Two kinds of **Secret** keep credentials out of those objects:

* `fluxify-worker-env` holds the settings every worker copies from the
  orchestrator — the NATS address, the encryption key, log shipping. Change
  one of those and the workers restart to pick it up.
* `fluxify-trigger-<trigger id>` holds one trigger's connection details, for
  triggers that read from an outside queue.

A useful way to look at everything Fluxify runs:

```bash
kubectl get deploy,svc,ingressroute,scaledobject,pods -l fluxify.managed-by=orchestrator
```

> \[!IMPORTANT]
> **Change claims, not the objects.** The orchestrator rewrites these objects
> every few seconds to match your claims, so an edit made with `kubectl`
> is undone on the next pass. In particular, `kubectl scale` has no lasting
> effect — the autoscaler owns the worker count.

## How a claim scales {#scaling}

A claim has a **minimum** and a **maximum** number of workers, both set in
the claim form. A maximum equal to the minimum means the claim runs exactly
that many and never grows. On Docker the form shows only one number, since
Docker does not autoscale.

What makes it grow depends on what it runs:

| Claim | Grows when |
| :--- | :--- |
| A project's **workflow** workers | Runs are waiting on any of its triggers. Each worker takes on about as many waiting runs as the instance's *queued per node* setting, and the busiest trigger decides. |
| Workers that serve **APIs**, or **both** | CPU or memory use is above the targets above. |
| Workers that serve **every project** | CPU or memory use is above the targets above. |

Growing happens as soon as it is needed. Shrinking waits until the load has
stayed low for the instance's *scale-down window* (five minutes by default), so
a short lull does not stop workers you are about to need again. A worker that
is removed first stops taking new requests, then finishes what it is running.

The maximum is a ceiling, never a promise. A claim never grows past what your
node pool and license leave room for, however high its maximum is set:
workers beyond the license would only be refused when they start.

### Triggers on an outside queue {#external-queues}

A workflow claim also grows on the triggers that read an outside queue —
Kafka, Amazon SQS, a NATS server of your own, a Redis stream, or a RabbitMQ queue. KEDA reads each queue itself,
next to the claim's other triggers, and the busiest one decides. Each worker
takes on about *queued runs per pod* waiting messages, as with any trigger.

| Trigger | Grows on | Needs |
| :--- | :--- | :--- |
| **Kafka** | The trigger's consumer group lag on its topics. | KEDA reaches the brokers. |
| **SQS** | Messages waiting in the queue. | Keys on the integration, or an AWS identity for KEDA itself (below). |
| **NATS** (your own server) | The trigger's pending messages on its stream. | The integration's *Monitoring endpoint*. |
| **Redis Streams** | Entries the trigger's consumer group has not read yet. | Redis 7 or newer, reachable from KEDA. |
| **RabbitMQ** | Messages ready in the queue. | KEDA reaches the broker's AMQP port. |

The connection details go into one Secret per trigger,
`fluxify-trigger-<trigger id>`, and a KEDA `TriggerAuthentication` of the same
name points at it. No credential is ever written into a ScaledObject.
Changing an integration's credentials reaches KEDA within one orchestrator
pass, without restarting anything. A trigger whose integration cannot be read
does not scale its claim, and everything else carries on.

The orchestrator's role needs `triggerauthentications` in the `keda.sh` group
alongside `scaledobjects`.

### Plugin triggers that report their lag {#plugin-lag}

A plugin trigger can tell Fluxify how far behind it is ([how a plugin does that](/plugins/authoring/triggers#lag)). A trigger that does, and was saved with a **Lag per worker**, makes its claim grow on that number: the claim runs about *backlog ÷ lag per worker* workers, between its minimum and maximum, and the busiest trigger decides, as with the queues above. Without a **Lag per worker**, a plugin trigger does not scale its claim.

KEDA reads the number from the claim's workers, over the cluster network, so such a claim gets a Service of its own named after it, with only the workers' health port on it. Nothing on the public side changes. The workers answer only with a token, which Fluxify derives from the master encryption key and keeps in the trigger's Secret and `TriggerAuthentication`, the same way as for the queues. There is nothing to configure.

If a number cannot be read, because a worker is still starting or the trigger's source is down, the claim **stays at its current size**. It never reads as an empty queue. This needs a KEDA version that supports the ScaledObject's `fallback.behavior` setting (the current releases do).

Changing a trigger's **Lag per worker** reaches KEDA within one orchestrator pass and does not restart the trigger.

#### Kafka: scaling past the partition count {#allow-idle-consumers}

By default KEDA never runs more workers than the topics have partitions,
because a worker without a partition has nothing to read.

A claim takes the highest number any of its triggers asks for. So a Kafka
trigger held at its partition count holds back **every other trigger in the
same group**: they can have a backlog and still get no more workers.

Turn on **Scale past the partition count** on the Kafka trigger to lift that
limit. The extra workers read nothing from Kafka, but they serve the group's
other triggers. If you put more triggers in a group that already holds a Kafka
trigger, turn it on, or scaling stops at the partition count.

#### SQS without keys

An SQS integration with empty keys reads with the server's own AWS identity.
KEDA cannot borrow that, so it uses **its own**: give KEDA's operator an AWS
identity that may read the queue (for example with IRSA on EKS). Without one,
KEDA reports an authentication error for that scaler and the claim does not
grow on that queue.

#### A NATS server of your own

KEDA reads NATS through its HTTP monitoring port, not the client port. Set the
integration's **Monitoring endpoint** to that port as KEDA reaches it
(`host:port`, or an `https://` URL), and **Account** if your streams are not in
the default account. Left empty, the trigger runs as usual but does not scale
its claim.

#### Redis Streams

KEDA counts the entries the trigger's group has not read yet, which Redis
reports from version 7. On an older Redis the trigger runs as usual, but KEDA
reports an error for that scaler and the claim does not grow on that stream.
The integration's password goes into the trigger's Secret; a Redis without one
needs no Secret at all.

#### RabbitMQ

KEDA logs in over AMQP, the same port the trigger uses, and counts the messages
ready in the queue. Messages already handed to a worker and not yet
acknowledged are not counted. The integration's connection details, login
included, go into the trigger's Secret as one `amqp://` or `amqps://` URL.

Growing a claim on a queue needs an enterprise license, for Redis Streams and
RabbitMQ as for every other trigger. The triggers themselves run without one.

### The scaling policy {#scaling-policy}

How quickly claims grow and shrink is one policy for the whole instance, set
under **Instance settings → Orchestration → Scaling policy** (shown only when
the orchestrator drives a cluster):

| Setting | What it does | Default |
| :--- | :--- | :--- |
| **Queued runs per pod** | A workflow claim adds a worker each time this many runs are waiting on one of its triggers. | 10 |
| **Check every** | How often waiting runs are counted. | 30 s |
| **Scale-down wait** | How long the load must stay low before a worker is removed. | 300 s |

### What needs what

* **KEDA** is required for any growth at all. Without it, every claim runs at
  its minimum.
* **Scaling on waiting runs** also needs `K8S_NATS_MONITORING_ENDPOINT`, which
  the chart sets.
  Without it, workflow claims fall back to CPU and memory — unless they run
  [triggers on an outside queue](#external-queues), which scale on their own.
* **Changing claims with `kubectl` or GitOps** needs the
  [`NodeClaim` definition](#nodeclaim). Without it, claims change in the portal only.
* A claim that serves **every project** always scales on CPU and memory. To
  scale on its queues it would have to watch every trigger in the instance.
  Treat it as a starting point; give a busy project its own claim.

## Running the orchestrator yourself {#orchestrator-settings}

[The install guide](./install) sets all of this for you. It matters only if you
run the orchestrator some other way, for example
[from a Fluxify checkout](./local).

| Setting | What it is |
| :--- | :--- |
| `ORCHESTRATOR_PROVIDER` | `kubernetes` to run workers on a cluster, `docker` for one Docker host. Default: `docker`. It is never guessed. |
| `K8S_API_URL` | The cluster's API address, when the orchestrator runs outside the cluster. Inside, it finds the cluster on its own. |
| `K8S_SA_TOKEN` | The token it signs in with, outside the cluster. |
| `K8S_NAMESPACE` | The namespace workers are created in. Default: its own namespace inside the cluster, else `default`. |
| `K8S_CA_CERT` | Path to the certificate that signed your cluster's, for clusters with their own (k3d, most self-hosted ones). It is always checked. |
| `K8S_NATS_MONITORING_ENDPOINT` | NATS's monitoring address as KEDA reaches it, e.g. `nats.fluxify.svc:8222`. Needed to scale on waiting workflow runs. |
| `ORCHESTRATOR_SCALE_CPU_PERCENT` | Average CPU use, as a percent of a worker's CPU, above which more workers start. Default: `65`. |
| `ORCHESTRATOR_SCALE_MEMORY_PERCENT` | The same, for memory. Default: `65`. |

Its service account needs only the role the chart creates, in its own
namespace. If the cluster cannot be reached when it starts, it stops and says
what is missing.
