> ## Documentation Index > Fetch the complete documentation index at: https://firebolt-aggregate-helm-docs-pr-38.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Architecture > High-level architecture for FireboltInstance, FireboltEngine, and FireboltEngineClass. This document describes the reconciliation architecture used by the Firebolt Operator. The Firebolt Operator manages three custom resources: * **FireboltInstance** provisions the metadata infrastructure * Metadata service * Gateway * PostgreSQL * **FireboltEngine** deploys compute nodes that run the query engine * **FireboltEngineClass** is an optional template that engines in the same namespace can reference to inherit shared pod-level properties. An engine cannot be created or updated without a ready instance in its namespace. An engine may optionally reference an `FireboltEngineClass` in the same namespace. ## FireboltInstance `FireboltInstance` represents the shared infrastructure that engines need before they can run. The Firebolt Operator creates and keeps this infrastructure healthy, then publishes the endpoints that engines use to connect to it. ### Metadata service The metadata service stores and serves engine metadata. Engines connect to it at startup and during operation so they can read the account and engine state they need to run queries. ### Gateway The gateway is the entry point for client query traffic. It receives a request, identifies the target engine, and forwards the request to healthy engine pods. ### PostgreSQL PostgreSQL is the backing database for the metadata service. The Firebolt Operator can provision a single node PostgreSQL instance for the metadata service. For production workloads you should provision your own PostgreSQL instance. ## FireboltEngine `FireboltEngine` represents the compute nodes that run the query engine. The Firebolt Operator creates the Kubernetes resources for those nodes, rolls new generations when the spec changes, and reports whether the engine is ready to serve traffic. ## FireboltEngineClass `FireboltEngineClass` is an optional namespaced class that engines can reference for shared pod-level configuration. It lets a namespace define common settings such as scheduling, service accounts, annotations, sidecars, and engine image details without repeating them on every engine. ## Resource dependency model The Firebolt Operator enforces a hierarchical dependency from engines to their instance, plus an optional in-namespace dependency on FireboltEngineClass: ```text theme={"theme":{"light":"css-variables","dark":"css-variables"}} ┌───────────────────┐ 1:N ┌────────────────────────┐ │ FireboltInstance │◄────────│ FireboltEngine │ │ │ reads │ │ │ Provisions: │ status │ spec.instanceRef │ │ - PostgreSQL │ │ points to the │ │ - Metadata Svc │ │ instance by name │ │ - Gateway │ │ │ │ │ │ spec.template: │ │ status: │ │ PodTemplateSpec │ │ metadataEndpoint │ │ (top of merge) │ │ │ │ │ │ │ │ Blocked until │ │ │ │ instance has: │ │ │ │ - metadataEndpoint │ │ │ │ - spec.id │ └───────────────────┘ └────────────┬───────────┘ | N:1 │ inherits properties │ when spec.engineClassRef set │ ▼ ┌─────────────────────┐ │ FireboltEngineClass│ │ (namespaced, │ │ optional) │ │ │ │ spec.template: │ │ PodTemplateSpec │ │ (under engine │ │ in the merge) │ │ │ └─────────────────────┘ ``` The rendered engine pod spec is the merge of three layers, top wins: operator defaults \< `FireboltEngineClass.spec.template` (when `spec.engineClassRef` is set) \< `FireboltEngine.spec.template`. List-typed fields (`tolerations`, `initContainers`, sidecars, `env`, `envFrom`, `volumeMounts`, `imagePullSecrets`, `volumes`) concatenate class-then-engine so the engine template extends rather than replaces the class. **Rules:** * Each `FireboltEngine` declares its parent via `spec.instanceRef` (the name of a `FireboltInstance` in the same namespace). * The engine reconciler resolves the referenced instance on every reconcile. If the instance does not exist, is still provisioning, or lacks a populated `metadataEndpoint` or `spec.id`, reconciliation returns an error and requeues. No engine resources are created with missing metadata configuration. This gate only applies to the **stable**, **stopped**, and **creating** phases. These phases may build ConfigMaps referencing instance data. `stopped` is included because a missing ConfigMap can be re-materialized in place against the current instance info even at zero replicas. Phases that operate on already-created resources (**switching**, **draining**, **cleaning**) proceed without blocking on instance readiness. * The engine controller watches `FireboltInstance` resources and re-reconciles all referencing engines when an instance's status changes. This eliminates backoff delay when an instance transitions to ready. * The engine reports its dependency status via a `status.conditions[]` entry of type `InstanceReady`. This condition is written as part of the single `updateStatus` call at the end of each reconcile, avoiding double status writes. Users can inspect this condition to understand why an engine is not progressing. * The instance reconciler is independent and has no dependency on engines. * The optional `spec.engineClassRef` references a namespaced `FireboltEngineClass` **in the engine's own namespace**. The reference is checked at admission time by the FireboltEngine validating webhook. It hard-rejects if no class with that name exists in the engine's namespace, so a runtime "class missing" state is not part of the steady-state status surface. The engine controller watches `FireboltEngineClass` and re-reconciles every same-namespace referencing engine when a class is created, edited, or deleted. A class spec edit therefore rolls a fresh blue-green generation on every consumer engine immediately, rather than waiting for the 30s drift requeue. * FireboltEngineClass is **namespaced** rather than cluster-scoped, unlike `StorageClass` / `IngressClass` / `GatewayClass`, because its template carries namespace-resolved identifiers: `serviceAccountName`, `volumes[*].secret/configMap/persistentVolumeClaim` references, and typically per-tenant IAM annotations. Kubernetes resolves those names in the engine's own namespace at pod-admission time, so co-locating the class and its consumer engines avoids the silent-divergence trap a cluster-scoped class would have created. For example, an identical SA name in different namespaces could have different IAM bindings with no admission error. * `FireboltEngineClass` has its own status reconciler that maintains `status.boundEngines` (the count of FireboltEngines in the same namespace referencing the class) for user-facing visibility. The FireboltEngineClass validating webhook refuses deletion by listing referencing engines live from the API server, scoped to the class's namespace, at admission time rather than trusting the cached count. `status.boundEngines` starts at zero on a freshly admitted class, so a status-based gate would race the reconciler. `failurePolicy: Fail` on the webhook configuration ensures a webhook outage cannot open a deletion window. ## Authentication, TLS, and signing-key rotation Authentication and TLS are Instance-wide policies that the Firebolt Operator provisions in dedicated reconcile steps, each isolated from Postgres, the Metadata Service, and the Gateway so a failure in one never blocks the others. Each step surfaces its outcome through its own status condition (`AuthReady`, `EngineTLSReady`, `GatewayTLSReady`). Engines gate their own reconcile on the Instance's `Status` fields directly (`Status.Auth`, `Status.EngineTLS`), never on the top-level `Ready` roll-up, so a still-provisioning security feature delays only the Engines that depend on it. The three steps run in a fixed order ahead of the components that read their output, and the reconciler logs their errors and continues rather than returning them: | Order | Step | Runs before | Because | | ----- | ----------- | ----------- | ----------------------------------------------------------------------------------- | | 1 | Auth | Engines | Engines mount the admin-password and signing-key Secrets recorded in `Status.Auth`. | | 2 | Engine TLS | Gateway | The Gateway reads `Status.EngineTLS` to decide whether to re-encrypt upstream. | | 3 | Gateway TLS | Gateway | The rendered Envoy config and pod template both read `Status.GatewayTLS`. | Isolation is what makes that safe: an Instance whose auth is misconfigured still brings up a usable Metadata Service and Gateway, and reports the problem on `AuthReady` rather than by failing the whole reconcile. ### Authentication model `spec.auth` configures authentication for every Engine in the Instance, not per Engine. Each Engine runs packdb's embedded `_local` authorization server, which both issues and validates JWTs, so every Engine must run byte-identical `instance.auth.*` config, including the same signing keys, or a token minted by one Engine fails validation on another. The Firebolt Operator resolves `spec.auth` once per Instance and renders the result into every Engine's `config.yaml` from that single source. The model combines two authorization servers: * Native local login through `spec.auth.local`: an admin username and password, plus the JWT signing parameters shared by all Engines. * Optional OIDC bearer-token validation through `spec.auth.oidc`: one or more trusted identity providers whose tokens Engines accept. packdb validates tokens against each provider's published keys; it never initiates a login flow itself. The Firebolt Operator provisions the crypto material auth needs. It runs an admin-password preflight that verifies the user-supplied password Secret exists and carries its key, and never generates an admin password, because a credential the user never sees is not usable. It provisions the shared JWT signing keypair through cert-manager. Both are mounted into every Engine pod and rendered as `instance.auth.admin.password_file` and `instance.auth.local.signing_keys[]`, so no plaintext password or private key appears in the rendered config. `AuthReady` is deliberately excluded from the top-level Instance `Ready` roll-up: auth provisioning has no bearing on whether the Metadata Service or Gateway are usable. `EngineTLSReady` and `GatewayTLSReady`, by contrast, are folded into `Ready`, so the Instance never advertises a secure posture it has not yet reached. ### Engine-listener TLS `spec.tls.engine` terminates TLS on each Engine's query listener. TLS replaces plaintext on the same port (3473): the Engine serves a single listener, not a dual plaintext-and-TLS pair. The Gateway then re-encrypts upstream to Engines, verifying each one against the Engine CA trust bundle. The Gateway's upstream TLS context explicitly offers every elliptic curve an Engine certificate may use. Envoy offers only X25519 and P-256 by default, which its own TLS stack then rejects a P-384 Engine certificate against — and P-384 is the Firebolt Operator's default. The curve list therefore has to track the key sizes `spec.tls.engine.certManager.size` accepts. The Gateway routes all Engine traffic through one `dynamic_forward_proxy` cluster whose `transport_socket` Envoy copies verbatim into every per-authority sub-cluster it synthesizes. Upstream TLS is therefore fleet-wide: the Gateway cannot speak TLS to one Engine and plaintext to another. Because of that, the upstream switch (`Status.EngineTLS.Reencrypting`) flips only once every Engine has converged onto a TLS-serving generation. Disable is symmetric: the Firebolt Operator retains the trust anchor and keeps re-encrypting until every Engine has drained back to plaintext. `EngineTLSReady` is convergence-gated, reporting `True` only once the whole fleet re-encrypts. Engines still roll onto TLS on the provisioned fact alone (`Status.EngineTLS` populated), not on the condition. Gating the Engine roll on the condition would deadlock the enable ramp, because the roll is what produces the convergence the condition waits for. ### Gateway downstream TLS and fail-closed posture `spec.tls.gateway` terminates client-facing TLS on the Envoy listener; adding a client CA through `clientCASecretRef` upgrades the listener to mutual TLS. On a tightening transition (plaintext to TLS, one-way to mutual TLS, or a client-CA swap), the Firebolt Operator serves fail-closed rather than risk a fail-open window. It clears `Status.GatewayTLS`, omits the client-facing listener entirely, and drains the old looser pods before any new secure pod starts (`MaxUnavailable=100%`, `MaxSurge=0`). The client port rejects all connections for a brief window during the transition, but the Gateway is never fail-open. Liveness and readiness probes ride the always-plaintext metrics port, so pods stay alive through the fail-closed wait. Loosening and steady-state transitions carry no fail-open risk and skip staging. ### Signing-key rotation When `spec.auth.local.signingKeys.rotationInterval` is set, the Firebolt Operator runs its own coordinated rotation instead of relying on cert-manager's renewal schedule. A new key is minted in a validate-only role first — used to validate tokens but not yet to sign them — and the state machine advances at most one step per reconcile. Every Engine signs with the first key in the config it has rolled onto, and rolling is not instantaneous, so at any moment different Engines may be signing and validating against different generations of that config. Each irreversible step is therefore gated on fleet convergence: every Engine's `Status.observedAuthHash` must equal the Instance-computed `authHash` before the step may run. What each gate prevents: | Step | Gate | Without the gate | | ------------------------ | --------------------------------------------- | ------------------------------------------------------------------------------ | | `mint` | none — the key is validate-only | Nothing signs with it, so there is nothing to protect. | | `promote` | every Engine renders the new key | A rolled Engine signs tokens a not-yet-rolled Engine cannot validate. | | stamp `RetireEligibleAt` | every Engine renders the promotion | The retain window starts while Engines are still signing with the demoted key. | | `retain` → `Removing` | `RetireEligibleAt` + `retainDuration` elapsed | A token still inside its validity window loses the key that validates it. | | `delete` | every Engine renders the removal | An Engine still needs the key to validate a token signed before demotion. | The third gate is the subtle one. `RetireEligibleAt` anchors the retain window at confirmed post-promotion convergence, not at demotion: every Engine keeps signing with the demoted key until it rolls onto the promoted configuration, so `retainDuration` must measure from the instant no Engine *can* still be signing with the key, not from the instant the operator decided it should stop. That is a machine-checked claim rather than an argued one. [formal/SigningKeyRotation.tla](https://github.com/firebolt-db/firebolt-kubernetes-operator/blob/main/formal/SigningKeyRotation.tla) models the state machine against a fleet whose Engines roll independently, and its `AnchorAtDemotion` constant switches the anchor to demotion. Under that anchor TLC produces a counterexample in which one Engine still signs with a key another has already dropped; under the shipped anchor it finds none. `make formal-check-counterexample` requires that counterexample to keep existing, so weakening a convergence gate fails the build. Because the gates wait indefinitely by design — an Engine that never converges parks the rotation rather than risking a validation gap — a parked rotation is reported rather than left to look idle. `Status.Auth.pendingRotationStep`, `pendingSince`, and `laggingEngines` name the step and the Engines it is waiting for, `AuthReady` carries reason `RotationWaitingForEngines` while staying `True` (the Active key still signs and validates, so the Instance is not unhealthy), and the `firebolt_instance_signing_key_rotation_*` metrics expose the same state for alerting. See [Monitoring](./monitoring) for the metrics and an example alert. The Firebolt Operator ships with the validating webhook off by default, so it does not lean on admission alone. Every `ValidateAuth` and `ValidateTLS` invariant has a controller-side re-check, and the transition-immutability invariants (such as a frozen signing algorithm or engine issuer while TLS stays enabled) have CRD CEL twins that hold at the API server regardless. ### Trust-bundle cutover The Gateway's Engine trust bundle is the union of the CA certificate behind every live Engine generation's TLS Secret. Because the CA material behind an issuer can rotate even though the issuer name is immutable, an older live generation may carry a different CA than a newer one; trusting the union lets the Gateway verify both during the overlap. The bundle is reassembled from live generations every reconcile, so a CA drops out on its own once the last generation using it retires, with no separate convergence bookkeeping. A generation's Service selector may not flip to that generation until the Gateway confirms it trusts the generation's CA, that is, until the CA's fingerprint appears in `Status.RolledEngineTrustCAs`. The gate is vacuous until the Gateway actually re-encrypts upstream, because before that the Gateway talks plaintext and verifies nothing. ### Known limitation: transient outage when toggling engine TLS Enabling or disabling engine TLS on an already-running, traffic-serving fleet has a transient Gateway-to-Engine outage window. Because Engines serve TLS only on the single query port, and the Gateway's upstream protocol is fleet-wide and rolled in a separate step, an Engine that has cut over ahead of the fleet is briefly unreachable from the not-yet-switched Gateway, and the reverse holds on disable. This window is bounded to the transition of a deliberate admin operation, and engine TLS is a new opt-in feature, so plaintext deployments see no regression. A fully hitless transition would require dual-listener (plaintext and TLS) support on the Engine side in packdb, which is outside the Firebolt Operator's scope. ## Design principles The Firebolt Operator uses a **level-triggered** (not edge-triggered) reconciliation model. Each invocation of `Reconcile` reads the full desired state (`.spec`) and the full observed state (cluster resources), computes the delta, and applies it. The reconciler does not depend on knowing *what changed*. It only depends on *what is*. This means: * **Idempotent**: calling `Reconcile` twice with the same inputs produces the same result. * **Crash-safe**: If the Firebolt Operator crashes at any point, the next reconciliation will observe the actual cluster state and resume from the correct phase. * **No queued operations**: there is no internal queue of "things to do". The status phase and observed resources determine the next action. ## Detailed pages The rest of the design is split by audience and task: * [Engine reconciliation](./engine/engine-reconciliation) describes the FireboltEngine reconcile loop, phase machine, generation model, status behavior, error handling, crash recovery, and resource ownership. * [Instance reconciliation](./instance/instance-reconciliation) describes how FireboltInstance provisions PostgreSQL, metadata, and gateway resources. * [FireboltEngineClass design](./engineclass/fireboltengineclass-design) describes FireboltEngineClass as a namespaced class abstraction and documents its pod-template merge behavior. * [Engine rollouts](./engine/engine-rollouts) describes drain checks and rolling update parameters. * [Auto-stop and wake-up](./engine/auto-stop-and-wake-up) describes Firebolt Operator-driven auto-stop and the gateway wake-up annotation protocol. * [Gateway query routing](./instance/gateway/gateway-query-routing) describes Envoy routing, zero-downtime shutdown behavior, and why the Firebolt Operator does not gate on EndpointSlice updates. * [Gateway sizing](./instance/gateway/gateway-sizing) describes replica count, memory limits, and the 2 MiB per-connection buffer constraint.