> ## Documentation Index > Fetch the complete documentation index at: https://firebolt-aggregate-helm-docs-pr-38.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Security model > Operator vs. platform responsibilities for pod hardening, network isolation, resource bounds, and secrets. This document records the Firebolt Operator's threat-model boundary: what the Firebolt Operator hardens by default versus what is the cluster-platform team's responsibility. ## What the Firebolt Operator enforces The Firebolt Operator stamps a hardened security posture on the workloads it renders. How much you can change that posture depends on the CRD surface. Engine pods expose a wide pod-template merge layer through `FireboltEngineClass` and `FireboltEngine`. Gateway and metadata primary containers keep operator-owned hardening and only let you set image and resources on the primary container. Per-field allowlists for each template surface live in `api/v1alpha1/operatorauthority.go` (`PodTemplateRules` per component) and in the CRD reference pages linked in each subsection below. ### Engine When neither a referenced `FireboltEngineClass` nor the engine's own `spec.template` sets a container `securityContext`, the Firebolt Operator applies these defaults on the `engine` container: | Field | Default | | -------------------------- | -------------------------------------------------------------------------------------------------------------------------- | | UID / GID | 3473 / 3473 | | `runAsNonRoot` | `true` | | Capabilities | drop `ALL` | | `allowPrivilegeEscalation` | `false` | | Read-only root filesystem | `true` (unix domain socket via emptyDir at `/run/firebolt`; status file and engine data on the PVC at `/var/lib/firebolt`) | | Seccomp | not set | You can replace that entire container `securityContext` wholesale through `FireboltEngineClass.spec.template` or `FireboltEngine.spec.template.spec.containers[name=="engine"].securityContext`. The engine template wins over the class. There is no merge or floor. A template value replaces the operator default completely, including the ability to weaken hardening. Pod-level `spec.template.spec.securityContext` is also user-settable. The Firebolt Operator only stamps `fsGroup` (3473) and `fsGroupChangePolicy: OnRootMismatch` when you leave those fields unset. Sidecar containers and init containers pass through from your template without operator hardening. #### Writable directories and arbitrary UIDs The engine image owns two writable directories: the data dir (`/var/lib/firebolt`, the PVC mount) and the unix socket dir (`/run/firebolt`, an `emptyDir`). Both are group-owned by root (GID 0) with mode `2770` (`g=u` plus the setgid bit), not the broad `777` the image used previously. The read-only application payload at `/opt/firebolt` stays `firebolt:firebolt 0755`. This follows the OpenShift "arbitrary UID" convention: a platform that assigns a random non-namespace UID still places the process in group 0, so the GID-0 group ownership keeps both directories readable and writable, while `other` gets nothing. The setgid bit makes files the engine creates at runtime inherit group 0. The engine runs with `umask 0007` so it does not strip those group bits off new files. Under the default UID/GID `3473/3473` the user owns the directories outright, so the same layout works without OpenShift. Mounted volumes override the image's directory permissions. When a platform assigns an arbitrary UID, give the engine a writable data dir either through the stamped `fsGroup` (which `chown`s the PVC to the group) or by mounting volumes already group-0-writable, and avoid pinning a `runAsUser` that is not a member of group 0. Validating webhooks on `FireboltEngine` and `FireboltEngineClass` (and the engine reconciler when webhooks are off) reject operator-owned paths on engine templates: command, args, ports, probes, reserved env keys, and `firebolt.io/*` labels. They do not reject a weakened `securityContext`. See the [FireboltEngineClass CRD reference](./crd-reference/fireboltengineclass-crd-reference) and [FireboltEngine CRD reference](./crd-reference/engine-crd-reference). ### Gateway (Envoy) The Envoy primary container is operator-rendered end to end. The validating webhook rejects user input on its `securityContext`. You may only set `image` and `resources` on `spec.gateway.template.spec.containers[name=="envoy"]`. The Firebolt Operator stamps: | Field | Value | | -------------------------- | ----------------------------------------- | | UID | 101 | | `runAsNonRoot` | `true` | | Capabilities | drop `ALL` | | `allowPrivilegeEscalation` | `false` | | Read-only root filesystem | `true` (scratch via `emptyDir` at `/tmp`) | | Seccomp | not set on the container | Pod-level fields on `spec.gateway.template` (node selector, tolerations, `securityContext`, sidecars, init containers) pass through when allowed by the template rules. See the [FireboltInstance CRD reference](./crd-reference/instance-crd-reference). ### Metadata (Pensieve) The metadata primary container follows the same pattern as the gateway. Operator-stamped hardening applies to `containers[name=="metadata"]`. You may set only `image` and `resources` there: | Field | Value | | -------------------------- | ------------------------------------------------------ | | UID | 1111 (pinned to the image's `dedicated-pensieve` user) | | `runAsNonRoot` | `true` | | Capabilities | drop `ALL` | | `allowPrivilegeEscalation` | `false` | | Read-only root filesystem | `true` (scratch via `emptyDir` at `/tmp`) | | Seccomp | not set on the container | At the pod level, the Firebolt Operator floors any user-supplied `PodSecurityContext` to `runAsNonRoot: true` with UID/GID 1111 and sets `seccompProfile: RuntimeDefault` when you do not supply one. A template cannot run the metadata pod under a different user without failing image ownership checks. ### PostgreSQL (internal) Internal PostgreSQL has no user template surface. The Firebolt Operator stamps hardening on every reconcile: | Level | Posture | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Pod | UID/GID 70, `runAsNonRoot: true`, `seccompProfile: RuntimeDefault` | | Container | UID 70, `runAsNonRoot: true`, drop `ALL` capabilities, `allowPrivilegeEscalation: false`, read-only root filesystem (`PGDATA` on the PVC, runtime sockets on `emptyDir`) | The Firebolt Operator's own manager RBAC (rendered by the chart from the canonical `config/rbac/role.yaml`) is the minimal set of verbs needed to manage the three CRDs and their generated resources. The Firebolt Operator does not request `*` on any namespaced verb and does not request `nodes`, `clusterrolebindings`, or any other cluster-scope mutation. The chart renders the manager rules in one of two shapes, picked at install time by `watchNamespaces`: * Empty list (default): one cluster-scoped `ClusterRole` and one `ClusterRoleBinding`. The manager cache spans every namespace. * Non-empty list, e.g. `{tenant-a, tenant-b}`: a `Role` plus `RoleBinding` pair in each listed namespace, no `ClusterRole`. The manager cache spans only those namespaces. Use this posture when multi-tenant compliance constraints bound the operator's blast radius. The default `FireboltInstance.spec.metricScrapeMode=PodIP` reaches engine metrics through pod IPs and does not need `pods/proxy: get`, so the chart's manager RBAC does not include that verb. Setting `metricScrapeMode=ApiserverProxy` on any FireboltInstance requires `rbac.apiserverProxyGrant=true` on the operator chart, which renders a dedicated `ClusterRole` (or per-namespace `Role` when `watchNamespaces` is set) granting only that one verb. Without the toggle, the metric scrape surfaces as a 403 from the apiserver. Resource maxima on the engine container's `resources` block (set under `FireboltEngine.spec.template.spec.containers[name=="engine"]` or inherited from a referenced `FireboltEngineClass`) are enforced by the validating webhook (see "Resource bounds" below). The bounds protect a namespace from accidentally admitting an engine whose requests would starve sibling workloads at scheduling time. ## Authentication and transport encryption Both are opt-in and Instance-wide. Neither is configured per Engine: every Engine in an Instance runs the same authentication configuration, because each one both issues and validates JWTs and a token minted by one must validate on another. See [architecture](./architecture) for the provisioning model. ### Authentication `spec.auth` layers two authorization servers: * **The local user store**, always present when auth is enabled. It holds the Instance admin plus any users created through SQL, and backs password login through the Engine's embedded authorization server. * **OIDC bearer validation** through `spec.auth.oidc`, optional. Engines accept JWTs issued by the trusted providers you configure. Engines validate those tokens; they never run a login flow themselves, so no client secret for your identity provider is ever held by the Instance. **The Firebolt Operator never generates the admin password.** It is supplied through a Secret you create, referenced by `spec.auth.local.admin.password`. A credential the operator invented and you never saw would not be usable, so the reconcile preflights the Secret's existence and reports `AuthReady=False/AdminSecretMissing` rather than inventing one. The password is mounted into each Engine as a file and referenced by path, so no plaintext credential appears in a rendered config. The shared JWT signing keypair is provisioned through cert-manager against an `Issuer` or `ClusterIssuer` you supply. Signing keys are cert-manager-only — unlike TLS, there is no bring-your-own `secretRef` path — and the private key is mounted by path, never inlined. ### Engine and Gateway TLS `spec.tls.engine` terminates TLS on each Engine's query listener; `spec.tls.gateway` terminates client-facing TLS on the Gateway. Both accept either a cert-manager issuer or a `secretRef` to a certificate you manage yourself. Two properties are worth knowing before you enable them: * **TLS replaces plaintext on the same port; it does not add a second listener.** An Engine serving TLS is not reachable in plaintext. Enabling or disabling Engine TLS on a fleet already serving traffic therefore has a brief Gateway-to-Engine outage window — see the transient-outage note in [architecture](./architecture). * **Tightening the Gateway's posture is fail-closed.** Going from plaintext to TLS, from one-way to mutual TLS, or swapping the client CA withholds the client-facing listener and drains the old, looser pods before any secure pod serves. The client port refuses connections for that window rather than briefly accepting clients the new posture is meant to reject. Liveness and readiness probes ride the always-plaintext metrics port, so pods stay alive through it. Adding `clientCASecretRef` upgrades the Gateway listener to mutual TLS, verifying client certificates against that CA. Loosening transitions carry no fail-open risk and are not staged. ## What the Firebolt Operator does not enforce ### Pod-template authors can mount other Secrets in the namespace A `FireboltEngine`, `FireboltEngineClass`, or `FireboltInstance` pod template may declare volumes and sidecar containers, and a volume may reference any Secret in that namespace. **Treat edit rights on those templates as equivalent to read rights on the namespace's Secrets**, and scope RBAC accordingly: someone who can edit a template can mount a Secret and read it from a sidecar whose image and command they control, without any `get secrets` permission. The operator's own auth and TLS Secrets are excluded from this. Every route a template has to them is covered, and the checks match the Secret being *referenced* rather than the name the author gave it, so renaming evades nothing: * A **volume** whose source reaches one — including through a `projected` source — is dropped from the rendered pod and reported on the component's condition. It is reported rather than refused on purpose: the kubelet re-syncs a Secret volume for the life of the pod, so a volume that named a not-yet-existing Secret starts yielding real key material once the operator creates it. The remedy is to replace the pod, and refusing to render would freeze the Engine on the very generation holding the exposed volume. * A **container** that reads one into its environment, through `env[].valueFrom.secretKeyRef` or `envFrom[].secretRef`, is refused outright. An env reference resolves once when the pod starts and is never re-synced, so declining to render is a complete remedy. Both apply to the primary container, sidecars, and init containers, and to a referenced `FireboltEngineClass` template as well as the resource's own. ### Network isolation between pods **The Firebolt Operator emits no NetworkPolicy objects.** All pod-to-pod and pod-to-external traffic is governed by whatever the cluster's CNI and NetworkPolicy controller already enforce. In a default Kubernetes install with no NetworkPolicy controller installed, every pod can reach every other pod on every port. This is a deliberate scoping decision: NetworkPolicy semantics depend on the CNI plugin (Calico, Cilium, Antrea, etc.), the cluster's default-allow-vs-default-deny posture, and the Firebolt Operator-vs-platform ownership boundary for security primitives. Encoding any of those assumptions into Firebolt Operator-emitted NetworkPolicies would either be a no-op (no controller installed) or actively wrong for the deployment target. Platform teams should apply NetworkPolicies covering at least the allowed flows below. Recommended selectors: | Pod kind | Selector | | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | | Engine pods | `firebolt.io/engine` exists (matches any generation of any engine in the namespace) | | Gateway / Metadata / PostgreSQL | `firebolt.io/component={gateway,metadata,postgres}` | | Instance scoping | `firebolt.io/instance=` (present on instance-level workloads only. Engines carry `firebolt.io/engine` instead, which is unique per engine) | #### Allowed flows | From | To | Port | Purpose | | ---------------- | --------------------------- | ------------------ | -------------------------------------------- | | External clients | Gateway | 8080 | Query traffic (HTTP) | | Gateway | Engine pods | 3473 | Query forwarding | | Engine pods | Metadata | 7000 | Metadata gRPC | | Metadata | PostgreSQL | 5432 | Metadata catalog reads/writes | | Engine pods | External object store | 443 / 80 | Managed-storage reads/writes (S3, GCS, etc.) | | Prometheus | Engine / Gateway / Operator | 9090 / 9090 / 8443 | Metrics scraping | | kube-apiserver | Operator webhook | 9443 | Admission control | Engine-to-engine, engine-to-PostgreSQL, gateway-to-metadata, and gateway-to-PostgreSQL are not required by any Firebolt Operator-managed control flow and should be denied. #### Example baseline NetworkPolicy The snippet below denies all ingress and egress by default in the instance's namespace, then re-allows the flows above. It assumes the gateway's external clients live in a `firebolt-clients` namespace. The selector should be adjusted to match the actual client topology. ```yaml theme={"theme":{"light":"css-variables","dark":"css-variables"}} apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: deny-all spec: podSelector: {} policyTypes: [Ingress, Egress] --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: gateway-ingress spec: podSelector: matchLabels: firebolt.io/component: gateway policyTypes: [Ingress] ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: firebolt-clients ports: - port: 8080 protocol: TCP --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: engine-from-gateway spec: podSelector: matchExpressions: - key: firebolt.io/engine operator: Exists policyTypes: [Ingress] ingress: - from: - podSelector: matchLabels: firebolt.io/component: gateway ports: - port: 3473 protocol: TCP --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: metadata-from-engine spec: podSelector: matchLabels: firebolt.io/component: metadata policyTypes: [Ingress] ingress: - from: - podSelector: matchExpressions: - key: firebolt.io/engine operator: Exists ports: - port: 7000 protocol: TCP --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: postgres-from-metadata spec: podSelector: matchLabels: firebolt.io/component: postgres policyTypes: [Ingress] ingress: - from: - podSelector: matchLabels: firebolt.io/component: metadata ports: - port: 5432 protocol: TCP ``` Add egress allow-rules in the same shape. The Firebolt Operator does not generate them. ### Namespace-level resource ceilings A `ResourceQuota` capping the aggregate of `requests.cpu` / `requests.memory` / pod count / PVC size across all engines in a namespace is a platform concern. The Firebolt Operator enforces per-engine upper bounds in admission (see below) but does not emit a ResourceQuota. The per-namespace budget is a deployment-target decision (test cluster vs. multi-tenant production). ### Image provenance and supply-chain attestation The Firebolt Operator pulls whatever image the user supplies via `FireboltEngine.spec.template.spec.containers[engine].image`, the referenced `FireboltEngineClass.spec.template.spec.containers[engine].image`, or the embedded defaults shipped with the Firebolt Operator binary (merged in that order, top wins). The Firebolt Operator does not validate signatures, attestations, or SBOMs. Use a cluster-level admission controller (Kyverno, Sigstore Policy Controller, etc.) if image-policy enforcement is required. ## Resource bounds The FireboltEngine validating webhook rejects engine-container `resources` entries above Firebolt Operator-configured maxima. The gate resolves the effective container the same way the reconciler will: the engine's own `spec.template.spec.containers[name=="engine"].resources` wins wholesale when set, otherwise the referenced FireboltEngineClass's container resources fill in. Both sources are checked, so a class with oversized requests cannot escape admission by being referenced from an engine that omits its own resources. The error message names the source class when the offending value came from class so the user knows which side to edit. This is a defense-in-depth control against accidental over-provisioning. A typoed `100Gi` instead of `10Gi` is caught at admission rather than at scheduling time when it would silently exhaust namespace capacity and block other engines. The maxima are configurable at Firebolt Operator install time. Defaults are sized for typical production deployments. Override via Helm values when running larger or smaller engines. ## Secrets handling The Firebolt Operator generates one Secret: the internal PostgreSQL credentials (`-metadata-postgres-creds`). The password is generated at first reconcile, persisted to a Kubernetes Secret with owner reference to the FireboltInstance, and never re-rotated by the Firebolt Operator. The metadata and PostgreSQL Deployments load it via `envFrom`. User-supplied credentials (external PostgreSQL, external object store) are referenced by name on the FireboltInstance / FireboltEngine spec and resolved at reconcile time. The Firebolt Operator never reads or materializes a user-supplied secret's value into its own status, events, or logs. When auth or TLS is enabled, these additional Secrets exist: | Secret | Origin | Held by | Rotation | | ----------------------- | ------------------------------------------------------------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------- | | Admin password | You create it; referenced by `spec.auth.local.admin.password` | Every Engine, mounted read-only by path | Yours to rotate. Editing the Secret rolls the fleet onto the new value. | | JWT signing keys | cert-manager, one per key generation | Every Engine, mounted read-only by path | Operator-coordinated when `rotationInterval` is set; otherwise a single permanent key. | | Engine TLS certificate | cert-manager, or your own via `secretRef` | Engine pods, plus its CA in the Gateway's trust bundle | cert-manager's own renewal, per generation. | | Gateway TLS certificate | cert-manager, or your own via `secretRef` | Gateway pods | cert-manager's own renewal. | | Gateway client CA | You create it; referenced by `clientCASecretRef` | Gateway pods (public certificate only) | Yours. A swap is treated as a tightening transition. | The operator reads a signing key's public-key fingerprint to detect key replacement, and a Secret's `resourceVersion` to detect an in-place password change. Neither the private key nor the password value is ever written to status, events, or logs. ## See also * [architecture](./architecture): Full Firebolt Operator architecture. * [monitoring](./monitoring): Metrics and alerts the Firebolt Operator exposes. * [instance-crd-reference](./crd-reference/instance-crd-reference): Firebolt Operator-owned vs. user-owned fields on FireboltInstance. * [fireboltengineclass-crd-reference](./crd-reference/fireboltengineclass-crd-reference): Firebolt Operator-owned vs. user-owned fields on FireboltEngineClass.