Terraform Enterprise on EKS: The Parts Nobody Documents

Terraform Enterprise's own install docs assume a plain Docker host or a generic Kubernetes cluster with an ingress controller that speaks HTTPS by default. Run it on EKS behind ArgoCD, cdk8s, ExternalSecrets, and Traefik, and you hit at least four failure modes that produce no useful error message — just a 404, a silently-ignored UPDATE, or an OAuth token that pretends a repository doesn't exist.

None of these are TFE bugs. They're the gap between "works on a demo cluster" and "works inside a real platform stack."

Architecture

Rendering diagram…

Infra (RDS, ElastiCache, S3, IAM) comes from Terraform. Kubernetes manifests are generated by cdk8s. ArgoCD syncs them. Secrets live in AWS Secrets Manager and land in the pod via ExternalSecrets. Standard platform-engineering plumbing — the failure modes below all live in the seams between these pieces, not inside any one of them.

Image: AMD64 only

TFE publishes no ARM64 image. If your GitHub Actions runners default to linux/arm64, or you've standardized on Graviton for cost, the build fails outright unless you explicitly override:

FROM images.releases.hashicorp.com/hashicorp/terraform-enterprise:v202507-1
LABEL maintainer="platform-team@example.com"

platforms: "linux/amd64"   # required in both build and deploy workflows

Cheap to fix once you know it's the reason — expensive to diagnose if you don't, because the build error just looks like a generic manifest mismatch.

The database needs the admin user, briefly

Aurora Postgres 16 is the ceiling — TFE does not support 17, and Aurora can't be downgraded in place, only destroyed and recreated. Budget for that constraint before you provision, not after.

The subtler trap: on first boot, TFE needs to create schemas. If TFE_DATABASE_USER points at a least-privilege application user (which is the instinct, and the right end state), schema creation fails with a permissions error that doesn't obviously say "wrong user." Use the admin/root credential for the first boot, let migrations run, then rotate to the scoped app user afterward.

TLS: the certificate header that isn't `CERTIFICATE`

TFE's internal nginx uses ssl_trusted_certificate, which requires the PEM header TRUSTED CERTIFICATE — not the standard CERTIFICATE header every other tool expects. A normal self-signed cert fails at container startup with:

nginx: [emerg] cannot load certificate: PEM_read_bio_X509_AUX() failed (Expecting: TRUSTED CERTIFICATE)

Generate it in the right format from the start:

openssl req -x509 -newkey rsa:2048 \
  -keyout /tmp/tfe-key.pem \
  -out /tmp/tfe-cert.pem \
  -days 3650 -nodes \
  -subj "/CN=tfe.example.internal"

openssl x509 -in /tmp/tfe-cert.pem -trustout -out /tmp/tfe-cert-trusted.pem

head -1 /tmp/tfe-cert-trusted.pem
# Expected: -----BEGIN TRUSTED CERTIFICATE-----

Two more ways this quietly breaks even after you get the header right:

Line wrapping matters. openssl x509 -trustout produces correctly wrapped 64-character lines. If anything downstream collapses the file onto a single line — cat in a shell substitution is the usual culprit — nginx fails to parse it even though the header is technically correct.
Secret key names must match the Helm chart's subPath exactly. The chart mounts subPath: tls.crt and subPath: tls.key. Name your ExternalSecret keys anything else (cert.pem, key.pem — the "obvious" names) and you get a silent empty mount: the pod starts, nginx just gets a zero-byte file, and the resulting error looks identical to a missing cert.

If you're piping any multi-line secret (cert, key, license) through an encryption or secrets API as part of provisioning, json.dumps() the file content in Python before sending it — never cat or $(cat file), which collapses newlines and silently produces the same "technically present, functionally broken" secret.

Traefik: HTTPS-only backend, no plain HTTP path

TFE's Helm chart only exposes an HTTPS backend — there's no plain-HTTP listener at all. That's unusual enough among services behind a typical ingress controller that it's easy to configure exactly the same way as everything else and get nothing but a 404.

Two things are non-negotiable:

A ServersTransport with insecureSkipVerify: true. Traefik verifies backend TLS by default; TFE's backend cert is self-signed. Without this resource, Traefik doesn't error — it silently drops the route and returns 404, which looks identical to a misconfigured IngressRoute.

apiVersion: traefik.io/v1alpha1
kind: ServersTransport
metadata:
  name: tfe-transport
  namespace: tfe
spec:
  insecureSkipVerify: true

The IngressRoute must reconnect over HTTPS, referencing that transport, even if your Traefik setup terminates external TLS upstream (ALB, Global Accelerator) and only forwards plain HTTP internally to everything else:

apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
  name: tfe
  namespace: tfe
spec:
  entryPoints:
    - web
  routes:
    - kind: Rule
      match: Host(`tfe.example.internal`)
      priority: 10
      services:
        - name: terraform-enterprise
          port: 8080
          scheme: https
          serversTransport: tfe-transport

Note the service name: the chart hardcodes it to terraform-enterprise regardless of your Helm release name. Reference the hardcoded name, not whatever you called the release.

If you're generating manifests with cdk8s.Include, put ServersTransport and IngressRoute in separate files. cdk8s.Include only processes the first document in a YAML file — a multi-document file separated by --- silently drops everything after the first ---.

A working route returns a 301 to /session:

curl -sk https://tfe.example.internal/ -o /dev/null -w "%{http_code} %{redirect_url}\n"
# Expected: 301 https://tfe.example.internal/session

The SSO trap that skips your first site admin

This is the one that actually costs people time.

TFE bootstraps its first site admin through a one-time signup endpoint, /admin/account/new. It only renders the signup form while no user exists yet. The moment any user record exists, the route redirects to /app and is permanently closed — there's no supported way to reopen it.

Here's the trap: if the first login to a fresh TFE instance happens via SSO/SAML instead of that local signup form, TFE still considers its job done — a user now exists — but that user is created with is_admin = false. Nobody has site-admin rights. The Site Admin menu and New Organization button don't appear anywhere, for anyone, and there's no UI or API path to fix it.

The correct sequence, in order:

Confirm the pod is healthy and the IngressRoute works.
Before anyone signs in via SSO for the first time, visit /admin/account/new directly and create a local admin account. This is the only flow that auto-sets is_admin = true.
If SSO has already beaten you to it — this happens more often than the docs imply, usually because someone tests the login flow before thinking about admin bootstrapping — promote the existing user directly in the database.

-- fetch admin creds from your secrets manager first
SET search_path TO rails, public;
UPDATE rails.users SET is_admin = true WHERE id = <user_id>;
SELECT id, email, username, is_admin FROM rails.users WHERE id = <user_id>;

The SET search_path line isn't optional. Without it, the UPDATE fails inside an internal trigger (user_hcp_user_email_reservation_check) with relation "hcp_user_email_reservations" does not exist — and depending on your client, that failure can look like it silently did nothing rather than raising a clear error. Always verify with a follow-up SELECT that is_admin actually flipped.

SAML team sync doesn't create the team for you

Once SSO works and an admin exists, the next surprise is team provisioning. TFE grants organization membership through team membership — not directly. Your IdP can send a perfectly correct MemberOf attribute on every login, but if no team with that exact name exists yet in the TFE organization, the user authenticates successfully, gets an account, and is simply not part of the organization. No error. Just an orphaned, authenticated user.

Fix: create the team first, with a name that matches the IdP attribute value exactly, then have the user log out and back in.

curl -sk --header "Authorization: Bearer $USER_TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request POST \
  --data '{
    "data": {
      "type": "teams",
      "attributes": {
        "name": "developers",
        "organization-access": { "manage-workspaces": true }
      }
    }
  }' \
  "https://tfe.example.internal/api/v2/organizations/<org>/teams"

Separately, TFE's SAML config also requires a Single Log-Out URL, even though SLO itself is optional in most SAML deployments and your IdP's setup page may not surface a dedicated logout URL. Leaving it blank fails validation outright. Your IdP's own sign-out endpoint is a legitimate value here, not a placeholder.

And when writing the IdP-side attribute expression for site-admin auto-provisioning: make sure it emits a string, not a boolean. isMemberOfGroupName("admins") alone evaluates to true/false, and TFE's Site Admin Role field matches against a specific string (e.g. site-admins) — a boolean silently never matches, so nobody gets auto-promoted and the failure is invisible unless you go looking for it. A ternary that emits the actual string fixes it.

No inbound webhooks, no VCS auto-apply

If TFE sits behind a private, VPN-only ingress — a reasonable default for an internal platform tool — connecting a VCS provider works fine for the OAuth handshake and API calls (TFE calling out to GitHub), but GitHub's webhook delivery (GitHub calling in to TFE on push) simply cannot reach a private endpoint. There's no stable, publishable GitHub IP range you can safely allow-list without reopening broad public ingress.

Practically: connecting a workspace to a repo in the UI will never auto-trigger a plan on push. The fix is to invert the flow — instead of waiting for GitHub to call TFE, have a CI job that already runs inside your private network call TFE's API directly:

Developer pushes to the repo.
A CI workflow runs on a self-hosted runner inside the network.
The runner calls TFE's API (POST /api/v2/runs, or the CLI-driven workflow) to create a run — outbound from inside the network, so no inbound exposure needed.
TFE executes the plan/apply and posts the commit status back via its own outbound call.

Still attach VCS metadata to the workspace for source display and commit-status reporting — just don't depend on the webhook trigger.

Module registry publish needs its own VCS scope — and a second, invisible SSO check

Two separate gotchas stack on top of each other here, and they produce the identical error message, which makes debugging genuinely confusing.

First: the VCS Provider's scope has three independent toggles — Workspaces, Policy Sets, Modules. Enabling the connection for workspaces does not enable it for module publishing. Miss this and publishing fails with:

Error publishing module — VCS Connection Validation failed: The specified repository "<repo>" doesn't exist or isn't accessible.

— even though the same repo works fine for VCS-linked workspaces. Fix: toggle Modules on in the VCS Provider settings.

Second, and much sneakier: if your GitHub organization enforces SAML SSO, and the VCS Provider is a plain OAuth App (not an installed GitHub App), the OAuth token must be separately authorized for SSO per organization — even for a service account that's already a full org member with read access to the repo. Until that authorization is granted, GitHub silently filters that org's repositories out of every API response made with the token. No error. The repo picker just shows "0 repositories," and direct publish fails with the exact same "doesn't exist or isn't accessible" message as the Modules-toggle issue above.

The tell that distinguishes the two: TFE's manual "can't see your repository? enter its ID" field accepts any string and advances you to a confirmation step regardless of the SSO block, because it doesn't call GitHub's API at that point — the real validation call only happens at final publish. Reaching that confirmation screen proves nothing.

Fix: as the service account, go to github.com/settings/connections/applications, find the OAuth App matching the Client ID shown on the VCS Provider's edit page, and grant/authorize it for the organization. One-click, one-time, per org. No token regeneration or TFE-side change needed.

Field notes

Use a service account for the VCS OAuth connection, not a personal one. It survives offboarding and doesn't break when someone's personal token expires or access is revoked.
Aurora Postgres 16 is a hard ceiling, not a soft recommendation — plan the destroy/recreate cost into your migration timeline if you're already running 17 elsewhere.
Test SSO from a session with a way back in. Confirm a working local admin login exists before testing SAML changes — don't lock yourself out mid-configuration.
The org-level "run/apply everywhere" permission has no narrower sibling. manage-workspaces: true at the org level grants admin on every workspace in the org — there's no org-level "plan/apply but not admin." Scoping down to specific workspaces or a lesser permission means configuring per-workspace Team Access individually.
Every one of these failures is silent by design, not by bug: a dropped route, a filtered repo list, a trigger that swallows an error, a boolean that never matches a string. None of them page you. All of them look like something else until you know the specific shape of this one.

Most of the actual engineering effort in a TFE install isn't the Terraform or the Helm values — it's building the mental map of which of these four subsystems (TLS, ingress, SAML, VCS OAuth) is failing silently, since none of them will tell you directly. Once you know the map, every one of these takes minutes to fix instead of hours to find.