Infinitely opening browser when using Pomerium with Pomerium-CLI for Kubernetes auth

This issue has been created since 2022-03-23.

What happened?

After upgrading to Pomerium v0.17.0, using the CLI to authenticate / authorize Kubernetes and running any kubectl command e.g. kubectl get pods, browser windows are opening indefinitely after the login!

Until Pomerium v0.16.4 this issue was never present.

What did you expect to happen?

Login to Kubernetes should be working as expected!

How'd it happen?

See above!

What's your environment like?

  • Pomerium version (retrieve with pomerium --version): 0.17.0
  • Server Operating System/Architecture/Cloud: Installed Pomerium with Helm using the official chart and running on AWS EKS

What's your config.yaml?

autocert: false
dns_lookup_family: V4_ONLY
address: :80
grpc_address: :80
authenticate_service_url: https://pomerium.mydomain.com
authorize_service_url: http://pomerium-authorize.pomerium.svc.cluster.local
databroker_service_url: http://pomerium-databroker.pomerium.svc.cluster.local
idp_provider: oidc
idp_scopes: [openid profile email groups offline_access]
idp_provider_url: https://myidp.domain.com
insecure_server: true
grpc_insecure: true
forward_auth_url: https://forwardauth.mydomain.com
idp_client_id: REDACTED
idp_client_secret: REDACTED
idp_service_account: pomerium-authenticate
databroker_storage_tls_skip_verify: true    
routes:
  - allow_spdy: true
    allow_websockets: true
    allowed_idp_claims:
      groups:
      - Group A
      - Group B
    from: https://dashboard.mydomain.com
    kubernetes_service_account_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    to: http://kubernetes-dashboard.kube-system.svc:443
  - allow_spdy: true
    allow_websockets: true
    allowed_idp_claims:
      groups:
      - Group A
      - Group B
    cors_allow_preflight: true
    from: https://kubernetes.mydomain.com
    kubernetes_service_account_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    tls_skip_verify: true
    to: https://kubernetes.default.svc
  - from: https://pomerium.mydomain.com
    to: http://pomerium-authenticate.pomerium.svc.cluster.local
    allow_public_unauthenticated_access: true

What did you see in the logs?

Authenticate:

"error":"rpc error: code = Unknown desc = unknown directory provider oidc","time":"2022-03-23T10:58:12Z","message":"directory: failed to refresh user data"

Authorize, Data Broker and Proxy nothing.

Additional context

My Kubeconfig file looks like:

apiVersion: v1
clusters:
- cluster:
    server: https://kubernetes.mydomain.com
    insecure-skip-tls-verify: true
  name: my-cluster-via-pomerium
contexts:
- context:
    cluster: my-cluster-via-pomerium
    user: my-cluster-via-pomerium
  name: my-cluster-via-pomerium
current-context: my-cluster-via-pomerium
kind: Config
preferences: {}
users:
- name: my-cluster-via-pomerium
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - k8s
      - exec-credential
      - --disable-tls-verification
      - https://kubernetes.mydomain.com
      command: pomerium-cli
      env: null
      provideClusterInfo: false
sherifkayad wrote this answer on 2022-03-23

@wasaga any idea why that might be happening? am I missing something in the configs?

travisgroth wrote this answer on 2022-03-23

@sherifkayad could you share your helm values as well?

sherifkayad wrote this answer on 2022-03-23

@travisgroth my values.yml file is as follows:

# For detailed explanation of each of the configuration settings see
# https://www.pomerium.io/reference/

# settings that are shared by all services
# settings that are shared by all services
config:
  # routes under this wildcard domain are handled by pomerium
  rootDomain: "mydomain.com"
  sharedSecret: "REDACTED"
  cookieSecret: "REDACTED"
  generateTLS: true
  extraOpts: {}
  insecure: true
  insecureProxy: false
  administrators: ""
  routes:
    - from: https://dashboard.mydomain.com
      to: http://kubernetes-dashboard.kube-system.svc:443
      allow_spdy: true
      allow_websockets: true
      allowed_idp_claims:
        groups:
          - Group A
          - Group B
      kubernetes_service_account_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    - from: https://kubernetes.mydomain.com
      to: https://kubernetes.default.svc
      tls_skip_verify: true
      allow_spdy: true
      allow_websockets: true
      cors_allow_preflight: true
      allowed_idp_claims:
        groups:
          - Group A
          - Group B
      kubernetes_service_account_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
authenticate:
  name: "pomerium"
  proxied: true
  # see https://www.pomerium.io/docs/identity-providers.html
  idp:
    provider: oidc
    clientID: "REDACTED"
    clientSecret: "REDACTED"
    url: "https://myidp.domain.com"
    scopes:
      - openid
      - profile
      - email
      - groups
      - offline_access
    serviceAccount: pomerium-authenticate
  replicaCount: 2
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 50
    targetMemoryUtilizationPercentage: 50
  pdb:
    enabled: true
    minAvailable: 1
  service:
    annotations: {}
    nodePort: ""
    type: ClusterIP
  deployment:
    annotations: {}
    extraEnv: {}
    podAnnotations: {}
  serviceAccount:
    annotations: {}
    nameOverride: ""
authorize:
  replicaCount: 2
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 50
    targetMemoryUtilizationPercentage: 50
  pdb:
    enabled: true
    minAvailable: 1
  service:
    annotations: {}
    type: ClusterIP
    clusterIP: None
  deployment:
    annotations: {}
    extraEnv: {}
    podAnnotations: {}
  serviceAccount:
    annotations: {}
    nameOverride: ""
databroker:
  replicaCount: 2
  pdb:
    enabled: true
    minAvailable: 1
  service:
    annotations: {}
    type: ClusterIP
    clusterIP: None
  deployment:
    annotations: {}
    extraEnv: {}
    podAnnotations: {}
  serviceAccount:
    annotations: {}
    nameOverride: ""
  storage:
    type: "redis"
    connectionString: "REDACTED"
    tlsSkipVerify: true
    clientTLS:
      existingSecretName: ""
      existingCASecretKey: ""
      cert: ""
      key: ""
      ca: ""
proxy:
  replicaCount: 2
  autoscaling:
    enabled: false
    minReplicas: 1
    maxReplicas: 5
    targetCPUUtilizationPercentage: 50
    targetMemoryUtilizationPercentage: 50
  pdb:
    enabled: true
    minAvailable: 1
  authenticateServiceUrl: ""
  authorizeInternalUrl: ""
  service:
    annotations: {}
    nodePort: ""
    type: ClusterIP
  deployment:
    annotations: {}
    extraEnv: {}
    podAnnotations: {}
  serviceAccount:
    annotations: {}
    nameOverride: ""
  redirectServer: true
apiProxy:
  enabled: true
  ingress: true
  name: "kubernetes"
ingressController:
  enabled: true
  ingressClassResource:
    enabled: false
    default: false
    name: pomerium
    controllerName: pomerium.io/ingress-controller
    parameters: {}
    defaultCertSecret: ""
  image:
    repository: "pomerium/ingress-controller"
    tag: "v0.17.0"
  deployment:
    annotations: {}
    extraEnv: {}
  serviceAccount:
    annotations: {}
    nameOverride: ""
  config:
    namespaces: []
    ingressClass: k8s.io/ingress-nginx
    updateStatus: true
    operatorMode: true
  service:
    annotations: {}
    type: ClusterIP
forwardAuth:
  name: ""
  enabled: true
  # Will not create an ingress. ForwardAuth is ony accessible as internal service.
  internal: false
service:
  # externalPort defaults to 80 or 443 depending on config.insecure
  externalPort: ""
  annotations: {}
  # ===  GKE load balancer tweaks; default on until I can figure out
  # how the hell to escape this string from the helm CLI
  # cloud.google.com/app-protocols: '{"https":"HTTPS"}'
  labels: {}
  grpcTrafficPort:
    nameOverride: ""
  httpTrafficPort:
    nameOverride: ""
ingress:
  secretName: ""
  secret:
    name: "pomerium-tls"
    cert: ""
    key: ""
  tls: {}
  enabled: true
  hosts: []
  # Sets Ingress/ingressClassName. This way ingress resources are able to bound specific ingress-controllers. Kubernetes version >=1.18 required.
  # Ref: https://kubernetes.io/docs/concepts/services-networking/ingress/#ingress-class
  className: nginx
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
  # Ingress pathType (e.g. ImplementationSpecific, Prefix, .. etc.) might also be required by some Ingress Controllers
  pathType: ImplementationSpecific
resources: {}
# limits:
#   cpu: 1
#   memory: 600Mi
# requests:
#   cpu: 100m
#   memory: 300Mi

priorityClassName: ""
# Affinity for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
affinity: {}
# Tolerations for pod assignment
# Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
tolerations: []
# Node labels for pod assignment
# Ref: https://kubernetes.io/docs/user-guide/node-selection/
nodeSelector: {}
podAnnotations: {}
podLabels: {}
replicaCount: 1
# For any other settings that are optional. for a complete listing see:
# https://www.pomerium.io/docs/config-reference.html
extraEnv:
  CERTIFICATE_AUTHORITY_FILE: "/ca-cert/MYCERT.crt"
  AUTOCERT: "false"
  LOG_LEVEL: "error"
extraEnvFrom: []
extraArgs: {}
extraVolumes:
  - name: ca-cert
    secret:
      secretName: ca-cert
extraVolumeMounts:
  - name: ca-cert
    mountPath: /ca-cert/
extraTLSSecrets: []
annotations: {}
imagePullSecrets: ""
image:
  repository: "pomerium/pomerium"
  tag: "v0.17.0"
  pullPolicy: "IfNotPresent"
metrics:
  enabled: true
  port: 9090
tracing:
  enabled: false
  provider: ""
  debug: false
  jaeger:
    collector_endpoint: ""
    agent_endpoint: ""
serviceMonitor:
  enabled: true
  namespace: ""
  labels:
    app: pomerium
rbac:
  create: true
redis:
  enabled: false
travisgroth wrote this answer on 2022-03-24

I'm able to reproduce this with a pretty minimal config. It looks like we're redoing the browser auth flow for every request, but they do actually succeed. I suspect something is up with either the storage or retrieval of the session credential cache on the cli side.

values:

authenticate:
  idp:
    provider: XXXXXX
    url: XXXXXX
    clientID: XXXXXX
    clientSecret: XXXXXX
    serviceAccount: XXXXXX
config:
  rootDomain: localhost.pomerium.io
  sharedSecret: XXXXXX
  cookieSecret: XXXXXX
  routes:
    - from: https://kubernetes.localhost.pomerium.io
      to: https://kubernetes.default.svc
      tls_skip_verify: true
      kubernetes_service_account_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      allow_websockets: true
      allowed_domains:
        - pomerium.com
        - gmail.com
ingress:
  enabled: false
ingressController:
  enabled: true

**edited to include api server route

sherifkayad wrote this answer on 2022-03-24

@travisgroth that's what I thought as well that it's a nasty bug in the CLI .. Web works fine as far as I have tested .. seems also to me that the user session is saved in the databroker cache.

Would you be working on a fix for this? Also how do you explain that it's happening with Pomerium 0.17.0 and not with the previous version?

travisgroth wrote this answer on 2022-03-24

@sherifkayad the behavior doesn't change with v0.16 of the cli so it may be some interaction with the authenticate and the callback to the cli. This does need more investigation.

Yes, we will be looking at fixing this. It doesn't appear to be working as intended.

sherifkayad wrote this answer on 2022-03-30

@travisgroth I assume this is still under investigation and hasn't made it to 0.17.1 yet, is that right?

calebdoxsey wrote this answer on 2022-03-31

So the reason this started happening is actually because of daylight savings time. We were parsing the expiry date:

time.Unix(claims.Expiry, 0)

And then doing an IsZero check. It turns out IsZero was true when we were in standard time, since 1970-01-01 is during standard time, but is no longer true, since we're now in daylight savings time (well a lot of us are anyway).

The result of this was we were immediately expiring the token, hence the constant re-opening.

More Details About Repo
Owner Name pomerium
Repo Name pomerium
Full Name pomerium/pomerium
Language Go
Created Date 2019-01-01
Updated Date 2022-09-20
Star Count 3229
Watcher Count 35
Fork Count 255
Issue Count 56

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
亲测事件USER_VOICE_VOLUME不触发,希望解决一下 0 2022-03-15 2022-09-14
UnhandledPromiseRejectionWarning: Error on Init API 3 2021-11-30 2022-04-21
how to set up webhook? 1 2021-11-26 2022-04-21
Request: Add API to fetch groups information 3 2021-12-01 2022-04-21
syntax error, unexpected ')' 3 2022-01-17 2022-09-14
Request: add graef.io to the filter 1 2022-05-09 2022-09-05
Trojan-Go WebSocket without host 4 2021-01-26 2022-08-01
How to add multiple selection date range and edit back (suggestion radio button) 0 2022-02-27 2022-09-02
picamera.exc.PiCameraRuntimeError: Timed out waiting for capture to end 1 2019-10-15 2022-09-21
Random failing test in benchmark 1 2020-05-05 2022-09-28
Please retire https://metacademy.org/ and put a pointer to this repo instead. 0 2021-04-03 2022-09-25
Failed to fetch pe-wasm 3 2019-04-01 2022-09-03
Create a label for discovery handlers issues/PRs and exclude it from stale bot 3 2022-04-05 2022-09-15
Silly question about commitizen and use for C/C++ repos 3 2020-10-01 2022-09-22
check for staged commits before questionnaire 4 2020-09-25 2022-09-20
pubsub cleanup delete all messages in topics 2 2020-01-23 2022-07-08
The variables index and extraDataSize not modified in the loop. 0 2022-09-21 2022-09-23
`path-excludes-pattern` built-in rule 2 2021-10-07 2022-09-05
Ingress without host and rewrite breaks when adding rule with host without rewrite 1 2021-04-28 2022-09-21
tips for more privacy and less telemetry 2 2021-08-23 2022-07-16
Cannot find Winston-telegram? 1 2021-12-29 2022-09-12
Failed to connect via standard <stream:stream message from Flutter Mobile App 2 2022-02-28 2022-09-23
Api Data does not match Dashboard 2 2021-05-08 2022-08-01
Ctrl+Backspace does not delete whole words in cmd.exe or pwsh.exe 0 2022-01-06 2022-07-22
`package.json` contain wrong values inside `extensionPack` for vscode 0 2022-03-18 2022-09-12
sync history across machines 14 2017-08-15 2022-03-21
502 Bad Gateway Error - Clair v4.0.6 1 2021-07-19 2022-09-12
The regular process to install lombok in Eclipse (https://projectlombok.org/setup/eclipse) fail . 4 2021-06-16 2022-08-16
get rid of NOTE in CRAN checks 1 2021-11-05 2022-09-16
Cleanup PrimaryKey error vs UniqueKey error vs DuplicateEntry 0 2021-08-08 2022-09-10
Add a disclaimer for non-endorsement 1 2022-07-13 2022-07-18
Saved variants page bug 1 2022-05-23 2022-08-10
Release opentelemetry-instrumentation-aws-lambda 1 2021-12-01 2022-09-24
Online update the HTTPS exceptions list 0 2022-01-19 2022-09-24
Cherry-pick c79c13cae61564d3a03831013ea24ff483ee3e82 6 2022-02-23 2022-07-16
Problem with java testfixtures and custom source path 0 2021-10-17 2022-09-09
There shouldn't be a link to concepts if concepts is disabled in the track config. 7 2021-10-20 2022-08-15
Exception message should say "*Un*resolved substitution ..." 1 2022-04-04 2022-09-21
Update RSS feed to support more than 10 episodes 3 2021-01-26 2022-09-19
Waker returns a Future, which breaks code in the "Async in depth" chapter 2 2022-02-02 2022-09-18
Reverse logic in LexborNode.decompose() 1 2021-09-02 2022-09-07
GRPC+ALTS client connection exception with Workload Identity 8 2022-04-05 2022-08-27
How to load the trained model on my own code 0 2021-07-27 2022-09-23
Invalid Field Names 1 2021-12-27 2022-01-15
cloudwatch-agent doesn't appear to be working with IMDSv2 w/ auth enabled 3 2021-07-26 2022-09-29
Problem with 1-click upgrade 3 2022-09-08 2022-09-18
Show unauthorized page when login callback fails or when claimCheck etc. fail? 4 2021-06-02 2022-08-31
facing problem when run the kitten code for ios with new app. please help, thnx 1 2020-10-21 2022-09-22
Function | decoder parameters 3 2021-08-25 2021-09-21
ARM64 can not bridge with TLS 4 2022-01-14 2022-09-14