500 Errors when redirected from AuthN when Redis is enabled

This issue has been created since 2022-03-23.

What happened?

I enabled redis, then went through an authentication flow and was greeted with 500 errors when successfully auth'ed
When redis is disabled, authentication flow succeeds.

What did you expect to happen?

I should be able to enable redis and successfully navigate to my page after authentication.

How'd it happen?

Enabled redis

databroker:
  storage:
    connectionString: rediss://pomerium-redis-master.pomerium.svc.cluster.local

redis:
  enabled: true
  auth:
    enabled: false
  usePassword: false
  generateTLS: false
  tls:
    certificateSecret: pomerium-redis-tls

What's your environment like?

Pomerium version (retrieve with pomerium --version): Current Helm chart - pomerium-30.1.1
Server Operating System/Architecture/Cloud: EKS K8s 1.21 with Google IDP

What's your config.yaml?

autocert: false
dns_lookup_family: V4_ONLY
address: :443
grpc_address: :443
certificate_authority_file: "/pomerium/ca/ca.crt"
certificates:
authenticate_service_url: https://authenticate.tools.dev.sw.io
authorize_service_url: https://pomerium-authorize.pomerium.svc.cluster.local
databroker_service_url: https://pomerium-databroker.pomerium.svc.cluster.local
idp_provider: google
idp_scopes:
idp_provider_url:
idp_client_id: ${client_id}
idp_client_secret: ${client_secret}
routes:

What did you see in the logs?

pomerium-proxy-dcf76cf56-bkm7r pomerium {"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"verify.tools.dev.sw.io","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36","referer":"","forwarded-for":"10.0.47.110","request-id":"d7c2ae26-e39c-4fac-b535-4723a8f94467","duration":9998.505467,"size":0,"response-code":500,"response-code-details":"ext_authz_error","time":"2022-03-23T21:47:05Z","message":"http-request"}
pomerium-redis-master-0 redis 1:M 23 Mar 2022 21:47:06.476 * Replica pomerium-redis-replicas-1.:6379 asks for synchronization
pomerium-redis-master-0 redis 1:M 23 Mar 2022 21:47:06.476 * Full resync requested by replica pomerium-redis-replicas-1.:6379
+ pomerium-redis-replicas-1 › redis
pomerium-redis-master-0 redis 1:M 23 Mar 2022 21:47:06.476 * Starting BGSAVE for SYNC with target: disk
pomerium-redis-master-0 redis 1:M 23 Mar 2022 21:47:06.476 * Background saving started by pid 84
pomerium-redis-master-0 redis 84:C 23 Mar 2022 21:47:06.481 * DB saved on disk
pomerium-redis-master-0 redis 84:C 23 Mar 2022 21:47:06.482 * RDB: 0 MB of memory used by copy-on-write
pomerium-redis-master-0 redis 1:M 23 Mar 2022 21:47:06.510 * Background saving terminated with success
pomerium-redis-master-0 redis 1:M 23 Mar 2022 21:47:06.510 * Synchronization with replica pomerium-redis-replicas-1.:6379 succeeded
pomerium-redis-replicas-1 redis 1:C 23 Mar 2022 21:47:05.561 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
pomerium-redis-replicas-1 redis 1:C 23 Mar 2022 21:47:05.561 # Redis version=6.2.6, bits=64, commit=00000000, modified=0, pid=1, just started
pomerium-redis-replicas-1 redis 1:C 23 Mar 2022 21:47:05.561 # Configuration loaded
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.562 * monotonic clock: POSIX clock_gettime
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.562 * Running mode=standalone, port=6379.
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.562 # Server initialized
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.563 * Reading RDB preamble from AOF file...
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.563 * Loading RDB produced by version 6.2.6
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.563 * RDB age 2278 seconds
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.563 * RDB memory usage when created 1.86 Mb
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.563 * RDB has an AOF tail
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.564 # Done loading RDB, keys loaded: 9, keys expired: 0.
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.564 * Reading the remaining AOF tail...
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.564 * DB loaded from append only file: 0.002 seconds
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:05.564 * Ready to accept connections
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.466 * Connecting to MASTER pomerium-redis-master-0.pomerium-redis-headless.pomerium.svc.cluster.local:6379
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.467 * MASTER <-> REPLICA sync started
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.472 * Non blocking connect for SYNC fired the event.
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.474 * Master replied to PING, replication can continue...
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.475 * Partial resynchronization not possible (no cached master)
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.477 * Full resync from master: 429e7901d8efedb1cfbc6a83b1719dccbe8920a2:14
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.510 * MASTER <-> REPLICA sync: receiving 47663 bytes from master to disk
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.511 * MASTER <-> REPLICA sync: Flushing old data
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.513 * MASTER <-> REPLICA sync: Loading DB in memory
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.515 * Loading RDB produced by version 6.2.6
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.515 * RDB age 0 seconds
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.515 * RDB memory usage when created 1.95 Mb
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.516 # Done loading RDB, keys loaded: 11, keys expired: 0.
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.516 * MASTER <-> REPLICA sync: Finished with success
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.516 * Background append only file rewriting started by pid 20
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.541 * AOF rewrite child asks to stop sending diffs.
pomerium-redis-replicas-1 redis 20:C 23 Mar 2022 21:47:06.541 * Parent agreed to stop sending diffs. Finalizing AOF...
pomerium-redis-replicas-1 redis 20:C 23 Mar 2022 21:47:06.541 * Concatenating 0.00 MB of AOF diff received from parent.
pomerium-redis-replicas-1 redis 20:C 23 Mar 2022 21:47:06.541 * SYNC append only file rewrite performed
pomerium-redis-replicas-1 redis 20:C 23 Mar 2022 21:47:06.542 * AOF rewrite: 0 MB of memory used by copy-on-write
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.568 * Background AOF rewrite terminated with success
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.568 * Residual parent diff successfully flushed to the rewritten AOF (0.00 MB)
pomerium-redis-replicas-1 redis 1:S 23 Mar 2022 21:47:06.568 * Background AOF rewrite finished successfully
pomerium-proxy-dcf76cf56-bkm7r pomerium {"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","time":"2022-03-23T21:47:07Z","message":"initial sync"}
pomerium-databroker-649596d7bf-rcvxd pomerium {"level":"info","type":"type.googleapis.com/pomerium.config.Config","time":"2022-03-23T21:47:07Z","message":"sync latest"}
pomerium-proxy-dcf76cf56-bkm7r pomerium {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-23T21:47:07Z","message":"error during initial sync"}
pomerium-proxy-dcf76cf56-bkm7r pomerium {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-23T21:47:07Z","message":"sync"}
pomerium-databroker-649596d7bf-rcvxd pomerium {"level":"info","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","time":"2022-03-23T21:47:07Z","message":"initial sync"}
pomerium-databroker-649596d7bf-rcvxd pomerium {"level":"info","type":"type.googleapis.com/pomerium.config.Config","time":"2022-03-23T21:47:07Z","message":"sync latest"}
pomerium-databroker-649596d7bf-rcvxd pomerium {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-23T21:47:07Z","message":"error during initial sync"}
pomerium-databroker-649596d7bf-rcvxd pomerium {"level":"error","syncer_id":"databroker","syncer_type":"type.googleapis.com/pomerium.config.Config","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-23T21:47:07Z","message":"sync"}

Additional context

This could definitely be an issue with my shared secret, but in that case more documentation on how to flush the db to reset the shared secret would be a welcome enhancement. There was a link to https://redis.io/commands/flushdb/ provided by @travisgroth in the original post, however when I tried to execute this command on the redis-master pod I got connection refused. At the same time if this happens frequently I would think pomerium should be able to handle this edge case.

Original support ticket: https://discuss.pomerium.com/t/external-domain-configuration/92

sarasensible wrote this answer on 2022-03-25

Now with a reinstall disabling TLS in redis specifically for the moment I am seeing 404s immediately with no redirect.

Logs:

pomerium-authorize-56456b5777-grcbq pomerium {"level":"error","syncer_id":"authorize","syncer_type":"","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-25T14:44:41Z","message":"error during initial sync"}
pomerium-databroker-6489c67c4d-7mhdm pomerium {"level":"info","type":"","time":"2022-03-25T14:44:41Z","message":"sync latest"}
pomerium-authorize-56456b5777-grcbq pomerium {"level":"error","syncer_id":"authorize","syncer_type":"","error":"rpc error: code = Unknown desc = cryptutil: decryption failed (mismatched keys?): chacha20poly1305: message authentication failed","time":"2022-03-25T14:44:41Z","message":"sync"}
pomerium-proxy-6884b678d5-hz267 pomerium {"level":"info","service":"envoy","upstream-cluster":"","method":"GET","authority":"metrics.ops.dev.sw.io","path":"/","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36","referer":"","forwarded-for":"10.0.25.163","request-id":"67d5618e-ce9b-4aaf-8f4e-361cb75973cf","duration":0.244955,"size":0,"response-code":404,"response-code-details":"route_not_found","time":"2022-03-25T14:44:52Z","message":"http-request"}
sarasensible wrote this answer on 2022-03-25

I got this working without TLS using the following configuration

databroker:
  existingTLSSecret: pomerium-tls
  storage:
    connectionString: redis://pomerium-redis-master.tools.svc.cluster.local
    type: redis
    tlsSkipVerify: true
#    clientTLS:
#      existingSecretName: pomerium-redis-tls
#      existingCASecretKey: ca.crt

authorize:
  existingTLSSecret: pomerium-tls

redis:
  enabled: true
  auth:
    enabled: false
  usePassword: false
  generateTLS: false
  tls:
    enabled: false

I managed to execute the FLUSHALL command after reading this post
https://stackoverflow.com/questions/22815364/flushall-and-flushdb-commands-on-redis-return-unk-command/71619197#71619197

I doc'ed a workaround in that post for the fact that the Helm chart disables the flushall and flushdb commands by default.

I'm going to try again with TLS and see if I can get that working by using flushall, if I can I will call this a success and contribute a PR for the docs.

sarasensible wrote this answer on 2022-03-25

I see that the issue here is the redis-cli is not able to connect over TLS, so it requires some tomfoolery perhaps with stunnel to execute the FLUSHALL command. If there is a simpler way to do this please let me know, otherwise I'm going to mount a sidecar with stunnel so I can execute this command.

Update - this is all wrong, poor googling on my part, you can most definitely connect to redis using the cli over TLS with options -> see https://redis.io/docs/manual/security/encryption/

sarasensible wrote this answer on 2022-03-25

Ok figured out how to do this with TLS enabled and FLUSHALL and FLUSHDB enabled.

Working config (note that this has the redis-master installed in the tools namespace):

databroker:
  existingTLSSecret: pomerium-tls
  storage:
    connectionString: rediss://pomerium-redis-master.tools.svc.cluster.local
    type: redis
    tlsSkipVerify: true
    clientTLS:
      existingSecretName: pomerium-redis-tls
      existingCASecretKey: ca.crt

authorize:
  existingTLSSecret: pomerium-tls

redis:
  enabled: true
  master:
    disableCommands: [ ]
  auth:
    enabled: false
  usePassword: false
  generateTLS: false
  tls:
    enabled: true
    certificateSecret: pomerium-redis-tls
kubectl exec -it pomerium-redis-master-0 -- redis-cli --tls --cert /opt/bitnami/redis/certs/tls.crt --key /opt/bitnami/redis/certs/tls.key --cacert /opt/bitnami/redis/certs/ca.crt FLUSHALL ASYNC

And voila, everything works. I still believe this should be handled on reinstall on the pomerium side, but this is acceptable as a workaround. I'll submit a PR to the docs to include this nugget when enabling redis.

More Details About Repo
Owner Name pomerium
Repo Name pomerium
Full Name pomerium/pomerium
Language Go
Created Date 2019-01-01
Updated Date 2022-09-20
Star Count 3229
Watcher Count 35
Fork Count 255
Issue Count 56

YOU MAY BE INTERESTED

Issue Title Created Date Comment Count Updated Date
Migrate from CRA to vitejs 1 2022-08-04 2022-09-03
Migrate to tailwind 3.x 0 2022-08-04 2022-09-03
运行报错:AttributeError: 'NoneType' object has no attribute 'group' 2 2020-09-13 2022-07-15
Response is always 202 Except for login API 1 2022-08-01 2022-09-18
Новое неизвестное поле от API 1 2021-07-29 2022-06-28
Threshold to say that it's the same person 3 2019-09-19 2022-08-31
Implement alternate build with tools-deps / deps.edn 0 2022-02-10 2022-08-27
Segmentation Fault error on posti_preparerecordpoints 12 2021-03-04 2022-09-05
Connection timeout but reconnecting fails 4 2021-11-28 2022-09-30
Unclear interpretation of Scan Results 1 2021-12-16 2022-09-19
For devs and contributors audience - How to handle an add new PHP modules in Docker images 21 2021-07-07 2022-09-18
Could not find a declaration file for module '@websanova/vue-auth/dist/drivers/auth/bearer.esm.js'. 2 2021-03-05 2022-09-09
prometheus-blackbox-exporter issue for team services 1 2022-05-10 2022-09-06
Uninstall deprecated helm release 3 2022-04-13 2022-09-17
The checkPolicies fails on gatekeeper install/upgrade 2 2022-05-16 2022-09-22
How do I get response header stream-media-id 1 2022-05-19 2022-09-22
[FCEUX 2.6.5 Update] Add More Sound Channels On/Off 1 2022-04-02 2022-05-01
Update DarshanGowda0 branch to cloud_firestore: ^3.1.6 and correctly point it in Flutter Pub Dev Doc 1 2022-01-10 2022-09-28
Removed package-lock.json + docs 1 2021-01-20 2022-02-04
Heatmap Chart Tooltip 2 2019-08-05 2022-09-13
amsmath(?): Addition of `\mathup` command 8 2022-02-01 2022-09-13
Tabulator custom formatter throws "Uncaught TypeError: setting getter-only property" when data class has getter property 4 2021-06-11 2022-04-11
"You already have a private chat portal with me at" points to a room I cannot join 0 2022-07-12 2022-09-13
🚀 Feature Request: Dedicated widget for Sale badge on the product page (woocommerce) 0 2022-07-23 2022-09-26
Private Git Repo instead of npm / private npm 4 2020-08-04 2022-09-11
[EAK-244] Nested nodes of components converted into multifiels with @Multiple, are duplicated 0 2021-11-16 2022-09-13
Procedure update service fails with ConstraintViolationException on receive of HL7 order messages containing multiple Scheduled Procedure Steps 0 2021-09-03 2022-07-22
Upgrade process failed: route error 3 2022-01-23 2022-09-20
"Could not check configuration values between execution and consensus client" error="method not found" 3 2022-08-22 2022-09-26
Change charting library 6 2021-06-22 2022-09-25
189CloudPC WebDAV 无法上传百兆文件 3 2022-03-27 2022-09-17
希望可以添加收藏功能与直链短链接化的功能 2 2022-03-27 2022-09-19
Fix configuration missing global section 0 2021-07-19 2022-07-31
segfault when inserting data into compressed Decimal, String, FixedString and Array columns 0 2021-12-18 2022-09-21
Can't install module on Magisk Canary 1 2022-01-06 2022-09-23
How to upgrade influxdb from 4.0.11 to 4.0.12 ? 3 2022-04-26 2022-09-23
SendGrid not working on Netlify Functions 3 2021-07-21 2022-05-03
Change program language for Linux 0 2022-01-18 2022-09-17
Size of Shared Memory for a particular Ring Size 1 2022-06-03 2022-09-28
linux-v4l2: Change search strategy for v4l2loopback devices 0 2021-08-25 2022-09-08
Support for `std::complex<__half>`? 3 2020-12-18 2022-09-29
Computing camera trajectories 5 2021-11-10 2022-09-21
Add `-Z unpretty` flags for the AST and the THIR 2 2021-02-19 2022-09-21
[PAGE ISSUE]: 'Customizing static analysis' has a broken code excerpt 0 2022-03-13 2022-09-23
Review Ceph storage docs 3 2022-07-12 2022-08-26
Z-Wave JS can't connect to host after update to core 2022.6.6 and OS 8.2 29 2022-06-15 2022-09-24
Not available in Graalvm native 0 2022-07-29 2022-09-20
Billing Provider column on operator accounts not populated correctly 0 2021-09-17 2022-09-13
CSS Fixes: DocSearchWidget and Pinned Items 0 2021-09-17 2022-08-13
Clarification on disableHostCheck and transpileDependencies in vue.config.js 1 2019-09-07 2022-09-29