Commit Graph

43 Commits

Author SHA1 Message Date
5406e24acb FEATURE: Introduce pg_force_readonly_mode GlobalSetting (#19612)
This allows the entire cluster to be forced into pg readonly mode. Equivalent to running `Discourse.enable_pg_force_readonly_mode` on the console.
2023-01-19 13:59:11 +00:00
7c77cc6a58 DEV: Apply syntax_tree formatting to config/* 2023-01-09 11:13:29 +00:00
008b700a3f DEV: Upgrade to Rails 7
This patch upgrades Rails to version 7.0.2.4.
2022-04-28 11:51:03 +02:00
5289fc7886 FIX: Improve failover for multisite clusters (#11150)
- Bump rails_failover for new per-backend callback feature
- If the master backend fails over, make all sites readonly. And vice-versa for fallback
- If a single backend fails over, make that individual site readonly. And vice-versa for fallback
- When a single backend fails, also check connection to the master backend
2020-11-11 10:27:24 +00:00
ce686a008f DEV: Refresh site settings on Redis fallback (#10700)
This ensures that all app instances have up-to-date data, even if they missed a message-bus update during the failover
2020-09-18 11:44:25 +01:00
d83d9ec408 DEV: Clear all DistributedCache on Redis fallback. 2020-09-18 11:31:49 +08:00
23778f4bfd DEV: Avoid logging Redis cannot connect error during failover.
Reduce amount of noise in our logs.
2020-07-21 14:53:46 +08:00
df66a559c9 DEV: Pause Sidekiq before readonly mode during PG failover. 2020-07-21 13:51:37 +08:00
031a6616a3 DEV: Allow writes to PG even if Redis is readonly mode.
Previously, I thought it was better to drop the site into reading mode
when Redis has failed over to the replica but it created more errors
while Redis is in readonly mode since ActiveRecord would prevent us from
writing to PG even though PG is up.
2020-07-21 12:34:07 +08:00
383537a2fa DEV: Force ActiveRecord reading role on Redis failover take 2. 2020-07-20 16:11:20 +08:00
5adf2ccaca DEV: Force ActiveRecord reading role on Redis failover. 2020-07-20 15:10:53 +08:00
54d002f7db DEV: Fix error introduced in d5c56a846. 2020-07-13 10:13:37 +08:00
d5c56a846a DEV: Only failover the entire cluster when the default db goes down. 2020-07-09 11:49:03 +08:00
fd38c2fac3 FIX: Force ActiveRecord reading role if Redis is down take 2.
follow-up f03c7a1ba1cf
2020-07-09 11:14:19 +08:00
f03c7a1ba1 FIX: Force ActiveRecord reading role if Redis is down. 2020-07-09 11:13:02 +08:00
2c4c953bf8 DEV: Avoid logging errors on bad Redis connection during PG failover. 2020-06-29 11:54:55 +08:00
c611f3703c DEV: Don't use logster when logging in Redis failover. 2020-06-16 11:53:52 +08:00
b08a0d15c4 DEV: Fix undefined method due to rails_failover. 2020-06-16 11:03:57 +08:00
092ae858af DEV: Bump rails_failover.
Avoid configuring AR stuff if `replica_host` and `replica_port` hasn't
been provided.
2020-06-16 10:51:21 +08:00
402b80f306 DEV: Make rails_failover compatible with SKIP_DB_AND_REDIS env. 2020-06-15 16:23:24 +08:00
e0d798c06c DEV: Fix undefined method. 2020-06-15 16:04:41 +08:00
e0fdf41537 DEV: Don't configure rails_failover is db and redis is skipped. 2020-06-15 15:56:57 +08:00
58e52c0e4f DEV: Use rails_failover gem for ActiveRecord and Redis failover handling 2020-06-15 15:47:07 +08:00
0ff86b00cb DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
78b5ab746c DEV: No longer need to clear anon cache when toggling readonly mode. 2020-06-12 09:58:17 +08:00
dc4071dfef DEV: Use Rails.logger instead of logster for rails_failover callbacks
`Discourse.warn_exception` logs to logger by default but it means we
lose all the backtrace when the logs are written to the log file.
2020-06-11 17:24:32 +08:00
1411b095eb DEV: Rescue errors when runngin AR failover/fallback callbacks. 2020-06-11 13:02:42 +08:00
34ee1f2e71 DEV: Fix undefined method in rails_failover initializer. 2020-06-11 12:30:01 +08:00
cbb4ea1ea8 DEV: Don't blow up request redis can't be reached during failover. 2020-06-11 11:41:06 +08:00
a3dfd553a1 Revert "Bump redis to 4.2.0."
This reverts commit 98bc28cea259b823552e1b953506bcf2f76347a9.
2020-06-10 14:52:05 +08:00
98bc28cea2 Bump redis to 4.2.0. 2020-06-10 14:28:56 +08:00
2ce829cc01 DEV: Allow multisite apps to boot with reading connection handler. 2020-06-10 14:18:29 +08:00
a1c13eb3c6 DEV: Redis failover should only clear redis recently readonly. 2020-06-09 16:36:31 +08:00
f6628e4f43 DEV: Disable messageBus keepalive when Redis fails over. 2020-06-08 12:33:08 +08:00
54f79ea3ba DEV: Pausing Sidekiq is multisite aware. 2020-06-04 15:46:30 +08:00
8e1681d356 Bump rails_failover. 2020-06-04 15:22:35 +08:00
439db7ca1e DEV: Add REDIS_RAILS_FAILOVER env to test our new redis failover. 2020-06-02 17:24:14 +08:00
ade60b0cbc DEV: Enable readonly mode for all multisite sites when PG goes down.
The risk here is that the database for one site goes down in the multisite setup and we drop everything to readonly mode. However, I discussed this with Sam and we agree that one database having problem is very rare. Most of the time, it is the entire DB cluster that goes down.
2020-06-02 11:32:07 +08:00
326d6d5b0f DEV: Pause Sidekiq when forcing pg readonly mode. 2020-06-02 09:20:03 +08:00
32735be5bd DEV: Fix publish to message_bus when forcing pg readonly. 2020-06-01 13:26:12 +08:00
2c3ff3e524 DEV: Missing readonly mode banner when forcing PG readonly. 2020-06-01 12:25:27 +08:00
b0b37bf5a3 DEV: Add force rails_over switch via Redis. 2020-06-01 11:23:58 +08:00
8c86a109bb DEV: Add ENV flag to test out ActiveRecord::Failover. 2020-05-28 16:24:22 +08:00