FIX: Redo Sidekiq monitoring to restart stuck sidekiq processes (#30198)

This commit reimplements how we monitor Sidekiq processes that are
forked from the Unicorn master process. Prior to this change, we rely on
`Jobs::Heartbeat` to enqueue a `Jobs::RunHeartbeat` job every 3 minutes.
The `Jobs::RunHeartbeat` job then sets a Redis key with a timestamp. In
the Unicorn master process, we then fetch the timestamp that has been set
by the job from Redis every 30 minutes. If the timestamp has not been
updated for more than 30 minutes, we restart the Sidekiq process. The
fundamental flaw with this approach is that it fails to consider
deployments with multiple hosts and multiple Sidekiq processes. A
sidekiq process on a host may be in a bad state but the heartbeat check
will not restart the process because the `Jobs::RunHeartbeat` job is
still being executed by the working Sidekiq processes on other hosts.

In order to properly ensure that stuck Sidekiq processs are restarted,
we now rely on the [Sidekiq::ProcessSet](https://github.com/sidekiq/sidekiq/wiki/API#processes)
API that is supported by Sidekiq. The API provides us with "near real-time (updated every 5 sec)
info about the current set of Sidekiq processes running". The API
provides useful information like the hostname, pid and also when Sidekiq
last did its own heartbeat check. With that information, we can easily
determine if a Sidekiq process needs to be restarted from the Unicorn
master process.

lib/demon/base.rb

@@ -9,6 +9,20 @@ class Demon::Base
     @demons
   end
 
+  if Rails.env.test?
+    def self.set_demons(demons)
+      @demons = demons
+    end
+
+    def self.reset_demons
+      @demons = {}
+    end
+
+    def set_pid(pid)
+      @pid = pid
+    end
+  end
+
   def self.start(count = 1, verbose: false, logger: nil)
     @demons ||= {}
     count.times { |i| (@demons["#{prefix}_#{i}"] ||= new(i, verbose:, logger:)).start }
@@ -21,10 +35,7 @@ class Demon::Base
 
   def self.restart
     return unless @demons
-    @demons.values.each do |demon|
-      demon.stop
-      demon.start
-    end
+    @demons.values.each { |demon| demon.restart }
   end
 
   def self.ensure_running
@@ -75,6 +86,11 @@ class Demon::Base
     "HUP"
   end
 
+  def restart
+    stop
+    start
+  end
+
   def stop
     @started = false
 
@@ -105,7 +121,9 @@ class Demon::Base
       wait_for_stop.call
 
       if alive?
-        log("Process would not terminate cleanly, force quitting. pid: #{@pid} #{self.class}")
+        log(
+          "Process would not terminate cleanly, force quitting. pid: #{@pid} #{self.class}\n#{caller.join("\n")}",
+        )
         Process.kill("KILL", @pid)
       end