Have server status updates applied during monitor loop

Previously, server status changes from MaxAdmin would be set immediately
as long as the server lock could be acquired. This meant that it might take
several seconds until the next monitor pass is executed. Usually, this was
fine but in some situations we would want the monitor to run immediately
after the change (MXS-740 and Galera). This patch changes the logic of
setting and clearing status bits to a delayed mode: changes are first applied
to a "status_pending"-variable, and only once the monitor runs will the
setting be applied. To reduce the delay, the monitor now has a flag
which is checked during sleep (between short 0.1s naps). If set, the
sleep is cut short.

If a server is not monitored, the status bits are set directly.

There is a small possibility of a race condition: If a monitor is stopped or
destroyed before the pending change is applied, the change is forgotten.

server/core/monitor.c

@@ -105,6 +105,7 @@ monitor_alloc(char *name, char *module)
     mon->interval = MONITOR_INTERVAL;
     mon->parameters = NULL;
     mon->created_online = false;
+    mon->server_pending_changes = false;
     spinlock_init(&mon->lock);
     spinlock_acquire(&monLock);
     mon->next = allMonitors;
@@ -1248,9 +1249,9 @@ void mon_log_state_change(MONITOR_SERVERS *ptr)
     MXS_FREE(next);
 }
 
-bool monitor_server_in_use(const SERVER *server)
+MONITOR* monitor_server_in_use(const SERVER *server)
 {
-    bool rval = false;
+    MONITOR *rval = NULL;
 
     spinlock_acquire(&monLock);
 
@@ -1262,7 +1263,7 @@ bool monitor_server_in_use(const SERVER *server)
         {
             if (db->server == server)
             {
-                rval = true;
+                rval = mon;
             }
         }
 
@@ -1439,7 +1440,7 @@ void mon_hangup_failed_servers(MONITOR *monitor)
     }
 }
 /**
-  * Acquire locks on all servers monitored my this monitor. There should
+  * Acquire locks on all servers monitored by this monitor. There should
   * only be max 1 monitor per server.
   * @param monitor The target monitor
   */
@@ -1452,6 +1453,11 @@ void lock_monitor_servers(MONITOR *monitor)
         ptr = ptr->next;
     }
 }
+/**
+  * Release locks on all servers monitored by this monitor. There should
+  * only be max 1 monitor per server.
+  * @param monitor The target monitor
+  */
 void release_monitor_servers(MONITOR *monitor)
 {
     MONITOR_SERVERS *ptr = monitor->databases;
@@ -1460,4 +1466,35 @@ void release_monitor_servers(MONITOR *monitor)
         spinlock_release(&ptr->server->lock);
         ptr = ptr->next;
     }
+}
+/**
+  * Sets the current status of all servers monitored by this monitor to
+  * the pending status. This should only be called at the beginning of
+  * a monitor loop, after the servers are locked.
+  * @param monitor The target monitor
+  */
+void servers_status_pending_to_current(MONITOR *monitor)
+{
+    MONITOR_SERVERS *ptr = monitor->databases;
+    while (ptr)
+    {
+        ptr->server->status = ptr->server->status_pending;
+        ptr = ptr->next;
+    }
+    monitor->server_pending_changes = false;
+}
+/**
+  *  Sets the pending status of all servers monitored by this monitor to
+  *  the current status. This should only be called at the end of
+  *  a monitor loop, before the servers are released.
+  *  @param monitor The target monitor
+  */
+void servers_status_current_to_pending(MONITOR *monitor)
+{
+    MONITOR_SERVERS *ptr = monitor->databases;
+    while (ptr)
+    {
+        ptr->server->status_pending = ptr->server->status;
+        ptr = ptr->next;
+    }
 }