The state of the monitored servers is only persisted if the states of the
servers have changed. This removes the unnecessary disk IO caused by the
writing on the monitor journal.
Using timestamps to detect whether MaxScale was active or passive can
cause problems if multiple events happen at the same time. This can be
avoided by separating events into actively observed and passively observed
events. This clarifies the logic by removing the ambiguity of timestamps.
As the monitoring threads are separate from the worker threads, it is
prudent to use atomic operations to modify and read the state of the
MaxScale. This will impose an happens-before relation between MaxScale
being set into passive mode and events being classified as being passively
observed.
Moved mon_process_failover() from monitor.cc to mysql_mon.cc. Renamed
some functions and variables related to previous failover functionality
to avoid confusion.
The parameter handling for monitors can now be done in a consistent manner
by establishing a rule that the monitor owns the parameter object as long
as it is running. This will allow parameters to be added and removed
safely both from outside and inside monitors.
Currently this functionality is only used by mysqlmon to disable failover
after an attempt to perform a failover has failed.
The `failover` and `failover_timeout` parameters are now declared as a
part of the mysqlmon module. Changed the implementation of the failover
function so that the dependencies on the monitor struct can be removed or
moved into parameters.
Split the state change processing and failover handling into two separate
functions and added a call to the failover function into mysqlmon. This
prevents unintended behavior when failover is enabled for non-mysqlmon
monitors. The parameter itself still needs to be moved into mysqlmon.
Moved the failover documentation to the mysqlmon documentation as it is
specific to this monitor.
The failover command is simulated by executing a call to /usr/bin/echo
with all possible monitor parameters. This allows testing of the failover
mechanism without actually using the failover command.
The timestamp of the last change from passive to active is now
tracked. This, with the timestamps of the last master_down and master_up
events, allows detection of cases when MaxScale was failed over but the
failover was not done.
Currently, only a warning is logged if no new master has appeared within
90 seconds of a master_down event and MaxScale was set to active from
passive.
The last event and when the event was triggered is now shown for all
servers. The latest change from passive to active is also shown.
When an event occurs on a server, it is now stored so that the last event
for each server is known. This allows a state change to trigger an event
even if, at the time of the event, no action was taken.
This change is only cosmetic as no functionality is implemented.
All monitors now persist the state of the server in a monitor journal
file.
Moved the removal of stale journals into the core and removed them from
the monitor journal interface.
If a destroyed monitor is created again, it will be reused. This should
prevent excessive memory growth when the same monitor is created and
destroyed again.
The parameter names for monitors and servers now use a set of constant
names. This removes some of the errors caused by spelling mistakes when
the same parameter name is repeated in multiple places.
The service, filter and listener parameters should also be converted to
constants. This allows for a consistent user experience.
The relationships from servers to services and monitors and filters to
services were not implemented. Now each server lists the services and
monitors that use it and each filter lists the services that use the
filter.
This enables the creation of a server and linking of that server to
services and monitors in one atomic operation.
When a resource has a relation to another resource, it should be expressed
as a working link to the resource. By passing the hostname of the server
to the functions, we are able to generate working relation links.
The modules that implement a diagnostics entry point now return a JSON
type object. This removes the need to format data inside the modules.
The module implementations of these are not yet complete which means that
MaxScale will fail to compile.
Monitors can now be printed in JSON format. The REST API resource
`/monitors` accepts GET requests and returns a JSON representation of the
monitors as a response.
This number (defaults to 1) sets how many times mon_connect_to_db
will try to connect to a backend before returning an error. Every
connection attempt may take backend_connect_timeout seconds to
complete.
Also refactored code a bit. Renamed mon_connect_to_db to
mon_ping_or_connect_to_db, since it does not connect if the connection
is already alive.
Moved some typedefs to router.h and server.h, changed a few
constants to these enums. Renamed some types in config.h to
remove "Gateway".
There are still some functions in the public header which are
only used in core, but they seem to fit the theme of public functions
so were not moved.
This removes parts of the nearly identical code from all monitors.
The removal of monitor type specific event checking is done based on the
assumption that only the monitor that is monitoring the server can be the
cause for a state change. This removes the need to actually check that the
state change is relevant for each monitor and allows the event handling to
be moved into the core.
Previously, server status changes from MaxAdmin would be set immediately
as long as the server lock could be acquired. This meant that it might take
several seconds until the next monitor pass is executed. Usually, this was
fine but in some situations we would want the monitor to run immediately
after the change (MXS-740 and Galera). This patch changes the logic of
setting and clearing status bits to a delayed mode: changes are first applied
to a "status_pending"-variable, and only once the monitor runs will the
setting be applied. To reduce the delay, the monitor now has a flag
which is checked during sleep (between short 0.1s naps). If set, the
sleep is cut short.
If a server is not monitored, the status bits are set directly.
There is a small possibility of a race condition: If a monitor is stopped or
destroyed before the pending change is applied, the change is forgotten.
MXS-873 To prevent monitors and MaxAdmin from interfering with each other,
changes to the server status flags now happen under a lock. To avoid
interfering with monitor logic, the monitors now acquire locks to all
of their servers at the start of the monitor loop and release them
before sleeping.
Adding a server to multiple monitors is forbidden. This should be detected
and reported to the end user.
The information provided by the config_runtime system to the client isn't
as detailed as it could be. Some sort of an error message stack should be
added so that client facing interfaces could properly report the reason
for the failure. Currently the only way to detect the reason of the
failure is to parse the log files.
The config_runtime.h header contains functions that can be used to
manipulate the running configuration. Currently the header contains the
function to create, add, remove and destroy servers.
When a server is added to a monitor, an supplementary configuration file
is generated to persist this information. This will allow dynamic
modifications to server lists which will survive restarts and unexpected
downtime.
The monitor will only add new servers to its list of monitored
servers. This prevents duplicate entries in the list and makes it safe to
persist all used servers to the supplementary configuration file instead of only the ones that are not listed in the main configuration.
