By resetting the replay state the transaction replay can start again on a
new server. This allows the replay process work when a master server is
shutting down.
By delaying the replay for a second, we give the monitor a small chance to
adap to master failures. It'll also prevent rapid re-querying if multiple
transaction replays are supported.
A transaction that just completed will go through the start_trx_replay
function as from the client protocol's point of view the transaction is
still open. The debug assertion did not take this into account and would
fail if a successful commit was the last thing done on master that failed.
Also fixed the formatting.
When a server is stopping, it'll send an error to the client before
terminating the TCP connection. The code in readwritesplit would detect
this error and create a hangup event on the DCB. This would cause it to
appear as if the TCP connection was broken and the router would
immediately try to reconnect to the same server.
By ignoring the error and allowing the connection to die on its own, we
avoid immediately reconnecting and retrying any transactions on the
stopping server. This increases the chances that the monitor will see it
first and assign the server states correctly before the transaction replay
is attempted.
The assertion would hold true for a single worker but it can't be
guaranteed to be true on a multi-worker system where the statistics are
distributed across the workers.
Enabling the feature by default prevents the master connection from dying
during times when there are very little or no writes. Having a modest ping
interval of 300 seconds serves to minimize the amount of extra work that
both MaxScale and the server have to do while still keeping the
connections in good shape.
If the server where a query is being executed is shutting down,
readwritesplit should treat it as an error to make retrying of the query
possible.
By treating server shutdowns as network errors, the same code path that is
used for actual network errors can be taken. This removes the need for any
extra retrying logic for this particular case.
If the master succeeds in executing a session command but the slave fails,
the error message could help explain why it failed. At the moment this is
mainly relevant for inspection of test results.
The transaction replay could get mixed up with new queries if the client
managed to perform one while the delayed routing was taking place. A
proper way to solve this would be to cork the client DCB until the
transaction is fully replayed. As this change would be relatively more
complex compared to simply labeling queries that are being retried the
corking implementation is left for later when a more complete solution can
be designed.
This commit also adds some of the missing info logging for the transaction
replaying which makes analysis of failures easier.
The servers with a zero weight would be always used over ones that have a
weight. This means that the behavior was inverted and caused the
mxs2054_hybrid_cluster test to fail in 2.3.
Also fixed a typo in the deprecation message.
Commit a9e236497963251f8b4afa07484b88ad97e73a03 changed where the PS ID
for a binary protocol command is replaced with the internal form. This
caused prepared statements that are also session commands to be always
routed with the external ID.
As the external ID is almost always the master's ID, the aforementioned
bug resulted in odd side-effects and the true cause of these was only
revealed when the error message sent by the slave was included in the log
messages.
If a PS command is routed multiple times, the ID will not be reverted to
the external ID in the failure cases. This prevented prepared statements
from being re-routed correctly.
When the connection to the master is broken, the session is not configured
to use the read-only modes and the monitor can still connect to the
server, the connection will be closed and and error is sent to the
client. To leave some trace of this problem in the MaxScale logs, a
message should always be logged when a network error occurs.
As the router is the only one that knows what backends a particular
statement has been sent to, it is the responsibility of the router
to keep the session bookkeeping up to date. If it doesn't we will
know what statements a session has received (provided at least some
component in the routing chain has RCAP_TYPE_STMT_INPUT capability),
but not how long their processing took. Currently only readwritesplit
does that.
All queries are stored and not just COM_QUERY as that makes the
overall bookkeeping simpler; at clientReply() time we do not need to
know whether or not to bookkeep information, we can just do it.
When session information is queried for, we report as much information
we have available.
The causal_reads_timeout default value is too long when considering the
behavioral changes that MXS-2141 introduced. With a 10 second default
value, a result is returned to the client in a reasonable amount of time.
With causal_reads enabled, the query would return with an error if the
slave was not able to catch up to the master fast enough. By automatically
retrying the query on the master, we're guaranteed that a valid result is
always returned to the client.
The NULL values were not stored as NULL Avro values due to the fact that
the file format has no native NULL-ness for the basic types. To solve
this, all values must be stored as a union that contains the actual type
as well as the null type.
Unions were not implemented in the maxavro library but implementing means
simply recursing one level down.
If the query statement is wrongly treated as a table creation statement it
could cause a crash. To handle this, unknown SQL is now reported and the
processing is stopped early. This does not solve the root cause of the
problem but makes it possible to detect it in the future.
The admin files are now created with 640 permissions and automatically
created directories now properly set the permissions for the group as
well. All files and directories created by avrorouter and binlogrouter
also now correctly limit the read and write permissions only to the owner
and the group.
When a connection to a server is lost and the session command history is
disabled, the session will continue as long as at least one connection is
open. Previously the open connection calculation used the same code that
was used when a new session was created which only inspected the
configured server count instead of the actual open connection count.
Drop the requirement that GTID based replication is used for
the BinLog Galera "failover" mechanism. There is no reason for
that restriction; it works just as well with file+position based
replication.
The table creation was not detected as the function used to extract the
table name did not return the fully qualified names. Even if it did return
a fully qualified name, it wouldn't have been correctly processed.
When a read-only transaction fails due to a connection error, no message
would be logged. Also added an info level message for the case when a
backend connection would get closed before the session is in the correct
state and a debug assertion that the router session should never be closed
when the handleError method is called.
The binlogrouter uses buffers across worker threads which is no longer OK
in 2.3. The correct solution would be to store data in something other
than a GWBUF (e.g. std::vector) and protect the sharing with a mutex. The
current solution simply works around the assertions by using macros
instead of functions.
By biasing the values of all counter type scores to positive integers, the
server weights are always taken into use.
This fixes the case when weights were ignored until all score base values
were larger than zero (the mxs922_server test).
If the DCB is closed in handleError, it would be NULL in closeSession. To
only close the DCB in one place, the handleError can be reduced to writing
an error to the client and marking the failure as a fatal one.
