To work around the limitation in the session command handling and
multi-part results, all session commands are now treated as gathered
results. This allows session commands which return result sets to be used
with MaxScale.
This change should not cause problems with practical workloads as they
usually do not return massive resultsets for session commands.
The optimal way to handle the multi-part responses would be to integrate
it into the result completion tracking process. This would allow the
prepared statement IDs to be extracted while the command is being
processed.
By relying on the server to tell us that it is requesting the loading of a
local infile, we can remove one state from the state machine that governs
the loading of local files. It also removes the need to handle error and
success cases separately.
A side-effect of this change is that execution of multi-statement LOAD
DATA LOCAL INFILE no longer hangs. This is done by checking whether the
completion of one command initiates a new load.
The current code recursively checks the reply state and clones the
buffers. Neither of these are required nor should they be done but
refactoring the code is to be done in a separate commit.
Added two helper functions that are used to detect requests for local
infiles and to extract the total packet length from a non-contiguous
GWBUF.
The individual servers were missing a statistic that would give an
estimated query count. As there is no simple way to count queries for all
modules, counting the number of routed protocol packets is a suitable
substitute.
Session commands that span multiple packets are now allowed and will
work. However, if one is executed the session command history is disabled
as no interface for appending to session commands exists.
The backend protocol modules now also correctly track the current
command. This was a pre-requisite for large session commands as they
needed to be gathered into a single buffer and to do this the current
command had to be accurate.
Updated tests to expect success instead of failure for large prepared
statements.
Readwritesplit had redundant parameter values in the
`router_diagnostics`. All module parameters with their current values are
already displayed in the `parameters` member of the resource.
The router did not take large packets into account when determining
whether the server will respond. This caused the response counts to be off
by one for all large packets.
The creation of the EOF packet is not needed as the last packet of a
result set is always guaranteed to be of the correct type. This also
allows non-resultsets to be correctly processed as the internal packet
number will be at 0 when the last result arrives.
Cleaned up some of the function names and changed the signatures to be
better suited for their use-cases.
Use angle bracket includes, combine some of the more unwieldly
conditionals into functions, added more comments.
The resultset processing for MySQL requires some extra work as it lacks
the proper SERVER_MORE_RESULTS_EXIST flag in the last EOF packet. Instead,
the first EOF packet has the SERVER_PS_OUT_PARAMS flag which needs to be
interpreted as a SERVER_MORE_RESULTS_EXIST flag for the second EOF packet.
Also corrected the EOF packet handling to do the flag checks in the code
that deals with the EOF packets.
As the modutil_state parameter is now used for more than large packet
tracking, the correct solution is to store this state object in the
readwritesplit session instead of interpreting it to a boolean value.
Added the `transaction_replay_max_size` parameter that controls the
maximum size of a transaction that can be replayed. If the limit is
exceeded, the stored statements are released thus preventing the
transaction from being replayed.
This limitation prevents accidental misuse of the transaction replaying
system when autocommit is disabled. It also allows the user to control the
amount of memory that MaxScale will use.
The transaction retrying behavior is now configurable and documented. The
`transaction_replay` parameter implicitly enables the required
functionality in the router that it needs.
As the current query was added to the transaction log before it finished,
the m_current_query contained a duplicate of the latest transaction log
entry. To correctly log only successful transactions, the statement should
be added only after it has successfully completed. This change also
removed the unnecessary cloning that took place when the statement was
added to the log before it finished.
With the fixed transaction logging, the value of m_current_query can be
stashed for later retrying while the replay process is happening. If the
replay completes successfully and the checksums match, the interrupted
query is retried.
Also added a clarifying comment to can_retry_query to explain why a query
inside a transaction cannot be retried.
Added the initial implementation of transaction replay. Transactions are
only replayed if the master fails when no statement is being executed.
The validity of the replayed transaction is done by verifying that the
checksums of the returned results are equal.
Added a close function into the Trx class to make resetting its state
easier. Also changed the return type of the pop_stmt to GWBUF* as the
places where it is used expect a raw GWBUF pointer.
The queries that make up the transaction are now stored in the router
session while the transaction is in progress. For the time being, the
queries are only used to log extra information about the transaction
contents.
Readwritesplit now calculates checksums for all successful and failed
transactions. This checksum is not of any practical use until the
transaction replaying is implemented.
Fixed string truncation warnings by reducing max parameter lengths by one
where applicable. The binlogrouter filename lengths are slightly different
so using memcpy to work around the warnings is an adequate "solution"
until the root of the problem is solved.
Removed unnecessary CMake policy settings from qc_sqlite. Adding a
self-dependency on the source file of an external project has no effect
and only caused warnings to be logged.
Large session commands weren't properly handled which caused the router to
think that the trailing end of a multi-packet query was actually a new
query.
This cannot be confidently solved in 2.2 which is why the router session
is now closed the moment a large session command is noticed.
Only commands that can contain an SQL statements should be stored for
retrying (COM_QUERY and COM_EXECUTE). Other commands are either session
commands or do not work with query retrying.
Worker is now the base class of all workers. It has a message
queue and can be run in a thread of its own, or in the calling
thread. Worker can not be used as such, but a concrete worker
class must be derived from it. Currently there is only one
concrete class RoutingWorker.
There is some overlapping in functionality between Worker and
RoutingWorker, as there is e.g. a need for broadcasting a
message to all routing workers, but not to other workers.
Currently other workers can not be created as the array for
holding the pointers to the workers is exactly as large as
there will be RoutingWorkers. That will be changed so that
the maximum number of threads is hardwired to some ridiculous
value such as 128. That's the first step in the path towards
a situation where the number of worker threads can be changed
at runtime.
The commands needs to be handled separately from the rest of the result
types.
Added a test case that reproduces the problem and verifies that the change
in code fixes it.
When a LOAD DATA LOCAL INFILE finishes, the client sends an empty
packet. The second case when the client sends an empty packet when the
previous packet was exactly 0xffffff bytes long. These two packets were
confused which caused the internal state to temporarily flip from inactive
to ending and back to inactive.
The aforementioned flip-flopping didn't have any practical differences but
it was caught by a debug assertion.
Only servers that qualify to be connected should be considered as
candidate servers. This triggered a debug assertion when a slave server
failed to execute a session command but it was chosen as a candidate
server later on.
