1457 Commits

Author SHA1 Message Date
MassimilianoPinto
7454ef1d0b MXS-1064: Encryption options to "show service ..."
The binlog encryption options are reported by “show service
$service_name” of binlog router if the encryption is enabled.
2016-12-16 08:48:36 +01:00
Markus Makela
916320cf67 Clean up maxadmin help output
The detailed output of the new help command was a bit too densely packed
for some commands.

Added missing values for the `alter server` command help output.
2016-12-14 11:23:17 +02:00
MassimilianoPinto
7047cdf6c6 MXS-867: Key file Key Management for binlog encryption
Binlog server option ‘encryption_key_file=’ can now use the same key
file the MariaDB 10.1 server might have  in my.cnf:
‘file_key_management_filename=‘

Note: the file content must be in clear, no key encryption.
2016-12-13 15:51:43 +01:00
Markus Makela
7a04259fc0 MXS-756: Retry reads on slave failure
If a slave fails while a non-critical read is being executed, the read is
retried on a different server. This is controlled by the new
`retry_failed_reads` option.

Only selects done that are done outside of a transaction and with
autocommit enabled are retried.
2016-12-13 09:12:58 +02:00
MassimilianoPinto
162ae04d20 MXS-867: Binlog server encryption key has same format as MariaDB key file
Default algorithm is now AES_CBC.
The MariaDB 10.1 key file cannot be read as it might contain more than
1 rows.
2016-12-12 14:01:54 +01:00
Markus Makela
5ccdfe54bd Hack binlogrouter to work with resultset buffering
The binlogrouter needs to manipulate the protocol structures in order for
the resultset buffering to work correctly. If the state isn't manipulated
for COM_QUERY statements, the resultsets aren't buffered and will be
routed in separate buffers.
2016-12-12 11:56:41 +02:00
Markus Makela
106f482f45 Add result set buffering to MySQLBackend
The backend MySQL protocol module now supports a new routing capability
which allows result sets to be gathered into one buffer before they are
routed onward. This should not be used by modules that expect large
result sets as the result set is buffered in memory.

Adding a limit on how large of a result set could be buffered would allow
relatively safe use of this routing capability without compromising the
stability of the system.
2016-12-12 10:48:53 +02:00
Markus Makela
cb218804ef Detect double monitoring of servers
Adding a server to multiple monitors is forbidden. This should be detected
and reported to the end user.

The information provided by the config_runtime system to the client isn't
as detailed as it could be. Some sort of an error message stack should be
added so that client facing interfaces could properly report the reason
for the failure. Currently the only way to detect the reason of the
failure is to parse the log files.
2016-12-12 10:48:53 +02:00
Markus Makela
570e12942b MXS-1047: Fix batch insert execution
Doing batch inserts though readwritesplit would stall due to the fact that
pending session commands were stored instead of executed immediately.

Session command responses that weren't complete also discarded the partial
event instead of storing it for later use.
2016-12-08 15:09:36 +02:00
Markus Makela
585c4554c8 Fix binlogrouter regression
Non-replication events were implicitly ignored but this was removed in a
recent change. The code that wasn't previously used didn't break the
replication event handling loop.
2016-12-08 12:24:50 +02:00
Markus Makela
66feeb8fbc Fix rwsplit debug assertion
A maximum of zero slaves is an accepted value.
2016-12-08 12:09:06 +02:00
Markus Makela
d9ced5bbab Fix binlogrouter build failure
The binlogrouter used the wrong variable name.
2016-12-08 10:11:55 +02:00
MassimilianoPinto
9444ded6da Binlog encryption is now working with events larger than 16MBytes
Events larger than 16MBytes are now encrypted when being saved.

Some changes to binlog event details report and maxbinlogcheck supports
-H option for replication header display
2016-12-07 12:17:22 +01:00
MassimilianoPinto
76a2475c33 Merge branch 'develop' into binlog_server_waitdata_encryption 2016-12-06 14:57:06 +01:00
MassimilianoPinto
9731bca42e Fix for rotate event
Fix for rotate event: missing offset results in wrong filename detection
2016-12-06 14:54:09 +01:00
Markus Makela
4ae85dc754 Merge branch '2.0' into develop 2016-12-05 21:19:59 +02:00
MassimilianoPinto
b2e8a2d8c5 Develop merge
Develop merge:
this includes the new handling of events larger than 16MBytes
2016-12-05 16:38:48 +01:00
Markus Makela
f5eb4e21dc Add more comments and clean up code
The binlog event processing code is now better commented and is slightly
easier to read.
2016-12-05 17:08:59 +02:00
Markus Makela
2f082cb7fb Store large events in memory
Storing the large events in memory allows checksum calculations to be done
in one step. This also makes the encryption of events easier as they
require the complete event in memory.
2016-12-05 17:08:59 +02:00
Markus Makela
83109a6e9e Use protocol module to process packets
The backend protocol module can be requested to provide complete and
contiguous packets to the router module. This removes the need to process
the packets in binlogrouter.
2016-12-05 17:08:59 +02:00
Markus Makela
d97e3587e0 MXS-878: Don't log an error for 10.1 slave connections
The error message was not needed as we know that the query is always
executed by 10.1 slaves.
2016-12-05 13:18:20 +02:00
Markus Makela
360d7d53b8 Fix call command error message
The error messages referred to functions and printed duplicate
information.
2016-12-05 13:18:20 +02:00
Markus Makela
943aa48fb5 Make sure DCBs are OK before closing them
Added a check for the validity of the backend DCBs before they are
closed. This should guarantee that only valid DCBs are closed by
readwritesplit.

However, this is not the correct solution for the problem. The DCB should
not be in an invalid state in the first place and this fix just removes
the bad side effects of the double closing.

With the added logging in the readwritesplit error handler, more detailed
information should become available.
2016-12-04 06:35:35 +02:00
Markus Makela
9df3f154cf Detect duplicate objects
Creating a monitor or a listener twice is now detected and the proper
error is printed.
2016-12-03 11:31:18 +02:00
Markus Makela
be7a315614 Exit on bad maxadmin arguments
The errors were detected but the code proceeded to call various functions
with bad pointers. This led to a crash if a bad server name was given to
'show server'.
2016-12-02 19:08:54 +02:00
Markus Makela
6f7b572822 Only close DCBs we know are valid
If a DCB is passed to the error handler for which we cannot find the
corresponding backend reference, it should not be closed.

Added extra logging for situations where the backend reference can't be
found or it is in the wrong state.
2016-12-02 14:26:43 +02:00
Markus Makela
bc3d6f2ac7 Change functions to commands
The module commands operations are now listed as `commands` instead of
`functions`. The output was also formatted and an optional filtering was
added to the `list commands` call.
2016-12-02 10:49:59 +02:00
Johan Wikman
f5280fecfe Fix avrouter/CMakeLists.txt
Needed to be updated when binlog was renamed to binlogrouter.
2016-12-01 15:55:25 +02:00
Johan Wikman
aef6c7b099 Make directory name equal to the library name
* avro -> avrorouter
* binlog -> binlogrouter
2016-12-01 15:23:42 +02:00
Markus Makela
b3e31a3da2 Add creation and destruction of monitors to maxadmin
Maxadmin can now create and destroy monitors. The created monitors are not
started as they would be useless without added servers and configuration
parameters.
2016-11-30 12:43:50 +02:00
Markus Makela
4c4bd24a40 Allow module specific monitor parameters to be altered
Module specific parameters can now be altered at runtime. This allows both
the removal and addition of arbitrary monitor parameters.
2016-11-30 12:41:43 +02:00
MassimilianoPinto
f831295847 In case of error ftruncate is called with router->binlog_position
In case of error router->binlog_position is used with truncate and
error message instead of router->last_written
2016-11-30 10:05:00 +01:00
MassimilianoPinto
4ac6fe88c7 Merge branch 'develop' into binlog_server_waitdata_encryption 2016-11-30 09:38:00 +01:00
Markus Makela
8573df6ee7 Disable concurrect maxadmin access
The administrative operations in debugcmd.c now share a global lock.
2016-11-30 10:28:23 +02:00
Markus Makela
2efa862944 Remove listmanager code
The code prevented scaling by imposing global spinlocks for the DCBs and
SESSIONs. Removing this list means that a thread-local list must be taken
into use to replace it.
2016-11-30 10:28:23 +02:00
Markus Makela
ed280aa81b Use client DCB to route requests
Using internal DCBs for query routing wasn't needed as the client DCB
could be used. This could also be done by simply routing the query again
with routeQuery.
2016-11-30 10:28:23 +02:00
Markus Makela
8efdaa1ea6 Move fake events to a thread-specific queue
The fake poll events are now stored in thread specific queues. This
removes the need for the poll event queue.
2016-11-30 10:28:22 +02:00
MassimilianoPinto
5d5b2c9d6c Binlog file is checked at max scale startup if encryption is enabled
Binlog file is checked at max scale startup if encryption is enabled.

The check might fail while calculating next pos or verifying event type.
A message is reported
2016-11-30 09:01:42 +01:00
MassimilianoPinto
abd73f3700 blr_read_binlog checks replication header after decryption
blr_read_binlog can now check the  replication header after decryption,
for encrypted events.

Added a small fix for slave server requesting position of
START_ENCRYPTION_EVENT: new pos points to first encrypted event.
2016-11-29 17:57:32 +01:00
MassimilianoPinto
c7c68410be Merge branch 'develop' into binlog_server_waitdata_encryption 2016-11-29 17:38:04 +01:00
Markus Makela
4ff4e69592 Add stopping and starting of listeners to maxadmin
Listeners can now be stopped and started individually.
2016-11-29 15:04:19 +02:00
Markus Makela
075968e947 Allow avrorouter converstion to be started and stopped
The modulecmd functionality allows the avrorouter to easily control the
conversion process with one command. The conversion can now be started and
stopped by the user.

This also fixes a bug where the conversion would stop if there were no
binlog files present when the service was started.
2016-11-29 14:19:08 +02:00
ekorh475
cc54d80a8b Fix memory leaks in maxinfo (modified for 2.0 and develop-branch)
MXS-1009. This commit adds a gwbuf_free after maxinfo_execute() to
free a buffer with an sql-query after it has been processed. Also,
the parse tree in maxinfo_execute_query() is now freed. The tree_free-
function was renamed to maxinfo_tree_free, since it is now globally
available.

This commit has additional changes (in relation to the 1.4.4 branch)
to remove errors caused by differences in the html and sql-sides of
MaxInfo.
2016-11-29 14:11:12 +02:00
ekorh475
05303f5b7e Fix memory leaks in maxinfo (modified for develop-branch)
MXS-1009. This commit adds a gwbuf_free after maxinfo_execute() to
free a buffer with an sql-query after it has been processed. Also,
the parse tree in maxinfo_execute_query() is now freed. The tree_free-
function was renamed to maxinfo_tree_free, since it is now globally
available.

This commit has additional changes (in relation to the 1.4.4 branch)
to remove errors caused by differences in the html and sql-sides of
MaxInfo.
2016-11-29 13:08:20 +02:00
MassimilianoPinto
241fa67cfc AES_CBC is now supported for binlog encryption
AES_CBC can be used for binlog files encryption

The AES_CBC could leaves some not handled bytes in the buffer and those
need a special encoding (ECB and XOR)

This way the output buffer of the whole encoding with AES_CBC will have
same size as the input (AES_CTR does it without any other step)
2016-11-29 11:09:11 +01:00
Markus Makela
0bc68742a6 Do error handling in one big locked action
This is not the optimal way to do error handling but it should solve all
problems that could rise from the multi-threaded model of MaxScale.

By taking a lock at the start of handleError, we'll be able to modify the
dcb error handling flag in a thread-safe manner. This should prevent
double error handling for all DCBs.
2016-11-29 12:07:12 +02:00
Markus Makela
f8f400bdfd Only close the errored DCB if it is not in use
It was theoretically possible that a DCB was closed while it was still in
use.
2016-11-29 12:07:12 +02:00
Markus Makela
8dae8136aa Add more logging to solve MXS-956
If a DCB is about to be closed but is has already been closed, a message
is logged. This will allow us to figure out exactly where the DCB is
closed.
2016-11-29 12:07:12 +02:00
Markus Makela
9362954e82 Add newlines to avrorouter JSON output
JSON does not have a concept of streams and a common way to stream JSON is
to separate each JSON object with a newline. Adding a newline makes it
easier to parse as JSON values do not natively contain newlines.
2016-11-29 00:53:38 +02:00
MassimilianoPinto
440cf9ae7c Merge branch 'develop' into binlog_server_waitdata_encryption 2016-11-28 19:06:22 +01:00