hcq/MaxScale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

All changes 2.0.0...develop

Browse Source 
973b983 Merge branch 'release-2.0.0' into develop
255dd23 Make spinlock functions take const argument
6e23bab Fix bitmask reallocation
338c189 Rename and clean up slavelag filter
3ea8f28 Fix possible NULL pointer dereference
bfe6738 MXS-830: Add module information to logged messages
1fad962 Fix strncat usage
d38997a Adjust log throttling policy
0be4e4b Add hashtable_item_strcasecmp
726100e Take hashtable convenience functions into use
5e7744a Fix typo in maxadmin.md
c5778c8 Merge branch 'release-2.0.0' into develop
b5762af Move from tmpnam to mkstemp
d6f2c71 Add convenience functions to hashtable
359058a MXS-825: Add support for --execdir
636347c Enable runtime reconfiguration of log throttling
ef9fba9 Improve log throttling documentation
aef917a Implement log throttling
e3a5349 Remove shardrouter.c
8051e80 Remove custom qc_sqlite allocation functions
fd34d60 Initial implementation of the learning firewall
a8752a8 Removed "filestem option" from example
1ef2519 Removed "filestem option" from example
0815cc8 Cleanup spinlock.h
ab4dc99 Clean up hashtable.h
ef2c078 Add prototypes for hashtable copy and free functions
fb5cfaf Add 'log_throttling' configuration entry
300d823 Add proper prototypes for hashtable hash and cmp functions
1c649aa qc_mysqlembedded: Include skygw_...h without path.
d276160 Add missing RPM scripts
e70e644 Fix HTTPAuth installation
1b2b389 Combine utils into the server directory
3ff9913 Add missing utils headers to devel package
407efb2 Fix minor packaging problems
99aa6ad Split MaxScale into core, experimental and devel packages
1290386 Merge branch 'develop' of ssh://github.com/mariadb-corporation/maxscale-new into develop
e59f148 Make scripts POSIX sh compatible
7319266 Fixed SHOW SLAVE STATUS in bonlog router
f8d760a Update Binlogrouter.md
0a904ed Update Replication-Proxy-Binlog-Router-Tutorial.md
75d4202 Update Replication-Proxy-Binlog-Router-Tutorial.md
b8651fc Add missing newline in listmanager.h
c7ad047 Add note about user data caches to release notes
70ccc2b Merge branch 'release-2.0.0' into develop
575d1b6 Mistake - dummy session needs list markers set.
8364508 Merge branch 'develop' into binlog_server_semisync
868b902 Update MaxScale limitations
2c8b327 Store listener caches in separate directories
6e183ec Create unique user data caches for each listeners
f643685 Don't free orphaned tee filter sessions
4179afa Allow binlogrouter to be used without a listener
7ad79af Add function for freeing a listener
677a0a2 Move authentication data from services to listeners
4f12af7 Merge remote-tracking branch 'origin/MXS-677' into develop
1419b81 Semi-Sync support to binlog server: code review updtate
0ea0f01 Semi-Sync support to binlog server: added missing routine
4aad909 Semi-Sync support to binlog server
b824e1e Add authenticator support to httpd.c
705a688 Change tabs to spaces
d0c419e Change method of adding list fields to e.g. DCB
25504fc Document the changed routing priority of hints
41666d1 Remove use_ssl_if_enabled global option
a3584e9 Make routing hints have highest priority
34a1d24 Updated document with new binlog router option
01eedc5 Updated documentation with SSL usage
8a4c0f6 Update Replication-Proxy-Binlog-Router-Tutorial.md
4e374aa Update Replication-Proxy-Binlog-Router-Tutorial.md
f3f3c57 Update Replication-Proxy-Binlog-Router-Tutorial.md
617b79f Binlog Server: error messages typo fix
fa8dfae Binlog Server: error messages review
1b8819c Fix freeing of schemarouter session memory
07f49e1 MXS-788: new code review fix
1fd3b09 MXS-788: show services now displays SSL info
6ca2584 MXS-788 code review fix
ae6a7d0 MXS-788 code review
43d3474 Master server SSL connection
90b2377 Use correct variable in listmanager pre-allocation
9a5b238 Fix listmanager pre-allocation
9c78625 Fix a memory leak when backend authentication fails
e59a966 Fix hang in list_find_free
ff30223 Fix freeing of shared data in schemarouter
fc8f9d3 Add missing include in luafilter
ecf7f53 Add missing NULL value to filter parameter array
636d849 Update memory allocation approach
f0d1d38 Add new allocation functions
97d00a0 Fix writing of uninitialized data to logs
e72c9b2 Merge branch 'release-2.0.0' into develop
cf2b712 Merge branch 'release-2.0.0' into develop
8917c5c Change the logic behind valid list entry checks
c10deff Improve documentation about version_string
f59f1f7 Merge branch 'develop' of ssh://github.com/mariadb-corporation/maxscale-new into develop
c88edb3 Backend authentication failure improvement
abd5bee Revert "Backend authentication failure improvement"
5bb3107 Backend authentication failure improvement
b7f434a Add new allocation functions
3f022fa Fix stupid mistake
99c4317 Merge remote-tracking branch 'origin/MXS-677' into develop
3c1ded6 Added connection/authentication failure error reporting in SHOW SLAVE STATUS
0a60f7b Tidy up and deal with review points.
ba103ff blr_slave.c: Update strncpy usage
467331e blr_master.c: Strncpy usage updates
d2b7c0c Merge remote-tracking branch 'origin/develop-nullauth-merge' into develop
5a8c1d0 qc: Measure execution time at the right place.
bccdb93 Merge branch 'NullAuthDeny' into develop
2e6511c Add 5.5.5 prefix to all version strings that lack it
314655a Improve DCB and session initialization and list handling
e1c43f0 MXS-655: Make MaxScale logging logrotate(8) compatible
ce36afd MXS-626: Don't log a header unless maxlog enabled
dcd47a7 blr_file.c: Replace uses of strncpy
6b8f576 bls_slave.c: Replace strncpy with memcpy
68a0039 Add list preallocation, tidy up, simplify init.
cb37d1b Fix copyright etc headers.
11a400d Tidy; comment; fix bad copies and mistakes.
7e36ec4 Add list manager files.
c4794e3 Initial code for list manager.
1b42e25 Merge remote-tracking branch 'origin/MXS-765' into develop
d50f617 Fix problems, extend tests, respond to review.
dcb4a91 Filter test folder removed
0b60dbe Add a couple of comments.
83cdba0 Fix overwriting problem.
ba5d353 Fix overwriting problem.
53671cb Small fixes in response to review.
173d049 blr.c: Review strncpy usage
4ff6ef2 binlog_common.c: Replace strncpy with memcpy
f238e03 maxbinlogcheck.s: Replace strncpy
9807f8d harness: Replace unnecessary use of strncpy
8c7fe6a avro: Modify strncpy usage
9b8008e Small improvements.
b7f784f Fix mistakes in testqueuemanager.c
cc26962 Restore missing poll.c code; add testqueuemanager.c.
2e91806 Format the filter harness
22059e6 Initial implementation connection queueing.
c604dc2 readwritesplit.c: Improve COM_INIT_DB handling
454d920 schemarouter.c: Replace strncpy with strcpy
8e85d66 sharding_common.c: Too long a database name handled explicitly
77f4446 Astyle schemarouter
491f7c2 maxinfo.c: Replace strncpy with memcpy
6b98105 maxinfo: Reformat with astyle
c1dbf08 Handle oversize user and database names
5fa4a0f Merge branch 'develop' of ssh://github.com/mariadb-corporation/maxscale-new into develop
706963b BLR_DBUSERS_TAIL new var in blr.h
d75b9af Tweak comments, remove trailing blanks.
ab2400a Optimise statistics gathering by inline & simpler fns.
fb59ddc Remove unnecessary strncpy/strncat usage in Binlog Server
bdcd551 resultset.c: Change strncpy to memcpy
c6b1c5e Reject rather than cut too long a path
6d8f112 Remove unnecessary strncpy/strncat usage
18bf5ed Remove unnecessary strncpy usage
dc0e2db Make maxpasswd more userfriendly
c9c8695 Fix calculation of padded_len in encryptPassword
2cfd2c6 dbusers.c: Check strncpy usage
7ab9342 Make more thorough checks in secrets_readKeys
be7d593 Format cli.c debugcli.c testroute.c webserver.c
1ee5efb config.c: Check usage of strncpy
3043b12 gq_utils.c: Unnecessary use of strncpy removed
77874ac Add help to maxkeys
38392a3 Update secrets_writeKeys documentation
2d1325c Make SSL optional in MaxScale's own communication
bda00da Fix avro build failures
b2cb31a Add more OOM macros
41ccf17 Fix strdup usage
a48f732 Fix realloc calls
20771f6 Add forgotten extern "C" block
8faf35a Add maxscale allocation functions
bb47890 Add macros for OOM logging
afea388 Fix silly mistakes.
6dafd22 Make deny default for null auth; move code from common to auth.
This commit is contained in:
Johan Wikman
2016-08-17 08:36:01 +03:00
parent b2035745fc
commit dc7f7b6fb4
360 changed files with 11641 additions and 15856 deletions
Download Patch File Download Diff File Expand all files Collapse all files

162
Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md
View File

@ -1,5 +1,5 @@
# MariaDB MaxScale as a Binlog Server
MariaDB MaxScale is a database proxy that sits between a database layer and the clients of that database, the binlog router described here is somewhat different to that original concept, moving MariaDB MaxScale down to play a role within the database layer itself.
MariaDB MaxScale is a dynamic data routing platform that sits between a database layer and the clients of that database, the binlog router described here is somewhat different to that original concept, moving MariaDB MaxScale down to play a role within the database layer itself.
In a traditional MySQL replication setup a single master server is created and a set of slaves MySQL instances are configured to pull the binlog files from that master to the slaves. There are some problems, however, in this setup; when the number of slaves grows an increasing load is placed on the master, to serve the binlogs to each slave. When the master server fails, some action must be performed on every slave server before a new server can become the master server.
@ -127,6 +127,25 @@ When MariaDB MaxScale starts an error message may appear if current binlog file
During normal operations binlog events are not distributed to the slaves until a *COMMIT* is seen.
The default value is off, set *transaction_safety=on* to enable the incomplete transactions detection.
### semisync
This parameter controls whether binlog server could ask Master server to start the Semi-Synchronous replication.
In order to get semi-sync working the Master server must have the *rpl_semi_sync_master* plugin installed.
The available plugin and the value of GLOBAL VARIABLE *rpl_semi_sync_master_enabled* are checked in the Master registration phase: if plugin is installed in the Master database the binlog server then requests the semi-sync option.
Note:
- the network replication stream from Master has two additional bytes before each binlog event.
- the Semi-Sync protocol requires an acknoledge packet to be sent back to Master only when requested: the semi-sync flag will have value of 1.
This flag is set only if *rpl_semi_sync_master_enabled=1* in the Master, otherwise it will always have value of 0 and no ack packet is sent back.
Please note that semi-sync replication is only related to binlog server to Master communication.
### ssl_cert_verification_depth
This parameter sets the maximum length of the certificate authority chain that will be accepted. Legal values are positive integers.
This applies to SSL connection to master server that could be acivated either by writing options in master.ini or later via CHANGE MASTER TO.
This parameter cannot be modified at runtime, default is 9.
A complete example of a service entry for a binlog router service would be as follows.
```
@ -137,7 +156,7 @@ A complete example of a service entry for a binlog router service would be as fo
version_string=5.6.17-log
user=maxscale
passwd=Mhu87p2D
router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1
router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1,ssl_cert_verification_depth=9,semisync=1
```
The minimum set of router options that must be given in the configuration are are *server-id* and *master-id*, default values may be used for all other options.
@ -154,6 +173,21 @@ As per any service in MariaDB MaxScale a listener section is required to define
The protocol used by slaves for connection to MariaDB MaxScale is the same *MySQLClient* protocol that is used for client applications to connect to databases, therefore the same MariaDB MaxScale protocol module can be used.
It's also possible to enable SSL from clients (MySQL clients or Slave servers) by adding SSL options in the listener, or in a new one:
```
[Replication Listener_SSL]
type=listener
service=Replication
protocol=MySQLClient
port=5309
ssl=required
ssl_key=/path_to/key.pem
ssl_cert=/path_to/cert.pem
ssl_ca_cert=/path_to/ca-cert.pem
#ssl_version=TLSv10
```
Check the [Configuration-Guide](../Getting-Started/Configuration-Guide.md) for SSL options details.
# MariaDB MaxScale replication diagnostics
The binlog router module of MariaDB MaxScale produces diagnostic output that can be viewed via the `maxadmin` client application. Running the maxadmin command and issuing a show service command will produce a considerable amount of output that will show both the master connection status and statistics and also a block for each of the slaves currently connected.
@ -232,7 +266,67 @@ The binlog router module of MariaDB MaxScale produces diagnostic output that can
Users data: 0x156c030
Total connections: 2
Currently connected: 2
-bash-4.1$
```
If a slave is connected to MaxScale with SSL, an entry will be present in the Slave report:
```
Slaves:
Server-id: 106
Hostname: SBslave6
Slave UUID: 00019686-7777-7777-7777-777777777777
Slave_host_port: 188.165.213.5:40365
Username: massi
Slave DCB: 0x7fc01be3ba88
Slave connected with SSL: Established
```
Another Binlog Server diagnostic output comes from SHOW SLAVE STATUS MySQL command
```
MySQL [(none)]> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Binlog Dump
Master_Host: 88.26.197.94
Master_User: repl
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.003140
Read_Master_Log_Pos: 16682679
Relay_Log_File: mysql-bin.003140
Relay_Log_Pos: 16682679
Relay_Master_Log_File: mysql-bin.003140
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 16682679
Relay_Log_Space: 16682679
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: Yes
Master_SSL_CA_File: /home/maxscale/packages/certificates/client/ca.pem
Master_SSL_CA_Path:
Master_SSL_Cert: /home/maxscale/packages/certificates/client/client-cert.pem
Master_SSL_Cipher:
Master_SSL_Key: /home/maxscale/packages/certificates/client/client-key.pem
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 1111
Master_UUID: 6aae714e-b975-11e3-bc33-0401152c3d01
Master_Info_File: /home/maxscale/binlog/first/binlogs/master.ini
```
# Binlog router compatibility
@ -261,6 +355,12 @@ Please note that is such condition the only user for MySQL protocol connection t
master_user=repl
master_password=somepass
filestem=repl-bin
# Master SSL communication options
master_ssl=0
master_ssl_key=/home/mpinto/packages/certificates/client/client-key.pem
master_ssl_cert=/home/mpinto/packages/certificates/client/client-cert.pem
master_ssl_ca=/home/mpinto/packages/certificates/client/ca.pem
#master_tls_version=TLSv12
Enabling replication from a master server requires:
@ -306,6 +406,15 @@ The supported options are:
MASTER_LOG_FILE
MASTER_LOG_POS
and SSL options as well:
MASTER_SSL (0|1)
MASTER_SSL_CERT (path to certificate file)
MASTER_SSL_KEY (path to key file)
MASTER_SSL_CA (path to CA cerificate file)
MASTER_TLS_VERSION (allowed level of encryption used)
Further details about level of encryption or certificates could be found here [Configuration Guuide](../Getting-Started/Configuration-Guide.md)
There are some constraints related to *MASTER_LOG_FILE* and *MASTER_LOG_POS*:
*MASTER_LOG_FILE* could be changed to next binlog in sequence with *MASTER_LOG_POS=4* or to current one at current position.
@ -348,6 +457,53 @@ This command removes *master.ini* file, blanks all master configuration in memor
Note: existing binlog files are not touched by this command.
Examples with SSL options:
MySQL [(none)]> CHANGE MASTER TO MASTER_SSL = 1, MASTER_SSL_CERT='/home/maxscale/packages/certificates/client/client-cert.pem', MASTER_SSL_CA='/home/maxscale/packages/certificates/client/ca.pem', MASTER_SSL_KEY='/home/maxscale/packages/certificates/client/client-key.pem', MASTER_TLS_VERSION='TLSv12';
MySQL [(none)]> CHANGE MASTER TO MASTER_TLS_VERSION='TLSv12';
MySQL [(none)]> CHANGE MASTER TO MASTER_SSL = 0;
#### Some constraints:
- In order to enable/re-enable Master SSL comunication the MASTER_SSL=1 option is required and all certificate options must be explicitey set in the same CHANGE MASTER TO command.
- New certificate options changes take effect after maxScale restart or after MASTER_SSL=1 with the new options.
Note:
- SHOW SLAVE STATUS displays all the options but MASTER_TLS_VERSION value.
- Maxadmin, 'show services' or 'show service $binlog_service' displays all the options when SSL is on.
- STOP SLAVE is required for CHANGE MASTER TO command (any option)
- START SLAVE will use new SSL options for Master SSL communication setup.
Examples:
mysql client
MySQL> SHOW SLAVE STATUS\G
Master_SSL_Allowed: Yes
Master_SSL_CA_File: /home/mpinto/packages/certificates/client/ca.pem
Master_SSL_CA_Path:
Master_SSL_Cert: /home/mpinto/packages/certificates/client/client-cert.pem
Master_SSL_Cipher:
Master_SSL_Key: /home/mpinto/packages/certificates/client/client-key.pem
maxadmin client
MaxScale>'show service BinlogServer'
Service: BinlogServer
Router: binlogrouter (0x7fd8c3002b40)
State: Started
Master connection DCB: 0x7fd8c8fce228
Master SSL is ON:
Master SSL CA cert: /home/mpinto/packages/certificates/client/ca.pem
Master SSL Cert: /home/mpinto/packages/certificates/client/client-cert.pem
Master SSL Key: /home/mpinto/packages/certificates/client/client-key.pem
Master SSL tls_ver: TLSv12
### Slave servers setup
Examples of *CHANGE MASTER TO* command issued on a slave server that wants to gets replication events from MariaDB MaxScale binlog router: