diff --git a/Documentation/About/About-MaxScale.md b/Documentation/About/About-MaxScale.md index ef0723914..61dc71bef 100644 --- a/Documentation/About/About-MaxScale.md +++ b/Documentation/About/About-MaxScale.md @@ -1,23 +1,47 @@ # About MaxScale -The MariaDB Corporation **MaxScale** is an intelligent proxy that allows forwarding of database statements to one or more database servers using complex rules, which can be based on a semantic understanding of the database statements and the roles of the various servers within the backend cluster of databases. +The MariaDB Corporation **MaxScale** is an intelligent proxy that allows +the forwarding of database statements to one or more database servers. +The forwarding is performed using rules that can be based on a the semantic +understanding of the database statements and on the roles of the various +servers within the backend cluster of databases. -MaxScale is designed to provide load balancing and high availability functionality transparently to the applications. In addition it provides a highly scalable and flexible architecture, with plugin components to support different protocols and routing decisions. +MaxScale is designed to provide, transparently to applications, load +balancing and high availability functionality. In addition, it provides +a highly scalable and flexible architecture, with plugin components to +support different protocols and routing approaches. -MaxScale is implemented in C so as to operate speedily. It also makes extensive use of the asynchronous I/O capabilities of the Linux operating system. The epoll system is used to provide the event driven framework for the input and output via sockets. Similar features in Windows® could be used in future development of MaxScale. +MaxScale makes extensive use of the asynchronous I/O capabilities of the +Linux operating system, combined with a fixed number of worker threads. +The epoll system is used to provide the event driven framework for the +input and output via sockets. Similar features in Windows® could +be used in future development of MaxScale. -Many of the services provided by MaxScale are implemented as external shared object modules which can be loaded at runtime. These modules support a fixed interface, communicating the entry points via a structure consisting of a set of function pointers. This structure is called the "module object". Additional modules can be created to work with MaxScale. +Many of the services provided by MaxScale are implemented as external +shared object modules, which are loaded at runtime. These modules +support a fixed interface, communicating the entry points via a structure +consisting of a set of function pointers. This structure is called the +"module object". Additional modules can be created to work with MaxScale. -One group of modules provides support for protocols, both for clients that communicate with MaxScale and for backend servers. The code that routes the queries to the backend servers is also loaded as external shared objects and they are referred to as routing modules. Another group of modules work on data as it passes through MaxScale, and they are known as filters. +One group of modules provides support for protocols, both for clients +that communicate with MaxScale and for backend servers. The code that +routes the queries to the backend servers is also loaded as external +shared objects and they are referred to as routing modules. Another +group of modules work on data as it passes through MaxScale, and they +are known as filters. -A Google Group exists for MaxScale that can be used to discuss ideas, issues and communicate with the MaxScale community: +A Google Group exists for MaxScale that can be used to discuss ideas, +issues and communicate with the MaxScale community: Send email to [maxscale@googlegroups.com](mailto:maxscale@googlegroups.com) or use the [forum](http://groups.google.com/forum/#!forum/maxscale) interface - + Bugs can be reported in the MariaDB Jira [https://mariadb.atlassian.net](https://mariadb.atlassian.net) ## Installing MaxScale -Information about installing MaxScale, either from a repository or by building from source code, is included in the [MariaDB MaxScale Installation Guide](../Getting-Started/MariaDB-MaxScale-Installation-Guide.md). - -The same guide also provides basic information on running MaxScale. More detailed information about configuring MaxScale is given in the [Configuration Guide](/Documentation/Getting-Started/Configuration-Guide.md). +Information about installing MaxScale, either from a repository or by +building from source code, is included in the +[MariaDB MaxScale Installation Guide](../Getting-Started/MariaDB-MaxScale-Installation-Guide.md). +The same guide also provides basic information on running MaxScale. +More detailed information about configuring MaxScale is given in the +[Configuration Guide](../Getting-Started/Configuration-Guide.md). diff --git a/Documentation/About/Limitations.md b/Documentation/About/Limitations.md index ec0d7f357..f3df71ff7 100644 --- a/Documentation/About/Limitations.md +++ b/Documentation/About/Limitations.md @@ -12,7 +12,7 @@ Compression is not included in MySQL server handshake ## Limitations with Galera Cluster Monitoring -Master selection is based only on MIN(wsrep_local_index), no other server parameter. +The default master selection is based only on MIN(wsrep_local_index). This can be influenced with the server priority mechanic described in the [Galera Monitor](../Monitors/Galera-Monitor.md) manual. ## Limitations in the connection router diff --git a/Documentation/About/SETUP.md b/Documentation/About/SETUP.md deleted file mode 100644 index e28fb0d40..000000000 --- a/Documentation/About/SETUP.md +++ /dev/null @@ -1,25 +0,0 @@ -Installation and startup - -Untar the binary distribution in the desired location, -e.g. /usr/local/mariadb-maxscale - -Alternatively build from the source code using the instructions -in the [Building MaxScale from Source Code](../Getting-Started/Building-MaxScale-from-Source-Code.md) document. - -You can start MaxScale using `service maxscale start` or `systemctl start maxscale` if you installed the init.d scripts -or by manually starting the process from the bin folder of the installation directory. - -Configuration - -You need to create or edit the maxscale.cnf file in the /etc folder. -Define the services you wish to provide, the set of server definitions -you require, with the addresses and ports of those servers and also -define the listening ports for your various services. - -In order to view the internal activity of MaxScale you can either use -the maxadmin client interface with the cli routing module or telnet to -the port defined for the telnet listener. Initially you may login with -the user name of "admin" and the password "mariadb". Once connected type -help for an overview of the commands and help for the more -detailed help on commands. Use the add user command to add a new user, -this will also remove the admin/mariadb user. For a detailed guide about using the MaxAdmin interface, take a look at the [MaxAdmin](../Reference/MaxAdmin.md) guide. diff --git a/Documentation/Changelog.md b/Documentation/Changelog.md index fa304852f..2a66c75fb 100644 --- a/Documentation/Changelog.md +++ b/Documentation/Changelog.md @@ -1,9 +1,11 @@ #Changelog -These are the changes introduced in the next MaxScale version. This is not the official change log and the latest changelog can always be found in here7: [MaxScale 1.1 Release Notes](Release-Notes/MaxScale-1.1-Release-Notes.md) - ## MaxScale 1.3 * Added support for persistent backend connections +* The *binlog server* is now an integral component of MaxScale. +* The logging has been changed; instead of different log files there is one log file and different message priorities. + +For more details, please refer to [MaxScale 1.3 Release Notes](Release-Notes/MaxScale-1.3.0-Release-Notes.md) ## MaxScale 1.2 * Logfiles have been renamed. The log names are now named error.log, messages.log, trace.log and debug.log. diff --git a/Documentation/Documentation-Contents.md b/Documentation/Documentation-Contents.md index 527590b28..52d3135a3 100644 --- a/Documentation/Documentation-Contents.md +++ b/Documentation/Documentation-Contents.md @@ -34,16 +34,19 @@ ## Tutorials +The main tutorial for MaxScale consist of setting up MaxScale for the environment you are using with either a connection-based or a read/write-based configuration. + + - [MaxScale Tutorial](Tutorials/MaxScale-Tutorial.md) + +These tutorials are for specific use cases and module combinations. + - [Administration Tutorial](Tutorials/Administration-Tutorial.md) - [Filter Tutorial](Tutorials/Filter-Tutorial.md) - - [Galera Cluster Connection Routing Tutorial](Tutorials/Galera-Cluster-Connection-Routing-Tutorial.md) - - [Galera Cluster Read-Write Splitting Tutorial](Tutorials/Galera-Cluster-Read-Write-Splitting-Tutorial.md) - [MaxScale Information Schema Tutorial](Tutorials/MaxScale-Information-Schema.md) - - [MySQL Replication Connection Routing Tutorial](Tutorials/MySQL-Replication-Connection-Routing-Tutorial.md) - - [MySQL Replication Read-Write Splitting Tutorial](Tutorials/MySQL-Replication-Read-Write-Splitting-Tutorial.md) - [MySQL Cluster Setup](Tutorials/MySQL-Cluster-Setup.md) - [Replication Proxy with the Binlog Router Tutorial](Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md) - [RabbitMQ Setup and MaxScale Integration Tutorial](Tutorials/RabbitMQ-Setup-And-MaxScale-Integration.md) + - [RabbitMQ and Tee Filter Data Archiving Tutorial](Tutorials/RabbitMQ-And-Tee-Archiving.md) - [Nagios Plugins for MaxScale Tutorial](Tutorials/Nagios-Plugins.md) - [Simple Schema Sharding Tutorial](Tutorials/Simple-Sharding-Tutorial.md) @@ -66,6 +69,13 @@ Here are detailed documents about the filters MaxScale offers. They contain conf - [Named Server Filter](Filters/Named-Server-Filter.md) ## Monitors + +Common options for all monitor modules. + + - [Monitor Common](Monitors/Monitor-Common.md) + +Module specific documentation. + - [MySQL Monitor](Monitors/MySQL-Monitor.md) - [Galera Monitor](Monitors/Galera-Monitor.md) - [Multi-Master Monitor](Monitors/MM-Monitor.md) diff --git a/Documentation/Filters/RabbitMQ-Filter.md b/Documentation/Filters/RabbitMQ-Filter.md index 3d04702ac..35fb285d6 100644 --- a/Documentation/Filters/RabbitMQ-Filter.md +++ b/Documentation/Filters/RabbitMQ-Filter.md @@ -1,7 +1,7 @@ #RabbitMQ Filter ## Overview -This filter is designed to extract queries and transform them into a canonical form e.g. `INSERT INTO dabata.table VALUES ("John Doe", "Downtown",100,50.0);` turns into `INSERT INTO dabata.table VALUES ("?", "?",?,?);`. The filter pushes these canonized queries and their replies in to a RabbitMQ broker where they can later be retrieved. The retrieval can be done with your own application or the [RabbitMQ Consumer Client](RabbitMQ-Consumer-Client.md) utility tool, which reads the messages from the broker and sends the contents of those messages as SQL queries to a database. +This filter is designed to extract queries and transform them into a canonical form e.g. `INSERT INTO database.table VALUES ("John Doe", "Downtown",100,50.0);` turns into `INSERT INTO database.table VALUES ("?", "?",?,?);`. The filter pushes these canonicalized queries and their replies in to a RabbitMQ broker where they can later be retrieved. The retrieval can be done with your own application or the [RabbitMQ Consumer Client](RabbitMQ-Consumer-Client.md) utility tool, which reads the messages from the broker and sends the contents of those messages as SQL queries to a database. ## Configuration @@ -46,24 +46,22 @@ The RabbitMQ filter has parameters to control which queries are logged based on Option | Description | Accepted Values | Default | --------|-------------|-----------------|------------- - logging_trigger | Set the logging level | `all, source, schema, object` | `all` | - logging_strict | Sets whether to trigger when any of the parameters match or only if all parameters match | `true, false` | `false` | - logging_log_all | Log only SELECT, UPDATE, DELETE and INSERT or all possible queries | `true, false` | `true` | - logging_source_user | Comma-separated list of usernames to log | | | - logging_source_host | Comma-separated list of hostnames to log | | | - logging_schema | Comma-separated list of databases | | | - logging_object | Comma-separated list of database objects | - hostname | The server hostname where the messages are sent | | `localhost` | - port | Port to send the messages to | | `5672` | - username | Server login username | | `guest` | - password | Server login password | | `guest` | - vhost | The virtual host location on the server, where the messages are sent | | `/` | - exchange | The name of the exchange | | `default_exchange` | - exchange_type | The type of the exchange | `direct, fanout, topic, headers` | `direct` | - key | The routing key used when sending messages to the exchange | | `key` | - queue | The queue that will be bound to the used exchange | | | - ssl_CA_cert | Path to the CA certificate in PEM format | | | - ssl_client_cert | Path to the client cerificate in PEM format | | | + logging_trigger | Set the logging level | `all, source, schema, object` | `all` | + logging_strict | Sets whether to trigger when any of the parameters match or only if all parameters match | `true, false` | `false` | + logging_log_all | Log only SELECT, UPDATE, DELETE and INSERT or all possible queries | `true, false` | `true` | + logging_source_user | Comma-separated list of usernames to log | | | + logging_source_host | Comma-separated list of hostnames to log | | | + logging_schema | Comma-separated list of databases | | | + logging_object | Comma-separated list of database objects | + hostname | The server hostname where the messages are sent | | `localhost` | + port | Port to send the messages to | | `5672` | + username | Server login username | | `guest` | + password | Server login password | | `guest` | + vhost | The virtual host location on the server, where the messages are sent | | `/` | + exchange | The name of the exchange | | `default_exchange` | + exchange_type | The type of the exchange | `direct, fanout, topic, headers` | `direct` | + key | The routing key used when sending messages to the exchange | | `key` | + queue | The queue that will be bound to the used exchange | | | + ssl_CA_cert | Path to the CA certificate in PEM format | | | + ssl_client_cert | Path to the client certificate in PEM format | | | ssl_client_key | Path to the client public key in PEM format | | | - - diff --git a/Documentation/Filters/Regex-Filter.md b/Documentation/Filters/Regex-Filter.md index db99debdb..fdd2053b5 100644 --- a/Documentation/Filters/Regex-Filter.md +++ b/Documentation/Filters/Regex-Filter.md @@ -2,7 +2,11 @@ Regex Filter # Overview -The regex filter is a filter module for MaxScale that is able to rewrite query content using regular expression matches and text substitution. +The regex filter is a filter module for MaxScale that is able to rewrite query content using regular expression matches and text substitution. It uses the PCRE2 syntax which differs from the POSIX regular expressions used in MaxScale versions prior to 1.3.0. + +For all details about the PCRE2 syntax, please read the [PCRE2 syntax documentation](http://www.pcre.org/current/doc/html/pcre2syntax.html). + +Please note that the PCRE2 library uses a different syntax to refer to capture groups in the replacement string. The main difference is the usage of the dollar character instead of the backslash character for references e.g. `$1` instead of `\1`. For more details about the replacement string differences, please read the [Creating a new string with substitutions](http://www.pcre.org/current/doc/html/pcre2api.html#SEC34) chapter in the PCRE2 manual. # Configuration @@ -93,7 +97,7 @@ MySQL 5.1 used the parameter TYPE = to set the storage engine that should be use type=filter module=regexfilter options=ignorecase -match=TYPE[ ]*= +match=TYPE\s*= replace=ENGINE= [MyService] diff --git a/Documentation/Getting-Started/Configuration-Guide.md b/Documentation/Getting-Started/Configuration-Guide.md index f418c4146..d08c5aaaa 100644 --- a/Documentation/Getting-Started/Configuration-Guide.md +++ b/Documentation/Getting-Started/Configuration-Guide.md @@ -4,6 +4,22 @@ The purpose of this document is to describe how to configure MaxScale and to discuss some possible usage scenarios for MaxScale. MaxScale is designed with flexibility in mind, and consists of an event processing core with various support functions and plugin modules that tailor the behavior of the MaxScale itself. +# Table of Contents + +* [Configuration](#configuration) + * [Global Settings](#global-settings) + * [Service](#service) + * [Service and SSL](#service-and-ssl) + * [Server](#server) + * [Listener](#listener) +* [Router Modules](#routing-modules) +* [Diagnostic Modules](#diagnostic-modules) +* [Monitor Modules](#monitor-modules) +* [Filter Modules](#filter-modules) +* [Reloading Configuration](#reloading-configuration) +* [Authentication](#authentication) +* [Error Reporting](#error-reporting) + ### Terms @@ -17,32 +33,15 @@ statement routing | Statement routing is a method of handling requests in which protocol | A protocol is a module of software that is used to communicate with another software entity within the system. MaxScale supports the dynamic loading of protocol modules to allow for increased flexibility. module | A module is a separate code entity that may be loaded dynamically into MaxScale to increase the available functionality. Modules are implemented as run-time loadable shared objects. monitor | A monitor is a module that can be executed within MaxScale to monitor the state of a set of database. The use of an internal monitor is optional, monitoring may be performed externally to MaxScale. - listener | A listener is the network endpoint that is used to listen for connections to MaxScale from the client applications. A listener is associated to a single service, however a service may have many listeners. + listener | A listener is the network endpoint that is used to listen for connections to MaxScale from the client applications. A listener is associated to a single service, however, a service may have many listeners. connection failover| When a connection currently being used between MaxScale and the database server fails a replacement will be automatically created to another server by MaxScale without client intervention backend database | A term used to refer to a database that sits behind MaxScale and is accessed by applications via MaxScale. filter | A module that can be placed between the client and the MaxScale router module. All client data passes through the filter module and may be examined or modified by the filter modules. Filters may be chained together to form processing pipelines. -# Table of Contents - -* [Configuration](#configuration) - * [Global Settings](#global-settings) - * [Service](#service) - * [Service and SSL](#service-and-ssl) - * [Server](#server) - * [Listener](#listener) - * [Filter](#filter) - * [Monitor](#monitor) - * [Protocol](#protocol) -* [Router Modules](#router-modules) -* [Monitor Modules](#monitor-modules) -* [Filter Modules](#filter-modules) -* [Reloading Configuration](#reloading-configuration) -* [Authentication](#authentication) -* [Error Reporting](#error-reporting) - ## Configuration -The MaxScale configuration is read from a file which can be located in a number of placing, MaxScale will search for the configuration file in a number of locations. +The MaxScale configuration is read from a file that MaxScale will look for +in a number of places. 1. Location given with the --configdir= command line argument @@ -50,7 +49,7 @@ The MaxScale configuration is read from a file which can be located in a number An explicit path to a configuration file can be passed by using the `-f` option to MaxScale. -The configuration file itself is based on the ".ini" file format and consists of various sections that are used to build the configuration, these sections define services, servers, listeners, monitors and global settings. Parameters which expect a comma-separated list of values can be defined on multiple lines. The following is an example of a multi-line definition. +The configuration file itself is based on the ".ini" file format and consists of various sections that are used to build the configuration; these sections define services, servers, listeners, monitors and global settings. Parameters, which expect a comma-separated list of values can be defined on multiple lines. The following is an example of a multi-line definition. ``` [MyService] @@ -63,15 +62,22 @@ servers=server1, The values of the parameter that are not on the first line need to have at least one whitespace character before them in order for them to be recognized as a part of the multi-line parameter. -Please see the section about [Protocol Modules](#protocol-modules) for more details about MaxScale and the default directories where modules will be searched for. - ### Global Settings The global settings, in a section named `[MaxScale]`, allow various parameters that affect MaxScale as a whole to be tuned. #### `threads` -This parameter controls the number of worker threads that are handling the events coming from the kernel. MaxScale will auto-detect the number of processors of the system unless number of threads is manually configured. It is recommended that you let MaxScale detect how many cores the system has and leave this parameter undefined. The number of used cores will be logged into the message logs and if you are not satisfied with the auto-detected value, you can manually configure it. Increasing the amount of worker threads beyond the number of processor cores does not improve performance and can consume resources needlessly. +This parameter controls the number of worker threads that are handling the +events coming from the kernel. MaxScale will auto-detect the number of +processors of the system unless number of threads is manually configured. +It is recommended that you let MaxScale detect how many cores the system +has and leave this parameter undefined. The number of used cores will be +logged into the message logs and if you are not satisfied with the +auto-detected value, you can manually configure it. Increasing the amount +of worker threads beyond the number of processor cores does not improve +the performance, rather is likely to degrade it, and can consume resources +needlessly. ``` # Valid options are: @@ -105,33 +111,106 @@ Enable or disable the high precision timestamps in logfiles. Enabling this adds ms_timestamp=1 ``` -#### `log_messages` +#### `syslog` +Enable to disable to logging of messages to *syslog*. -Enable or disable logging of status messages. This logfile is enabled by default and contains information about the modules MaxScale is using and details about the configuration. +By default logging to *syslog* is enabled. +``` +# Valid options are: +# syslog=<0|1> +syslog=1 +``` + +To enable logging to syslog use the value 1 and to disable use +the value 0. + +#### `maxlog` +Enable to disable to logging of messages to MaxScale's log file. + +By default logging to *maxlog* is enabled. +``` +# Valid options are: +# syslog=<0|1> +maxlog=1 +``` + +To enable logging to the MaxScale log file use the value 1 and to +disable use the value 0. + +#### `log_to_shm` +Enable or disable the writing of the *maxscale.log* file to shared memory. +If enabled, then the actual log file will be created under `/dev/shm` and +a symbolic link to that file will be created in the *MaxScale* log directory. + +Logging to shared memory may be appropriate if *log_info* and/or *log_debug* +are enabled, as logging to a regular file may in that case cause performance +degradation, due to the amount of data logged. However, as shared memory is +a scarce resource, logging to shared memory should be used only temporarily +and not regularly. + +Since *MaxScale* can log to both file and *syslog* an approach that provides +maximum flexibility is to enable *syslog* and *log_to_shm*, and to disable +*maxlog*. That way messages will normally be logged to *syslog*, but if +there is something to investigate, *log_info* and *maxlog* can be enabled +from *maxadmin*, in which case informational messages will be logged to +the *maxscale.log* file that resides in shared memory. + +By default, logging to shared memory is disabled. ``` # Valid options are: -# log_messages=<0|1> -log_messages=1 +# log_to_shm=<0|1> +log_to_shm=1 ``` -To disable the log use the value 0 and to enable it use the value 1. +To enable logging to shared memory use the value 1 and to disable use +the value 0. -#### `log_trace` - -Enable or disable logging of tracing messages. This logfile is disabled by default due to the verbose nature of it. It contains information about the internal logic of MaxScale and the modules it is using. The trace log can be used to find out the reasons why some actions were done e.g routing a query to a master instead of a slave. +#### `log_warning` +Enable or disable the logging of messages whose syslog priority is *warning*. +Messages of this priority are enabled by default. ``` # Valid options are: -# log_trace=<0|1> -log_trace=1 +# log_warning=<0|1> +log_warning=0 ``` -To disable the log use the value 0 and to enable it use the value 1. +To disable these messages use the value 0 and to enable them use the value 1. + +#### `log_notice` +Enable or disable the logging of messages whose syslog priority is *notice*. +Messages of this priority provide information about the functioning of +MaxScale and are enabled by default. + +``` +# Valid options are: +# log_notice=<0|1> +log_notice=0 +``` + +To disable these messages use the value 0 and to enable them use the value 1. + +#### `log_info` + +Enable or disable the logging of messages whose syslog priority is *info*. +These messages provide detailed information about the internal workings of +MaxScale and should not, due to their frequency, be enabled, unless there +is a specific reason for that. For instance, from these messages it will be +evident, e.g., why a particular query was routed to the master instead of +to a slave. These informational messages are disabled by default. + +``` +# Valid options are: +# log_info=<0|1> +log_info=1 +``` + +To disable these messages use the value 0 and to enable them use the value 1. #### `log_debug` -Enable or disable logging of debugging messages. This logfile is disabled by default since it contains information only useful to the developers. +Enable or disable the logging of messages whose syslog priority is *debug*. This kind of messages are intended for development purposes and are disabled by default. ``` # Valid options are: @@ -139,7 +218,15 @@ Enable or disable logging of debugging messages. This logfile is disabled by def log_debug=1 ``` -To disable the log use the value 0 and to enable it use the value 1. +To disable these messages use the value 0 and to enable them use the value 1. + +#### `log_messages` + +**Deprecated** Use *log_notice* instead. + +#### `log_trace` + +**Deprecated** Use *log_info* instead. #### `log_augmentation` @@ -320,7 +407,7 @@ enable_root_user=true #### `localhost_match_wildcard_host` -This parameter enables matching of "127.0.0.1" (localhost) against "%" wildcard host for MySQL protocol authentication. The default value is `0`, so in order to authenticate a connection from the same machine as the one on which MaxScale is running, an explicit user@lcoalhost entry will be required in the MySQL user table. +This parameter enables matching of "127.0.0.1" (localhost) against "%" wildcard host for MySQL protocol authentication. The default value is `0`, so in order to authenticate a connection from the same machine as the one on which MaxScale is running, an explicit user@localhost entry will be required in the MySQL user table. #### `version_string` @@ -537,865 +624,81 @@ The `socket` option may be included in a listener definition, this configures th If a socket option and an address option is given then the listener will listen on both the specific IP address and the Unix socket. -### Filter +#### Available Protocols -Filters provide a means to manipulate or process requests as they pass through MaxScale between the client side protocol and the query router. A filter should be defined in a section with a type of filter. +The protocols supported by MaxScale are implemented as external modules that are loaded dynamically into the MaxScale core. They allow MaxScale to communicate in various protocols both on the client side and the backend side. Each of the protocols can be either a client protocol or a backend protocol. Client protocols are used for client-MaxScale communication and backend protocols are for MaxScale-database communication. -``` -[QLA] -type=filter -module=qlafilter -options=/tmp/QueryLog -``` - -The section name may then be used in one or more services by using the filters= parameter in the service section. In order to use the above filter for a service called "QLA Service", an entry of the following form would exist for that service. - -``` -[QLA Service] -type=service -router=readconnroute -router_options=slave -servers=server1,server2,server3,server4 -user=massi -passwd=6628C50E07CCE1F0392EDEEB9D1203F3 -filters=QLA -``` - -![image alt text](images/image_10.png) - -See the Services section for more details on how to configure the various options of a service. Note that some filters require parsing of the statement which makes them compatible with statement-based routers only, such as Read/Write Split router. - -#### `module` - -The module parameter defines the name of the loadable module that implements the filter. - -#### `options` - -The options parameter is used to pass options to the filter to control the actions the filter will perform. The values that can be passed differ between filter implementation, the inclusion of an options parameter is optional. - -#### Other Parameters - -Any other parameters present in the filters section will be passed to the filter to be interpreted by the filter. An example of this is the regexfilter that requires the two parameters `match` and `replace`: - -``` -[regex] -type=filter -module=regexfilter -match=form -replace=from -``` - -### Monitor - -In order for the various router modules to function correctly they require information about the state of the servers that are part of the service they provide. MaxScale has the ability to internally monitor the state of the back-end database servers or that state may be feed into MaxScale from external monitoring systems. If automated monitoring and failover of services is required this is achieved by running a monitor module that is designed for the particular database architecture that is in use. - -Monitors are defined in much the same way as other elements in the configuration file, with the section name being the name of the monitor instance and the type being set to monitor. - -This is an example configuration of the MySQL monitor module. It is intended for Master-Slave replication clusters and allows for replication lag detection. - -``` -[MySQL Monitor] -type=monitor -module=mysqlmon -servers=server1,server2,server3 -user=dbmonitoruser -passwd=dbmonitorpwd -monitor_interval=8000 -backend_connect_timeout=3 -backend_read_timeout=1 -backend_write_timeout=2 - -# mysqlmon specific options -detect_replication_lag=0 -detect_stale_master=0 -``` - -Here is an example configuration of the Galera cluster monitor. It detects when nodes are in sync and also assigns master and slave roles to nodes within MaxScale, allowing it to be used with modules designed for Master-Slave replication clusters. - -``` -[Galera Monitor] -type=monitor -module=galeramon -servers=server1,server2,server3 -user=dbmonitoruser -passwd=dbmonitorpwd -monitor_interval=8000 -backend_connect_timeout=3 -backend_read_timeout=1 -backend_write_timeout=2 - -# galeramon specific options -disable_master_failback=0 -available_when_donor=0 -disable_master_role_setting=0 -``` - -#### `module` - -The module parameter defines the name of the loadable module that implements the monitor. This module is loaded and executed on a separate thread within MaxScale. - -#### `servers` - -The servers parameter is a comma separated list of server names to monitor, these are the names defined elsewhere in the configuration file. The set of servers monitored by a single monitor need not be the same as the set of servers used within any particular server, a single monitor instance may monitor servers in multiple servers. - -Multiple monitors monitoring the same servers should be avoided. They can possibly make the whole cluster inoperable and a good example is the mixed use of the MySQL and the Galera monitors. The MySQL monitor requires a working Master-Slave replication for it to assign the Master and Slave roles inside MaxScale but the Galera monitor only looks for Galera specific status variables. These two monitors will cause a conflict when one tries to clear server states it sees as valid while the other is simultaneously setting new states to the rest of the servers. - -#### `user` - -The user parameter defines the username that the monitor will use to connect to the monitored databases. Depending on the monitoring module used this user will require specific privileges in order to determine the state of the nodes, details of those privileges can be found in the sections on each of the monitor modules. - -Individual servers may define override values for the user and password the monitor uses by setting the monuser and monpasswd parameters in the server section. - -#### `passwd` - -The password parameter may be either a plain text password or it may be an encrypted password. See the section on encrypting passwords for use in the `maxscale.cnf` file. - -#### `monitor_interval` - -The monitor_interval parameter sets the sampling interval in milliseconds for each monitor, the default value is 10000 milliseconds. - -#### `detect_replication_lag` - -This options if set to 1 will allow MySQL monitor to collect the replication lag among all configured slaves by checking the content of `maxscale_schema.replication_heartbeat` table. The master server writes in and slaves fetch a UNIX timestamp from that there. - -This timestamp, evaluated in seconds, is updated in each node server struct and it's used to calculate the replication lag. - -That value is also used by the Read / Write split module via `max_slave_replication_lag` and `LEAST_BEHIND_MASTER` options. - -Replication lag is measured by writing to a table, replication_heartbeat in the maxscale_schema, updates to this table will be observed on the slave in order to determine the lag between the slave and the master on which it was written. If the slave is many minutes behind the master and MaxScale is then started the information in the slave table is not available and that slave may be excluded from the routing decision. - -A specific grant for the monitor user might be required in order to create schema/table and for read/write operations. - -This monitor option is not enabled by default. - -#### `detect_stale_master` - -This options if set to 1 will allow MySQL monitor to select the previous selected Master for next operations even if no slaves at all are found by the monitor polling. - -This is such a case when the replication on all slave has been stopped via `STOP SLAVE` or the current configuration was removed by `RESET SLAVE ALL`. - -As there are no slaves the replication topology cannot be computed and MaxScale can only check if the current monitored server was the master before: if that's the case - -MySQL monitor adds to the server status field the `SERVER_STALE_STATUS` bit and a log entry appears in the Message Log file. - -If MaxScale or monitor is restarted and the Replication is still not configured or started there will not be any master server available even with this option enabled. - -This option is not enabled by default and should be used at the administrator risk. - -#### `disable_master_failback` - -This option if set to 1 will allow Galera monitor to keep the existing selected master even if another node, after joining back the cluster may be selected as candidate master. - -The master role assignment currently follows one rule: take the server with lowest `wsrep_local_index` value. - -By default, if a node takes a lower index than the current master one the monitor will set the master role to that node: this monitor option, if set, prevents the master change. - -The server status field may have the `SERVER_MASTER_STICKINESS` bit, meaning the current master selection is not based on the available rules but it's the one previously selected and then kept, accordingly to option value equal 1. - -Anyway, a new master will be selected in case of current master failure, regardless the option value. - -#### `available_when_donor` - -This option if set to 1 will allow Galera monitor to keep a node in `Donor` status in the server pool if it is using any xtrabackup method for SST, e.g. `wsrep_sst_method` equal to `xtrabackup` or `xtrabackup-v2`. - -As xtrabackup is a non-locking SST method, a node in `Donor` status can still be considered in sync. This option is not enabled by default and should be used as the administrator's discretion. - -#### `disable_master_role_setting` - -This option if set to 1 will stop the Galera monitor from setting the status of -backend servers to master or slave. It is applicable when the Galera router is -being used to spread writes across multiple nodes, so that no server is to be -nominated as the master. - -#### `backend_connect_timeout` - -This option, with default value of `3` sets the monitor connect timeout to backends. - -#### `backend_read_timeout` - -Default value is `1`. Read Timeout is the timeout in seconds for each attempt to read from the server. There are retries if necessary, so the total effective timeout value is three times the option value. That's for `mysql_real_connect` C API. - -#### `backend_write_timeout` - -Default value is `2`. Write Timeout is the timeout in seconds for each attempt to write to the server. There is a retry if necessary, so the total effective timeout value is two times the option value. That's for `mysql_real_connect` C API. - -## Protocol - -The protocols supported by MaxScale are implemented as external modules that are loaded dynamically into the MaxScale core. These modules reside in the directory `/usr/lib64/maxscale`. The location can be overridden with the `libdir=PATH` parameter under the `[maxscale]` section. It may also be set by passing the `-B PATH` or `--libdir=PATH` option on the MaxScale command line. - -### MySQLClient +##### `MySQLClient` This is the implementation of the MySQL protocol that is used by clients of MaxScale to connect to MaxScale. -### MySQLBackend +##### `MySQLBackend` The MySQLBackend protocol module is the implementation of the protocol that MaxScale uses to connect to the backend MySQL, MariaDB and Percona Server databases. This implementation is tailored for the MaxScale to MySQL Database traffic and is not a general purpose implementation of the MySQL protocol. -### telnetd +##### `telnetd` The telnetd protocol module is used for connections to MaxScale itself for the purposes of creating interactive user sessions with the MaxScale instance itself. Currently this is used in conjunction with a special router implementation, the debugcli. -### maxscaled +##### `maxscaled` The protocol used used by the maxadmin client application in order to connect to MaxScale and access the command line interface. -### HTTPD +##### `HTTPD` This protocol module is currently still under development, it provides a means to create HTTP connections to MaxScale for use by web browsers or RESTful API clients. -## Router Modules +## Routing Modules The main task of MaxScale is to accept database connections from client applications and route the connections or the statements sent over those connections to the various services supported by MaxScale. -There are two flavors of routing that MaxScale can perform, connection based routing and statement based routine. These each have their own characteristics and costs associated with them. - -### Connection Based Routing - -Connection based routing is a mechanism by which MaxScale will, for each incoming connection decide on an appropriate outbound server and will forward all statements to that server without examining the internals of the statement. Once an inbound connection is associated to a particular backend database it will remain connected to that server until the connection is closed or the server fails. The Read Connection Router is an example of connection-based routing. - -### Statement Based Routing - -Statement based routing is somewhat different, the routing modules examine every statement the client sends and determines, on a per statement basis, which of the set of backend servers in the service is best to execute the statement. This gives better dynamic balancing of the load within the cluster but comes at a cost. The query router must understand the statement that is being routed and may have to parse the statement in order to achieve this. - -Parsing within the router adds overhead to the cost of routing and makes this type of router best suitable for loads in which the gains outweigh this added cost. The added cost from statement parsing also gives the possibility to create and use new type of filters which are based on statement processing. In contrast to the added processing cost, statement-based routing may increase the performance of the cluster by offloading statements away from the master when possible. - -### Available Routing Modules - -Currently a small number of query routers are available, these are in different stages of completion and offer different facilities. +Currently a number of routing modules are available, these are designed for a range of different needs. +Connection based load balancing: * [ReadConnRoute](../Routers/ReadConnRoute.md) + +Read/Write aware statement based router: * [ReadWriteSplit](../Routers/ReadWriteSplit.md) + +Simple sharding on database level: * [SchemaRouter](../Routers/SchemaRouter.md) -In addition to these routing modules, the binlogrouter module can act as a binary log proxy between a master and slave servers. +Binary log server: +* [Binlogrouter](../Routers/Binlogrouter.md) -#### Readconnroute +## Diagnostic modules -This is a connection based query router that was originally targeted at environments in which the clients already performed splitting of read and write queries into separate connections. +These modules are used for diagnostic purposes and can tell about the status of MaxScale and the cluster it is monitoring. -Whenever a new connection is received the router will examine the state of all the servers that form part of the service and route the connection to the server with least connections currently that matches the filter constraints given in the router options. This results in a balancing of the active connections, however different connections may have different lifetimes and the connections may become unbalanced when later viewed. - -The read connection router can be configured to balance the connections from the clients across all the backend servers that are running, just those backend servers that are currently replication slaves or those that are replication masters when routing to a master slave replication environment. When a Galera cluster environment is in use the servers can be filtered to just the set that are part of the cluster and in the _Synced_ state. These options are configurable via the router_options that can be set within a service. The `router_option` values supported are `master`, `slave` and `synced`. - -##### Master/Slave Replication Setup - -To set up MaxScale to route connections evenly between all the current slave servers in a replication cluster, a service entry of the form shown below is required: - -``` -[Read Service] -type=service -router=readconnroute -router_options=slave -servers=server1,server2,server3,server4 -user=maxscale -passwd=thepasswd -``` - -And then add a listener for this service, which defines the port and protocol that MaxScale uses: - -``` -[Read Listener] -type=listener -service=Read Service -protocol=MySQLClient -port=4006 -``` - -The client can now connect to port 4006 on the host where MaxScale runs. Statements sent using this connection will then be routed to one of the slaves in the server set defined in the Read Service. Exactly which is selected will be determined by balancing the number of connections to each of those whose current state is *Slave*. - -Altering `router_options` to be `slave,master` would result in the connections being balanced between all the servers within the cluster. - -It is assumed that the client will have a separate connection to the master server, however this can be routed via MaxScale, allowing MaxScale to designate which server is master. To do this you would add a second service and listener definition for the master server. - -``` -[Write Service] -type=service -router=readconnroute -router_options=master -servers=server1,server2,server3,server4 -user=maxscale -passwd=thepasswd - -[Write Listener] -type=listener -service=Write Service -protocol=MySQLClient -port=4007 -``` - -This allows the clients to direct write requests to port 4007 and read requests to port 4006 of the MaxScale host without the clients needing to understand the configuration of the Master/Slave replication cluster. - -Connections to port 4007 would automatically be directed to the server that is the master for replication at the time connection is opened. Whilst this is a simple mapping to a single server it does give the advantage that the clients have no requirement to track which server is currently the master, devolving responsibility for managing the failover to MaxScale. - -In order for MaxScale to be able to determine the state of these servers the **mysqlmon** monitor module should be run against the set of servers that comprise the service. - -##### Galera Cluster Configuration for Read Connection router - -Although not primarily designed for a multi-master replication setup, it is possible to use **readconnroute** in this situation. The **readconnroute** connection router can be used to balance the connections across a Galera cluster. A special monitor is available that detects if nodes are joined to a Galera Cluster, with the addition of a router option to only route connections to nodes marked as synced. MaxScale can ensure that users are never connected to a node that is not a full cluster member. - -``` -[Galera Service] -type=service -router=readconnroute -router_options=synced -servers=server1,server2,server3,server4 -user=maxscale -passwd=thepasswd - -[Galera Listener] -type=listener -service=Galera Service -protocol=MySQLClient -port=3336 - -[Galera Monitor] -type=monitor -module=galeramon -servers=server1,server2,server3,server4 -user=galeramon -passwd=galeramon -``` - -The specialized Galera monitor can also select one of the node in the cluster as _Master_, the others will be marked as _Slave_. These roles are only assigned to _Synced_ nodes. - -It then possible to have services/listeners with `router_options=master` or `slave` accessing a subset of all Galera nodes. The _Synced_ state simply means: access all nodes. Examples of different **readconn** router configurations for Galera: - -``` -[Galera Master Service] -type=service -router=readconnroute -router_options=master - -[Galera Slave Service] -type=service -router=readconnroute -router_options=slave -``` - -##### MySQL Cluster Configuration for Read Connection router - -The **readconnroute** connection router can be used to balance the connections across a MySQL cluster SQL nodes. A special monitor is available that detects if SQL nodes are connected to data nodes, with the addition of a router option to only route connections to nodes marked as NDB. -MaxScale can ensure that users are never connected to a node that is not a full cluster member. - -``` -[NDB Cluster Monitor] -type=monitor -module=ndbclustermon -servers=server1,server2 -user=monitor -passwd=monitor - -[MySQL Cluster Service] -type=service -router=readconnroute -router_options=ndb -servers=server1,server2 - -[Cluster Listener] -type=listener -service=MySQL Cluster Service -protocol=MySQLClient -port=4906 -``` - -The `ndb` router option simply means: access all SQL nodes marked with NDB status, i.e. they are members of the cluster. - -#### Read/Write Split Router - -The Read/Write Split Router is implemented in readwritesplit module. It is a statement-based router that has been designed for use within Master/Slave replication environments. It examines and optionally parses every statement to find out whether the statement can be routed to slave instead of master. - -##### Starting a readwritesplit router session - -When client connects to readwritesplit service for the first time, client is authenticated against user data loaded from backend database. After successful authentication connection for client queries is created and followed by that, a readwritesplit router session is initialized. - -Router session processes its specific configuration parameters and establishes connections to master and slaves. The number of slaves in each session depends on the value of `max_slave_connections` parameter (default is `1`) and the availability of slaves. Most suitable number of slaves varies as it depends on the number of clients, and the backend servers and the type of load. In Figure below Server 1 is the master and Servers 2-7 are the available slaves. In this example `max_slave_connections=3`. - -![image alt text](images/image_11.png) - -##### Routing to *Master* - -Routing to master is important for data consistency and because majority of writes are written to binlog and thus become replicated to slaves. - -The following operations are routed to master: - -* write statements, -* all statements within an open transaction, -* stored procedure calls, and -* user-defined function calls. -* DDL statements (`DROP`|`CREATE`|`ALTER TABLE` … etc.) -* `EXECUTE` (prepared) statements -* all statements using temporary tables - -In addition to these, if the **readwritesplit** service is configured with the `max_slave_replication_lag` parameter, and if all slaves suffer from too much replication lag, then statements will be routed to the _Master_. (There might be other similar configuration parameters in the future which limit the number of statements that will be routed to slaves.) - -##### Routing to *Slave*s - -The ability to route some statements to *Slave*s is important because it also decreases the load targeted to master. Moreover, it is possible to have multiple slaves to share the load in contrast to single master. - -Queries which can be routed to slaves must be auto committed and belong to one of the following group: - -* read-only database queries, -* read-only queries to system, or user-defined variables, -* `SHOW` statements, and -* system function calls. - -##### Routing to every session backend - -A third class of statements includes those which modify session data, such as session system variables, user-defined variables, the default database, etc. We call them session commands, and they must be replicated as they affect the future results of read and write operations, so they must be executed on all servers that could execute statements on behalf of this client. - -Session commands include for example: - -* `SET` statements -* `USE `*``* -* system/user-defined variable assignments embedded in read-only statements, such as `SELECT (@myvar := 5)` -* `PREPARE` statements -* `QUIT`, `PING`, `STMT RESET`, `CHANGE USER`, etc. commands - -**NOTE: if variable assignment is embedded in a write statement it is routed to _Master_ only. For example, `INSERT INTO t1 values(@myvar:=5, 7)` would be routed to _Master_ only.** - -The router stores all of the executed session commands so that in case of a slave failure, a replacement slave can be chosen and the session command history can be repeated on that new slave. This means that the router stores each executed session command for the duration of the session. Applications that use long-running sessions might cause MaxScale to consume a growing amount of memory unless the sessions are closed. This can be solved by setting a connection timeout on the application side. - -##### Configuring the Read/Write Split router - -Read/Write Split router-specific settings are specified in the configuration file of MaxScale in its specific section. The section can be freely named but the name is used later as a reference from listener section. - -The configuration consists of mandatory and optional parameters. For a complete list of these, please read the [ReadWriteSplit](../Routers/ReadWriteSplit.md) documentation. - -An example of Read/Write Split router configuration : - -``` -[RWSplit Service] -type=service -router=readwritesplit -router_options=slave_selection_criteria=LEAST_BEHIND_MASTER -max_slave_connections=50% -max_slave_replication_lag=61 -servers=server1,server2,server3,server4 -user=myuser -passwd=mypass -filters=qla|fetch|from -``` - -In addition to this, readwritesplit needs configuration for a listener, for all servers listed, and for each filter. Listener, server - and filter configurations are described in their own sections in this document. - -An important parameter is the `max_slave_connections=50%` parameter. This sets the number of slaves each client connection will use. With the default values, client connections will only use a single slave for reads. For example, setting the parameter value to 100% will use all available slaves and read queries will be balanced evenly across all slaves. Changing the `max_slave_connections` parameter and `slave_selection_criteria` router option allows you to change the way MaxScale will balance reads. For more information about the `slave_selection_criteria` router option, please read the ReadWriteSplit documentation. - -Below is a listener example for the "RWSplit Service" defined above: - -``` -[RWSplit Listener] -type=listener -service=RWSplit Service -protocol=MySQLClient -port=4044 -``` - -The client would merely connect to port 4044 on the MaxScale host and statements would be directed to the master, slave or all backends as appropriate. Determination of the master or slave status may be done via a monitor module within MaxScale or externally. In this latter case the server flags would need to be set via the MaxScale debug interface, in future versions an API will be available for this purpose. - -##### Galera Cluster Configuration for Read/Write Split router - - -Galera monitor assigns Master and Slave roles to appropriate sync'ed Galera nodes. Using **readwritesplit** with Galera is seamless; the only change needed to the configuration above is replacing the list of MySQL replication servers with list of Galera nodes. With the same example as above: - -Simply configure a RWSplit Service with Galera nodes: - -``` -[RWSplit Service] -type=service -router=readwritesplit -max_slave_connections=50% -servers=galera_node1,galera_node2,galera_node3 -user=myuser -passwd=mypass -filters=qla|fetch|from -``` - -#### CLI - -The command line interface as used by `maxadmin`. This is a variant of the debugcli that is built slightly differently so that it may be accessed by the client application `maxadmin`. The CLI requires the use of the `maxscaled` protocol. - -##### CLI Configuration - -There are two components to the definition required in order to run the command line interface to use with MaxAdmin; a service and a listener. - -The default entries required are shown below. - -``` -[CLI] -type=service -router=cli - -[CLI Listener] -type=listener -service=CLI -protocol=maxscaled -address=localhost -port=6603 -``` - -Note that this uses the default port of 6603 and confines the connections to localhost connections only. Remove the address= entry to allow connections from any machine on your network. Changing the port from 6603 will mean that you must allows pass a -p option to the MaxAdmin command. - -#### Debug CLI - -The **debugcli** router is a special kind of statement based router. Rather than direct the statements at an external data source they are handled internally. These statements are simple text commands and the results are the output of debug commands within MaxScale. The service and listener definitions for a debug cli service only differ from other services in that they require no backend server definitions. - -##### Debug CLI Configuration - -The definition of the debug cli service is illustrated below - -``` -[Debug Service] -type=service -router=debugcli - -[Debug Listener] -type=listener -service=Debug Service -protocol=telnetd -port=4442 -``` - -Connections using the telnet protocol to port 4442 of the MaxScale host will result in a new debug CLI session. A default username and password are used for this module, new users may be created using the add user command. As soon as any users are explicitly created the default username will no longer continue to work. The default username is admin with a password of mariadb. - -The debugcli supports two modes of operation, `developer` and `user`. The mode is set via the `router_options` parameter. The user mode is more suited to end-users and administrators, whilst the develop mode is explicitly targeted to software developing adding or maintaining the MaxScale code base. Details of the differences between the modes can be found in the debugging guide for MaxScale. The default is `user` mode. The following service definition would enable a developer version of the debugcli. - -``` -[Debug Service] -type=service -router=debugcli -router_options=developer -``` - -It should be noted that both `user` and `developer` instances of debugcli may be defined within the same instance of MaxScale, however they must be defined as two distinct services, each with a distinct listener. - -``` -[Debug Service] -type=service -router=debugcli -router_options=developer - -[Debug Listener] -type=listener -service=Debug Service -protocol=telnetd -port=4442 - -[Admin Service] -type=service -router=debugcli - -[Admin Listener] -type=listener -service=Debug Service -protocol=telnetd -port=4242 -``` +* [MaxAdmin Module](../Routers/CLI.md) +* [Telnet Module](../Routers/Debug-CLI.md) ## Monitor Modules -Monitor modules are used by MaxScale to internally monitor the state of the backend databases in order to set the server flags for each of those servers. The router modules then use these flags to determine if the particular server is a suitable destination for routing connections for particular query classifications. The monitors are run within separate threads of MaxScale and do not affect the MaxScale performance. +Monitor modules are used by MaxScale to internally monitor the state of the backend databases in order to set the server flags for each of those servers. The router modules then use these flags to determine if the particular server is a suitable destination for routing connections for particular query classifications. The monitors are run within separate threads of MaxScale and do not affect MaxScale's routing performance. -The use of monitors is optional, it is possible to run MaxScale with external monitoring, in which case arrangements must be made for an external entity to set the status of each of the servers that MaxScale can route to. +The use of monitors is highly recommended but it is also possible to run MaxScale without a monitor module. In this case an external monitoring system which sets the status of each server via MaxAdmin is needed. -Parameters that apply to all monitors are: - -* `monitor_interval` -* `backend_connect_timeout` -* `backend_read_timeout` -* `backend_write_timeout` - -Other parameters are monitor specific. - -### mysqlmon - -The MySQLMon monitor is a simple monitor designed for use with MySQL Master/Slave replication cluster. To execute the mysqlmon monitor an entry as shown below should be added to the MaxScale configuration file. - -``` -[MySQL Monitor] -type=monitor -module=mysqlmon -servers=server1,server2,server3,server4 -``` - -This will monitor the 4 servers; `server1`, `server2`, `server3` and `server4`. It will set the status of running or failed and master or slave for each of the servers. - -The monitor uses the username given in the monitor section or the server specific user that is given in the server section to connect to the server. This user must have sufficient permissions on the database to determine the state of replication. The roles that must be granted to this user are `REPLICATION SLAVE` and `REPLICATION CLIENT`. - -To create a user that can be used to monitor the state of the cluster, the following commands could be used, assuming that MaxScale is running on the host 'maxscalehost' - -``` -MariaDB [mysql]> create user 'maxscalemon'@'maxscalehost' identified by 'Ha79hjds'; -Query OK, 0 rows affected (0.01 sec) - -MariaDB [mysql]> grant REPLICATION SLAVE on *.* to 'maxscalemon'@'maxscalehost'; -Query OK, 0 rows affected (0.00 sec) - -MariaDB [mysql]> grant REPLICATION CLIENT on *.* to 'maxscalemon'@'maxscalehost'; -Query OK, 0 rows affected (0.00 sec) -``` - -MySQL monitor fetches the `@@server_id` variable and other information from `SHOW SLAVE STATUS` in order to compute the replication topology tree that may include intermediate master servers, called relay servers. - -The *Master* server used by router modules is the so called "root master": a server that has the `SERVER_MASTER` status bit set and it's at the lowest level of the replication depth. - -MySQL monitor may optionally (`detect_replication_lag=1`) detect the replication lag among servers by using the `maxscale_schema.replication_heartbeat` table: the monitor user must have rights to create it and write into. - -Another option (`detect_stale_master=1`) may also allow to set a Stale Master when the replication has been stopped or the configuration doesn't allow to have both IO and SQL replication threads running on all slaves: the previous detected working Master will be selected for read and write operations. - -Please note, those two options are not enabled by default. - -### galeramon - -The Galeramon monitor is a simple monitor designed for use with MySQL Galera cluster. To execute the galeramon monitor an entry as shown below should be added to the MaxScale configuration file. - -``` -[Galera Monitor] -type=monitor -module=galeramon -servers=galera_node1,galera_node2,galera_node3 -``` - -This will monitor the 4 servers; server1, server2, server3 and server4. It will set the status of *Running* or *Failed* and *Joined* for those servers that reported the Galera JOINED status. - -To create a user that can be used to monitor the state of the cluster, the following commands could be used, assuming that MaxScale is running on the host maxscalehost. - -``` -MariaDB [mysql]> create user 'maxscalemon'@'maxscalehost' identified by 'Ha79hjds'; -Query OK, 0 rows affected (0.01 sec) -``` - -The Galera monitor also assigns *Master* and *Slave* roles to the configured nodes. Among the set of synced servers, the one with the lowest value of `wsrep_local_index` is selected as the current *Master* while the others are given the role of *Slave*; that's the only available master selection rule right now. - -In this way it is possible to configure the node access based not only on *Synced* state but even on *Master* and *Slave* role enabling the use of the Read/Write Split router on a Galera cluster and avoiding any possible write conflict. - -It may happen that after a node failure or reboot or node joining back the cluster, the node's `wsrep_local_index` in the cluster nodes changes. This might result in monitor assigning the *Master* role to another server. In order to avoid such situation, the `disable_master_failback` configuration option helps keep the current master regardless of the value of `wsrep_local_index`. This option is not enabled by default. - -This is an example status for a Galera server node: - -``` -Server 0x261fe50 (server2) - - Server: 192.168.1.101 - Status: Master, Synced, Running - -Protocol: MySQLBackend - -Port: 3306 - - Server Version: 5.5.40-MariaDB-wsrep-log - Node Id: 0 - -Server 0x2d1b3c0 (server4) - - Server: 192.168.122.144 - Status: Slave, Synced, Running - Protocol: MySQLBackend - Port: 3306 - Server Version: 5.5.40-MariaDB-wsrep-log - Node Id: 1 -``` - -### ndbclustermon - -The NDB Cluster Monitor (ndbclustermon) is a simple router designed for use with MySQL Cluster. To execute the ndclustermon monitor an entry as shown below should be added to the MaxScale configuration file. - -Example for monitor section: - -``` -[NDB Cluster Monitor] -type=monitor -module=ndbclustermon -servers=server1,server2 -``` - -This will monitor the two SQL nodes `server1` and `server2` and will set the status of *NDB* and *Running* or *Failed* for those servers with the value of status variable `Ndb_number_of_ready_data_nodes` greater than 0, i.e. the monitored SQL node is able to contact one or more data nodes. - -Example of a monitored server: - -``` - Server 0x3873a40 (server2) - Server: 192.168.90.81 - Status: NDB, Running - Protocol: MySQLBackend - Port: 3306 - Server Version: 5.5.38-ndb-7.2.17-cluster-gpl - Node Id: 13 -``` - -The MySQL Cluster variables fetched by the monitor are: - -``` -mysql> SHOW STATUS LIKE 'Ndb_number_of_ready_data_nodes'; -+--------------------------------+-------+ -+ -| Variable_name | Value | -+--------------------------------+-------+ -| Ndb_number_of_ready_data_nodes | 2 | -+--------------------------------+-------+ -1 row in set (0.00 sec) -``` - -The result is greater than 0 so the NBD status is added to status - -``` -mysql> SHOW STATUS LIKE 'Ndb_cluster_node_id'; -+---------------------+-------+ -| Variable_name | Value | -+---------------------+-------+ -| Ndb_cluster_node_id | 13 | -+---------------------+-------+ -1 row in set (0.00 sec) -``` - -The value is stored in `node_id` server field. +* [Mysql Monitor](../Monitors/MySQL-Monitor.md) +* [Galera Monitor](../Monitors/Galera-Monitor.md) +* [NDBCluster Monitor](../Monitors/NDB-Cluster-Monitor.md) +* [Multi-Master Monitor](../Monitors/MM-Monitor.md) ## Filter Modules -Currently four example filters are included in the MaxScale distribution +![image alt text](images/image_10.png) - - - - - - - - - - - - - - - - - - - - - - - - - -
ModuleDescription
testfilterStatement counting Filter - a simple filter that counts the number of SQL statements executed within a session. Results may be viewed via the debug interface.
qlafilterQuery Logging Filter - a simple query logging filter that write all statements for a session into a log file for that session.
regexfilterQuery Rewrite Filter - an example of how filters can alter the query contents. This filter allows a regular expression to be defined, along with replacement text that should be substituted for every match of that regular expression.
teeA filter that duplicates SQL requests and sends the duplicates to another service within MaxScale.
topfilterA filter that records the top running queries in terms of execution time. The number of queries to maintain is configurable, upon completion of a session a log file is written with the details of those top queries.
+Filters provide a means to manipulate or process requests as they pass through MaxScale between the client side protocol and the query router. A full explanation of each filter's functionality can be found in its documentation. +The [Filter Tutorial](../Tutorials/Filter-Tutorial.md) document shows how you can add a filter to a service and combine multiple filters in one service. -These filters are merely examples of what may be achieved with the filter API and are not sophisticated or consider as suitable for production use, they merely illustrate the functionality possible. - -### Statement Counting Filter - -The statement counting filter is implemented in the module names testfilter and merely keeps a count of the number of SQL statements executed. The filter requires no options to be passed and takes no parameters. The statement count can be viewed via the diagnostic and debug interface of MaxScale. - -In order to add this filter to an existing service create a filter section to name the filter as follows - -``` -[counter] -type=filter -module=testfilter -``` - -Then add the filter to your service by including the filters= parameter in the service section. - - filters=counter - -### Query Log All (QLA) Filter - -The Query Log All Filter (qlafilter) simply writes all SQL statements to a log file along with a timestamp for the statement. An example of the file produced by the QLA filter is shown below - -``` -00:36:04.922 5/06/2014, select @@version_comment limit 1 -00:36:12.663 5/06/2014, SELECT DATABASE() -00:36:12.664 5/06/2014, show databases -00:36:12.665 5/06/2014, show tables -``` - -A new file is created for each client connection, the name of the logfile can be controlled by the use of the router options. No parameters are used by the QLA filter. The filter is implemented by the loadable module qlafilter. - -To add the QLA filter to a service you must create a filter section to name the filter, associated the loadable module and define the filename option. - -``` -[QLA] -type=filter -module=qlafilter -options=/tmp/QueryLog -``` - -Then add the filters= parameter into the service that you wish to log by adding this parameter to the service section - - filters=QLA - -A log file will be created for each client connection, the name of that log file will be `/tmp/QueryLog.`*``* - -### Regular Expression Filter - -The regular expression filter is a simple text based query rewriting filter. It allows a regular expression to be used to match text in a SQL query and then a string replacement to be made against that match. The filter is implemented by the regexfilter loadable module and is passed two parameters, a match string and a replacement string. - -To add the filter to your service you must first create a filter section to name the filter and give the match and replacement strings. Here we define a filter that will convert to MariaDB 10 command show all slaves status to the older form of show slave status for MariaDB 5.5. - -``` -[slavestatus] -type=filter -module=regexfilter -match=show *all *slaves -replace=show slave -``` - -You must then add this filter to your service by adding the filters= option - - filters=slavestatus - -Another example would be a filter to convert from the MySQL 5.1 create table syntax that used the `TYPE` keyword to the newer `ENGINE` keyword. - -``` -[EnginerFilter] -type=filter -module=regexfilter -match=TYPE -replace=ENGINE -``` - -This would then change the SQL sent by a client application written to work with MySQL 5.1 into SQL that was compliant with MySQL 5.5. The statement - - create table supplier(id integer, name varchar(80)) TYPE=innodb - -would be replaced with - - create table supplier(id integer, name varchar(80)) ENGINE=innodb - -before being sent to the server. Note that the text in the match string is case-insensitive. - -### Tee Filter - -The **tee** filter is a filter module for MaxScale that acts as a "plumbing" fitting in the MaxScale filter toolkit. It can be used in a filter pipeline of a service to make a copy of requests from the client and dispatch a copy of the request to another service within MaxScale. - -The configuration block for the **tee** filter requires the minimal filter parameters in its section within the `maxscale.cnf` file that defines the filter to load and the service to send the duplicates to. - -``` -[ArchiveFilter] -type=filter -module=tee -service=Archive -``` - -In addition parameters may be added to define patterns to match against to either include or exclude particular SQL statements to be duplicated. You may also define that the filter is only active for connections from a particular source or when a particular user is connected. - -### Top Filter - -The top filter is a filter module for MaxScale that monitors every SQL statement that passes through the filter. It measures the duration of that statement, the time between the statement being sent and the first result being returned. The top N times are kept, along with the SQL text itself and a list sorted on the execution times of the query is written to a file upon closure of the client session. - -The configuration block for the **top** filter requires the minimal filter options in its section within the `maxscale.cnf` file, stored in `/etc/maxscale.cnf`. - -``` -[MyLogFilter] -type=filter -module=topfilter -filebase=/var/log/Top10Queries -count=10 -``` - -In addition parameters may be added to define patterns to match against to either include or exclude particular SQL statements to be duplicated. You may also define that the filter is only active for connections from a particular source or when a particular user is connected. - +* [Query Log All (QLA) Filter](../Filters/Query-Log-All-Filter.md) +* [Regular Expression Filter](../Filters/Regex-Filter.md) +* [Tee Filter](../Filters/Tee-Filter.md) +* [Top Filter](../Filters/Top-N-Filter.md) +* [Database Firewall Filter](../Filters/Database-Firewall-Filter.md) +* [Query Redirection Filter](../Filters/Named-Server-Filter.md) +* [RabbitMQ Filter](../Filters/RabbitMQ-Filter.md) ## Reloading Configuration @@ -1467,7 +770,7 @@ and short notations ## Error Reporting -MaxScale is designed to be executed as a service, therefore all error reports, including configuration errors, are written to the MaxScale error log file. By default, MaxScale will log to a set of files in the directory `/var/log/maxscale`, the only exception to this is if the log directory is not writable, in which case a message is sent to the standard error descriptor. +MaxScale is designed to be executed as a service, therefore all error reports, including configuration errors, are written to the MaxScale error log file. By default, MaxScale will log to a file in `/var/log/maxscale`, the only exception to this is if the log directory is not writable, in which case a message is sent to the standard error descriptor. ### Troubleshooting diff --git a/Documentation/Getting-Started/MariaDB-MaxScale-Installation-Guide.md b/Documentation/Getting-Started/MariaDB-MaxScale-Installation-Guide.md index bf752b1fd..bcb2ff48b 100644 --- a/Documentation/Getting-Started/MariaDB-MaxScale-Installation-Guide.md +++ b/Documentation/Getting-Started/MariaDB-MaxScale-Installation-Guide.md @@ -36,7 +36,7 @@ An example configuration file is installed into the `/etc/` folder. This file sh ## Building MaxScale From Source Code -Alternatively you may download the MaxScale source and build your own binaries. To do this, refer to the separate document [Building MaxScale from Source Code](/Documentation/Getting-Started/Building-MaxScale-from-Source-Code.md) +Alternatively you may download the MaxScale source and build your own binaries. To do this, refer to the separate document [Building MaxScale from Source Code](Building-MaxScale-from-Source-Code.md) ## Configuring MaxScale diff --git a/Documentation/Monitors/Galera-Monitor.md b/Documentation/Monitors/Galera-Monitor.md index e91373297..1b82506b2 100644 --- a/Documentation/Monitors/Galera-Monitor.md +++ b/Documentation/Monitors/Galera-Monitor.md @@ -18,41 +18,9 @@ passwd=mypwd ``` -## Optional parameters for all monitors +## Common Monitor Parameters -Here are optional parameters that are common for all the monitors. - -### `monitor_interval` - -This is the time the monitor waits between each cycle of monitoring. The default value of 10000 milliseconds (10 seconds) should be lowered if you want a faster response to changes in the server states. The value is defined in milliseconds and the smallest possible value is 100 milliseconds. - -``` -monitor_interval=2500 -``` - -### `backend_connect_timeout` - -This parameter controls the timeout for connecting to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 3 seconds. - -``` -backend_connect_timeout=6 -``` - -### `backend_write_timeout` - -This parameter controls the timeout for writing to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 2 seconds. - -``` -backend_write_timeout=4 -``` - -### `backend_read_timeout` - -This parameter controls the timeout for reading from a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 1 seconds. - -``` -backend_read_timeout=2 -``` +For a list of optional parameters that all monitors support, read the [Monitor Common](Monitor-Common.md) document. ## Galera Monitor optional parameters @@ -90,10 +58,6 @@ Enable interaction with server priorities. This will allow the monitor to determ use_priority=true ``` -### Common Monitor Parameters - -For a list of optional parameters that all monitors support, read the [Monitor Common](Monitor-Common.md) document. - ## Interaction with Server Priorities If the `use_priority` option is set and a server is configured with the `priority=` parameter, galeramon will use that as the basis on which the master node is chosen. This requires the `disable_master_role_setting` to be undefined or disabled. The server with the lowest value in `priority` will be chosen as the master node when a replacement Galera node is promoted to a master server inside MaxScale. diff --git a/Documentation/Monitors/MM-Monitor.md b/Documentation/Monitors/MM-Monitor.md index a90d2bf71..039840122 100644 --- a/Documentation/Monitors/MM-Monitor.md +++ b/Documentation/Monitors/MM-Monitor.md @@ -18,41 +18,9 @@ passwd=mypwd ``` -## Optional parameters for all monitors +## Common Monitor Parameters -Here are optional parameters that are common for all the monitors. - -### `monitor_interval` - -This is the time the monitor waits between each cycle of monitoring. The default value of 10000 milliseconds (10 seconds) should be lowered if you want a faster response to changes in the server states. The value is defined in milliseconds and the smallest possible value is 100 milliseconds. - -``` -monitor_interval=2500 -``` - -### `backend_connect_timeout` - -This parameter controls the timeout for connecting to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 3 seconds. - -``` -backend_connect_timeout=6 -``` - -### `backend_write_timeout` - -This parameter controls the timeout for writing to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 2 seconds. - -``` -backend_write_timeout=4 -``` - -### `backend_read_timeout` - -This parameter controls the timeout for reading from a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 1 seconds. - -``` -backend_read_timeout=2 -``` +For a list of optional parameters that all monitors support, read the [Monitor Common](Monitor-Common.md) document. ## Multi-Master Monitor optional parameters @@ -67,7 +35,3 @@ This is a situation which can happen if all slave servers are unreachable or the ``` detect_stale_master=true ``` - -### Common Monitor Parameters - -For a list of optional parameters that all monitors support, read the [Monitor Common](Monitor-Common.md) document. diff --git a/Documentation/Monitors/Monitor-Common.md b/Documentation/Monitors/Monitor-Common.md index 1d27dc419..eb89805b3 100644 --- a/Documentation/Monitors/Monitor-Common.md +++ b/Documentation/Monitors/Monitor-Common.md @@ -4,6 +4,38 @@ This document lists optional parameters that all current monitors support. ## Parameters +### `monitor_interval` + +This is the time the monitor waits between each cycle of monitoring. The default value of 10000 milliseconds (10 seconds) should be lowered if you want a faster response to changes in the server states. The value is defined in milliseconds and the smallest possible value is 100 milliseconds. + +``` +monitor_interval=2500 +``` + +### `backend_connect_timeout` + +This parameter controls the timeout for connecting to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 3 seconds. + +``` +backend_connect_timeout=6 +``` + +### `backend_write_timeout` + +This parameter controls the timeout for writing to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 2 seconds. + +``` +backend_write_timeout=4 +``` + +### `backend_read_timeout` + +This parameter controls the timeout for reading from a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 1 seconds. + +``` +backend_read_timeout=2 +``` + ### `script` This command will be executed when a server changes its state. The parameter should be an absolute path to a command or the command should be in the executable path. The user which is used to run MaxScale should have execution rights to the file itself and the directory it resides in. diff --git a/Documentation/Monitors/MySQL-Monitor.md b/Documentation/Monitors/MySQL-Monitor.md index f32af7fde..165007389 100644 --- a/Documentation/Monitors/MySQL-Monitor.md +++ b/Documentation/Monitors/MySQL-Monitor.md @@ -25,41 +25,9 @@ MariaDB [(none)]> grant replication client on *.* to 'maxscale'@'maxscalehost'; Query OK, 0 rows affected (0.00 sec) ``` -## Optional parameters for all monitors +## Common Monitor Parameters -Here are optional parameters that are common for all the monitors. - -### `monitor_interval` - -This is the time the monitor waits between each cycle of monitoring. The default value of 10000 milliseconds (10 seconds) should be lowered if you want a faster response to changes in the server states. The value is defined in milliseconds and the smallest possible value is 100 milliseconds. - -``` -monitor_interval=2500 -``` - -### `backend_connect_timeout` - -This parameter controls the timeout for connecting to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 3 seconds. - -``` -backend_connect_timeout=6 -``` - -### `backend_write_timeout` - -This parameter controls the timeout for writing to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 2 seconds. - -``` -backend_write_timeout=4 -``` - -### `backend_read_timeout` - -This parameter controls the timeout for reading from a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 1 seconds. - -``` -backend_read_timeout=2 -``` +For a list of optional parameters that all monitors support, read the [Monitor Common](Monitor-Common.md) document. ## MySQL Monitor optional parameters @@ -92,10 +60,6 @@ Enable support for MySQL 5.1 replication monitoring. This is needed if a MySQL s mysql51_replication=true ``` -### Common Monitor Parameters - -For a list of optional parameters that all monitors support, read the [Monitor Common](Monitor-Common.md) document. - ## Example 1 - Monitor script Here is an example shell script which sends an email to an admin when a server goes down. diff --git a/Documentation/Monitors/NDB-Cluster-Monitor.md b/Documentation/Monitors/NDB-Cluster-Monitor.md index 806c7858f..d7cbe6a29 100644 --- a/Documentation/Monitors/NDB-Cluster-Monitor.md +++ b/Documentation/Monitors/NDB-Cluster-Monitor.md @@ -18,42 +18,6 @@ passwd=mypwd ``` -## Optional parameters for all monitors - -Here are optional parameters that are common for all the monitors. - -### `monitor_interval` - -This is the time the monitor waits between each cycle of monitoring. The default value of 10000 milliseconds (10 seconds) should be lowered if you want a faster response to changes in the server states. The value is defined in milliseconds and the smallest possible value is 100 milliseconds. - -``` -monitor_interval=2500 -``` - -### `backend_connect_timeout` - -This parameter controls the timeout for connecting to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 3 seconds. - -``` -backend_connect_timeout=6 -``` - -### `backend_write_timeout` - -This parameter controls the timeout for writing to a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 2 seconds. - -``` -backend_write_timeout=4 -``` - -### `backend_read_timeout` - -This parameter controls the timeout for reading from a monitored server. It is in seconds and the minimum value is 1 second. The default value for this parameter is 1 seconds. - -``` -backend_read_timeout=2 -``` - ### Common Monitor Parameters For a list of optional parameters that all monitors support, read the [Monitor Common](Monitor-Common.md) document. diff --git a/Documentation/Reference/MaxAdmin.md b/Documentation/Reference/MaxAdmin.md index 2c801ae81..f6865b764 100644 --- a/Documentation/Reference/MaxAdmin.md +++ b/Documentation/Reference/MaxAdmin.md @@ -8,20 +8,20 @@ Mark Riddoch Last Updated: 24th June 2015 -[Overview](#overview) -[Running MaxAdmin](#running) -[Working With Administration Interface Users](#interface) -[Getting Help](#help) -[Working with Services](#services) -[Working with Servers](#servers) -[Working with Sessions](#sessions) -[Descriptor Control Blocks](#dcbs) -[Working with Filters](#filters) -[Working with Monitors](#monitors) -[MaxScale Status Commands](#statuscommands) -[Administration Commands](#admincommands) -[Configuring MaxScale to Accept MaxAdmin Connections](#connections) -[Tuning MaxScale](#tuning) + - [Overview](#overview) + - [Running MaxAdmin](#running) + - [Working With Administration Interface Users](#interface) + - [Getting Help](#help) + - [Working with Services](#services) + - [Working with Servers](#servers) + - [Working with Sessions](#sessions) + - [Descriptor Control Blocks](#dcbs) + - [Working with Filters](#filters) + - [Working with Monitors](#monitors) + - [MaxScale Status Commands](#statuscommands) + - [Administration Commands](#admincommands) + - [Configuring MaxScale to Accept MaxAdmin Connections](#connections) + - [Tuning MaxScale](#tuning) # Overview @@ -821,29 +821,38 @@ In order to determine what modules are in use, and the version and status of tho debugcli | Router | V1.1.1 | 1.0.0 | Alpha cli | Router | V1.0.0 | 1.0.0 | Alpha ----------------+-------------+---------+-------+------------------------- - MaxScale> + MaxScale> This command provides important version information for the module. Each module has two versions; the version of the module itself and the version of the module API that it supports. Also included in the output is the status of the module, this may be "In Development", “Alpha”, “Beta”, “GA” or “Experimental”. -## Rotating the log files +## Enabling syslog and maxlog logging -MaxScale write a number of log files in the log directory within MaxScale home directory. The default option for these is that the grow continually, it is recommended that periodically the log files are rotated. This will close the current log file and open a new one with a new name. The log file names use a sequence number which is incremented each time the logs are rotated. +MaxScale can log messages to syslog, to a log file or to both. The approach can be set in the config file, but can also be changed from maxadmin. Syslog logging is identified by *syslog* and file logging by *maxlog*. -It is possible to rotate just a single log file, using the flush log command and the name of the log to flush. The names that are recognized by MaxAdmin are error, message, trace or debug. + MaxScale> enable syslog + MaxScale> disable maxlog - MaxScale> flush log message +**NOTE** If you disable both, then you will see no messages at all. + +## Rotating the log file + +MaxScale logs messages to a log file in the log directory of MaxScale. As the log file grows continuously, it is recommended to periodically rotate it. When rotated, the current log file will be closed and a new one with a new name opened. The log file name contain a sequence number, which is incremented each time the log is rotated. + +There are two ways for rotating the log - *flush log maxscale* and *flush logs* - and the result is identical. The two alternatives are due to historical reasons; earlier MaxScale had several different log files. + + MaxScale> flush log maxscale MaxScale> The flush logs command may be used to rotate all logs with a single command. - MaxScale> flush log + MaxScale> flush logs MaxScale> ## Change MaxScale Logging Options -Two commands are provided to change the logging levels within MaxScale, disable log and enable log. Using these commands the various log levels can be turned on and off, the supported levels are trace, debug and message. The error log level can not be turned off. +From version 1.3 onwards, MaxScale has a single log file where messages of various priority (aka severity) are logged. Consequently, you no longer enable or disable log files but log priorities. The priorities are the same as those of syslog and the ones that can be enabled or disabled are *debug*, *info*, *notice* and *warning*. *Error* and any more severe messages can not be disabled. - MaxScale> enable log trace - MaxScale> disable log debug - MaxScale> + MaxScale> enable log-priority info + MaxScale> disable log-priority notice + MaxScale> Please note that changes made via this interface will not persist across restarts of MaxScale. To make a permanent change edit the maxscale.cnf file. diff --git a/Documentation/Release-Notes/MaxScale-1.0-Release-Notes.md b/Documentation/Release-Notes/MaxScale-1.0-Release-Notes.md index 4ce52f6c3..1d6649143 100644 --- a/Documentation/Release-Notes/MaxScale-1.0-Release-Notes.md +++ b/Documentation/Release-Notes/MaxScale-1.0-Release-Notes.md @@ -14,7 +14,7 @@ The MaxScale monitor module for Master/Slave replication is now able to correctl ### Support For Prepared Statements -Prepared statements are now correctly recognised by MaxScale, with the prepare stage being sent to all the eligible servers that could eventually run the statement. Statements are then execute on a single server. +Prepared statements are now correctly recognized by MaxScale, with the prepare stage being sent to all the eligible servers that could eventually run the statement. Statements are then execute on a single server. ### Slave Failure Resilience diff --git a/Documentation/Release-Notes/MaxScale-1.2.0-Release-Notes.md b/Documentation/Release-Notes/MaxScale-1.2.0-Release-Notes.md index 4ffdcba1f..acd4fdd5d 100644 --- a/Documentation/Release-Notes/MaxScale-1.2.0-Release-Notes.md +++ b/Documentation/Release-Notes/MaxScale-1.2.0-Release-Notes.md @@ -5,7 +5,7 @@ This document details the changes in version 1.2 since the release of the 1.1.1 GA Release of the MaxScale product. ###***PLEASE NOTICE: MaxScale installation directories have changed in this version*** -The 1.2 version of MaxScale differs from previos versions in its installation layout. Please take great care when upgrading MaxScale from previous versions to version 1.2. An automatic upgrade will not work due to the severe changes in the installation layout. +The 1.2 version of MaxScale differs from previous versions in its installation layout. Please take great care when upgrading MaxScale from previous versions to version 1.2. An automatic upgrade will not work due to the severe changes in the installation layout. ## New Features @@ -31,7 +31,7 @@ MaxScale now supports SSL/TLS encrypted connections to MaxScale. Now you can configure MaxScale monitor module to automatically launch a script when it detects change in the state of a backend server. The script can be any customer script defined by you to take diagnostic or reporting action. With this you can easily customize MaxScale's behavior. ### Lsyncd configuration guide -A new tutorial has beed added which helps you keep MaxScale's configuration files in sync across multiple hosts. This allows for easier HA setups with MaxScale and guarantees up-to-date configuration files on all nodes. The tutorial can be found [here](../Reference/MaxScale-HA-with-lsyncd.md). +A new tutorial has been added which helps you keep MaxScale's configuration files in sync across multiple hosts. This allows for easier HA setups with MaxScale and guarantees up-to-date configuration files on all nodes. The tutorial can be found [here](../Reference/MaxScale-HA-with-lsyncd.md). ## Bug fixes diff --git a/Documentation/Release-Notes/MaxScale-1.3.0-Release-Notes.md b/Documentation/Release-Notes/MaxScale-1.3.0-Release-Notes.md new file mode 100644 index 000000000..8f645d2c2 --- /dev/null +++ b/Documentation/Release-Notes/MaxScale-1.3.0-Release-Notes.md @@ -0,0 +1,245 @@ +# MariaDB MaxScale 1.3 Release Notes + +This document describes the changes in release 1.3, when compared to +release 1.2.1. + +## 1.3.0 Beta + +**NOTE** This is a beta release. We think it is better than 1.2.1, +but there may still be rough edges. Keep that in mind when trying it +out. + +For any problems you encounter, please consider submitting a bug +report at [Jira](https://mariadb.atlassian.net). + +## New Features + +### Persistent Connections + +MaxScale 1.3.0 introduces the concept of *Persistent Connections*. With +that is meant that the connection from MaxScale to the backend server is +not terminated even if the connection from the client to MaxScale is. +If a client makes frequent short connections, there may be a benefit from +using the *Persistent Connection* feature as it may reduce the time it +takes from establishing a connection from the client through MaxScale to +the backend server. + +Additional information is available in the following document: +* [Administration Tutorial](../Tutorials/Administration-Tutorial.md) + +### Binlog Server + +There are new administrative commands: STOP SLAVE, START SLAVE, RESET SLAVE +and CHANGE MASTER TO. The master server details are now provided by a +master.ini file located in binlog directory and could be changed via +CHANGE MASTER TO command issued via MySQL connection to MaxScale. + +Before migrating to 1.3.0 it is necessary to put a writable master.ini file +into binlog directory, containing these parameters: + +``` +[binlog_configuration] +master_host=127.0.0.1 +master_port=3308 +master_user=repl +master_password=somepass +filestem=repl-bin +``` + +Users may change parameters according to their configuration. + +**Note**: the "servers" parameter is no longer required in the service +definition. + +Additional information is available in the following documents: +* [Binlogrouter Tutorial](../Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md) +* [Upgrading binlogrouter to 1.3.0](../Upgrading/Upgrading-Binlogrouter-to-1.3.0.md) + +### Logging Changes + +Before 1.3, MaxScale logged data to four different log files; *error*, +*message*, *trace* and *debug*. Complementary and/or alternatively, MaxScale +could also log to syslog, in which case messages intended for the error and +message file were logged there. What files were enabled and written to was +controlled by entries in the MaxScale configuration file. + +This has now been changed so that MaxScale logs to a single +file - *maxscale.log* - and each logged entry is prepended with *error*, +*warning*, *notice*, *info* or *debug*, depending on the seriousness or +priority of the message. The levels are the same as those of syslog. +MaxScale is still capable of complementary or alternatively logging to syslog. + +What used to be logged to the *message* file is now logged as a *notice* +message and what used to be written to the *trace* file, is logged as an +*info* message. + +By default, *notice*, *warning* and *error* messages are logged, while +*info* and *debug* messages are not. Exactly what kind of messages are +logged can be controlled via the MaxScale configuration file, but enabling +and disabling different kinds of messages can also be performed at runtime +from maxadmin. + +Earlier, the *error* and *message* files were written to the filesystem, +while the *trace* and *debug* files were written to shared memory. The +one and only log file of MaxScale is now by default written to the filesystem. +This will have performance implications if *info* and *debug* messages are +enabled. + +If you want to retain the possibility of turning on *info* and *debug* +messages, without it impacting the performance too much, the recommended +approach is to add the following entries to the MaxScale configuration file: + +``` +[maxscale] +syslog=1 +maxlog=0 +log_to_shm=1 +``` + +This will have the effect of MaxScale creating the *maxscale.log* into +shared memory, but not logging anything to it. However, all *notice*, +*warning* and *error* messages will be logged to syslog. + +Then, if there is a need to turn on *info* messages that can be done via +the maxadmin interface: + +``` +MaxScale> enable log-priority info +MaxScale> enable maxlog +``` + +Note that *info* and *debug* messages are never logged to syslog. + +### PCRE2 integration + +MaxScale now uses the PCRE2 library for regular expressions. This has been +integrated into the core configuration processing and most of the modules. +The main module which uses this is the regexfilter which now fully supports +the PCRE2 syntax with proper substitutions. For a closer look at how this +differs from the POSIX regular expression syntax take a look at the +[PCRE2 documentation](http://www.pcre.org/current/doc/html/pcre2syntax.html). + +**Please note**, that the substitution string follows different rules than +the traditional substitution strings. The usual way of referring to capture +groups in the substitution string is with the backslash character followed +by the capture group reference e.g. `\1` but the PCRE2 library uses the dollar +character followed by the group reference. To quote the PCRE2 native API manual: + +``` +In the replacement string, which is interpreted as a UTF string in UTF mode, and is checked for UTF validity unless the PCRE2_NO_UTF_CHECK option is set, a dollar character is an escape character that can specify the insertion of characters from capturing groups in the pattern. The following forms are recognized: + + $$ insert a dollar character + $ insert the contents of group + ${} insert the contents of group +``` + +### Improved launchable scripts + +The launchable scripts were modified to allow usage without wrapper scripts. +The scripts are now executed as they are in the configuration files with certain +keywords being replaced with the initiator, event and node list. For more +details, please read the [Monitor Common](../Monitors/Monitor-Common.md) document. + +## Bug fixes + +Here is a list of bugs fixed since the release of MaxScale 1.2.1. + + * [MXS-414](https://mariadb.atlassian.net/browse/MXS-414): Maxscale crashed every day! + * [MXS-415](https://mariadb.atlassian.net/browse/MXS-415): MaxScale 1.2.1 crashed with Signal 6 and 11 + * [MXS-351](https://mariadb.atlassian.net/browse/MXS-351): Router error handling can cause crash by leaving dangling DCB pointer + * [MXS-428](https://mariadb.atlassian.net/browse/MXS-428): Maxscale crashes at startup. + * [MXS-376](https://mariadb.atlassian.net/browse/MXS-376): MaxScale terminates with SIGABRT. + * [MXS-269](https://mariadb.atlassian.net/browse/MXS-269): Crash in MySQL backend protocol + * [MXS-500](https://mariadb.atlassian.net/browse/MXS-500): Tee filter hangs when statement aren't duplicated. + * [MXS-447](https://mariadb.atlassian.net/browse/MXS-447): Monitors are started before they have been fully configured + * [MXS-417](https://mariadb.atlassian.net/browse/MXS-417): Single character wildcard doesn't work in MaxScale + * [MXS-409](https://mariadb.atlassian.net/browse/MXS-409): prepare should not hit all servers + * [MXS-405](https://mariadb.atlassian.net/browse/MXS-405): Maxscale bin router crash + * [MXS-412](https://mariadb.atlassian.net/browse/MXS-412): show dbusers segmentation fault + * [MXS-289](https://mariadb.atlassian.net/browse/MXS-289): Corrupted memory or empty value are in Master_host field of SHOW SLAVE STATUS when master connection is broken + * [MXS-283](https://mariadb.atlassian.net/browse/MXS-283): SSL connections leak memory + * [MXS-54](https://mariadb.atlassian.net/browse/MXS-54): Write failed auth attempt to trace log + * [MXS-501](https://mariadb.atlassian.net/browse/MXS-501): Use hangs when Tee filter uses matching + * [MXS-499](https://mariadb.atlassian.net/browse/MXS-499): Init script error on Debian Wheezy + * [MXS-323](https://mariadb.atlassian.net/browse/MXS-323): mysql_client readwritesplit handleError seems using wrong dcb and cause wrong behavior + * [MXS-494](https://mariadb.atlassian.net/browse/MXS-494): Weight calculation favors servers without connections + * [MXS-493](https://mariadb.atlassian.net/browse/MXS-493): SIGFPE when weightby parameter is 0 and using LEAST_GLOBAL_CONNECTIONS + * [MXS-492](https://mariadb.atlassian.net/browse/MXS-492): Segfault if server is missing weighting parameter + * [MXS-360](https://mariadb.atlassian.net/browse/MXS-360): Persistent connections: maxadmin reports 0 all the time even if connections are created + * [MXS-429](https://mariadb.atlassian.net/browse/MXS-429): Binlog Router crashes due to segmentation fault with no meaningful error if no listener is configured + * [MXS-416](https://mariadb.atlassian.net/browse/MXS-416): Orphan sessions appear after many network errors + * [MXS-472](https://mariadb.atlassian.net/browse/MXS-472): Monitors update status in multiple steps + * [MXS-361](https://mariadb.atlassian.net/browse/MXS-361): crash on backend restart if persistent connections are in use + * [MXS-403](https://mariadb.atlassian.net/browse/MXS-403): Monitor callback to DCBs evades thread control causing crashes + * [MXS-392](https://mariadb.atlassian.net/browse/MXS-392): Update to "Rabbit MQ setup and MaxScale Integration" document + * [MXS-491](https://mariadb.atlassian.net/browse/MXS-491): MaxScale can time out systemd if startup of services takes too long + * [MXS-329](https://mariadb.atlassian.net/browse/MXS-329): The session pointer in a DCB can be null unexpectedly + * [MXS-479](https://mariadb.atlassian.net/browse/MXS-479): localtime must not be used in the multi-threaded program. + * [MXS-480](https://mariadb.atlassian.net/browse/MXS-480): Readwritesplit defaults cause connection pileup + * [MXS-464](https://mariadb.atlassian.net/browse/MXS-464): Upgrade 1.2.0 to 1.2.1 blocking start of `maxscale` service + * [MXS-365](https://mariadb.atlassian.net/browse/MXS-365): Load data local infile connection abort when loading certain files + * [MXS-431](https://mariadb.atlassian.net/browse/MXS-431): Backend authentication fails with schemarouter + * [MXS-394](https://mariadb.atlassian.net/browse/MXS-394): Faults in regex_replace function of regexfilter.c + * [MXS-379](https://mariadb.atlassian.net/browse/MXS-379): Incorrect handing of a GWBUF may cause SIGABRT. + * [MXS-321](https://mariadb.atlassian.net/browse/MXS-321): Incorrect number of connections in maxadmin list view + * [MXS-413](https://mariadb.atlassian.net/browse/MXS-413): MaxAdmin hangs with show session + * [MXS-408](https://mariadb.atlassian.net/browse/MXS-408): Connections to backend databases do not clear promptly + * [MXS-385](https://mariadb.atlassian.net/browse/MXS-385): disable_sescmd_history can cause false data to be read. + * [MXS-386](https://mariadb.atlassian.net/browse/MXS-386): max_sescmd_history should not close connections + * [MXS-373](https://mariadb.atlassian.net/browse/MXS-373): If config file is non-existent, maxscale crashes. + * [MXS-366](https://mariadb.atlassian.net/browse/MXS-366): Multi-source slave servers are not detected. + * [MXS-271](https://mariadb.atlassian.net/browse/MXS-271): Schemarouter and unknown databases + * [MXS-286](https://mariadb.atlassian.net/browse/MXS-286): Fix the content and format of MaxScale-HA-with-Corosync-Pacemaker document + * [MXS-274](https://mariadb.atlassian.net/browse/MXS-274): Memory Leak + * [MXS-254](https://mariadb.atlassian.net/browse/MXS-254): Failure to read configuration file results in no error log messages + * [MXS-251](https://mariadb.atlassian.net/browse/MXS-251): Non-thread safe strerror + * [MXS-291](https://mariadb.atlassian.net/browse/MXS-291): Random number generation has flaws + * [MXS-342](https://mariadb.atlassian.net/browse/MXS-342): When ini_parse fails to parse config file, no log messages are printed. + * [MXS-345](https://mariadb.atlassian.net/browse/MXS-345): maxscale.conf in /etc/init.d prevents puppet from starting maxscale + * [MXS-333](https://mariadb.atlassian.net/browse/MXS-333): use_sql_variables_in=master doesn't work + * [MXS-260](https://mariadb.atlassian.net/browse/MXS-260): Multiple MaxScale processes + * [MXS-184](https://mariadb.atlassian.net/browse/MXS-184): init script issues in CentOS 7 + * [MXS-280](https://mariadb.atlassian.net/browse/MXS-280): SELECT INTO OUTFILE query succeeds even if backed fails + * [MXS-202](https://mariadb.atlassian.net/browse/MXS-202): User password not handled correctly + * [MXS-282](https://mariadb.atlassian.net/browse/MXS-282): Add example to "Routing Hints" document + * [MXS-220](https://mariadb.atlassian.net/browse/MXS-220): LAST_INSERT_ID() query is redirect to slave if function call is in where clause + * [MXS-196](https://mariadb.atlassian.net/browse/MXS-196): DCB state is changed prior to polling operation + * [MXS-281](https://mariadb.atlassian.net/browse/MXS-281): SELECT INTO OUTFILE query goes several times to one slave + * [MXS-197](https://mariadb.atlassian.net/browse/MXS-197): Incorrect sequence of operations with DCB + * [MXS-195](https://mariadb.atlassian.net/browse/MXS-195): maxscaled.c ineffective DCB disposal + * [MXS-363](https://mariadb.atlassian.net/browse/MXS-363): rpm building seems to do something wrong with maxscale libraries + * [MXS-35](https://mariadb.atlassian.net/browse/MXS-35): bugzillaId-451: maxscale main() exit code is always 0 after it daemonizes + * [MXS-29](https://mariadb.atlassian.net/browse/MXS-29): bugzillaId-589: detect if MAXSCALE_SCHEMA.HEARTBEAT table is not replicated + * [MXS-436](https://mariadb.atlassian.net/browse/MXS-436): Invalid threads argument is ignored and MaxScale starts with one thread + * [MXS-427](https://mariadb.atlassian.net/browse/MXS-427): Logging a large string causes a segmentation fault + * [MXS-352](https://mariadb.atlassian.net/browse/MXS-352): With no backend connection, services aren't started + * [MXS-293](https://mariadb.atlassian.net/browse/MXS-293): Bug in init script, and maxscale --user=maxscale does run as root + * [MXS-210](https://mariadb.atlassian.net/browse/MXS-210): Check MaxScale user privileges + * [MXS-111](https://mariadb.atlassian.net/browse/MXS-111): maxscale binlog events shown in show services seems to be double-counted for the master connection + * [MXS-3](https://mariadb.atlassian.net/browse/MXS-3): Remove code for atomic_add in skygw_utils.cc + * [MXS-258](https://mariadb.atlassian.net/browse/MXS-258): ERR_error_string could overflow in future + * [MXS-310](https://mariadb.atlassian.net/browse/MXS-310): MaxScale 1.2 does not completely cleanly change to the maxscale user + * [MXS-450](https://mariadb.atlassian.net/browse/MXS-450): Syslog default prefix is MaxScale not maxscale + * [MXS-297](https://mariadb.atlassian.net/browse/MXS-297): postinstall on debian copies wrong file in /etc/init.d + +## Known Issues and Limitations + +There are a number bugs and known limitations within this version of MaxScale, +the most serious of this are listed below. + +* MaxScale can not manage authentication that uses wildcard matching in hostnames in the mysql.user table of the backend database. The only wildcards that can be used are in IP address entries. + +* When users have different passwords based on the host from which they connect MaxScale is unable to determine which password it should use to connect to the backend database. This results in failed connections and unusable usernames in MaxScale. + +* LONGBLOB are currently not supported. + +* Galera Cluster variables, such as @@wsrep_node_name, are not resolved by the embedded MariaDB parser. + +* The Database Firewall filter does not support multi-statements. Using them will result in an error being sent to the client. + +* The SSL support is known to be unstable. + +## Packaging + +RPM and Debian packages are provided for the Linux distributions supported +by MariaDB Enterprise. diff --git a/Documentation/Routers/Binlogrouter.md b/Documentation/Routers/Binlogrouter.md new file mode 100644 index 000000000..a1c2b1728 --- /dev/null +++ b/Documentation/Routers/Binlogrouter.md @@ -0,0 +1,104 @@ +# Binlogrouter + +The binlogrouter is a replication protocol proxy module for MaxScale. This module allows MaxScale to connect to a master server and retrieve binary logs while slave servers can connect to MaxScale like they would connect to a normal master server. If the master server goes down, the slave servers can still connect to MaxScale and read binary logs. You can switch to a new master server without the slaves noticing that the actual master server has changed. This allows for a more highly available replication setup where replication is high-priority. + +# Configuration + +## Mandatory Router Parameters + +The binlogrouter requires the `server`, `user` and `passwd` parameters. These should be configured according to the [Configuration Guide](../Getting-Started/Configuration-Guide.md#service). + +In addition to these two parameters, `router_options` needs to be defined. This is the main way the binlogrouter is configured and it will be covered in detail in the next section. + +## Router Options + +Binlogrouter is configured with a comma-separated list of key-value pairs. The following options should be given as a value to the `router_options` parameter. + +### `binlogdir` + +This parameter allows the location that MaxScale uses to store binlog files to be set. If this parameter is not set to a directory name then MaxScale will store the binlog files in the directory /var/cache/maxscale/. +In the binlog dir there is also the 'cache' directory that contains data retrieved from the master dureing registration phase and the master.ini file wich contains the configuration of current configured master. + +### `uuid` + +This is used to set the unique uuid that the binlog router uses when it connects to the master server. +If no explicit value is given for the uuid in the configuration file then a uuid will be generated. + +### `server-id` + +As with uuid, MaxScale must have a unique server-id for the connection it makes to the master, this parameter provides the value of server-id that MaxScale will use when connecting to the master. + +### `master-id` + +The server-id value that MaxScale should use to report to the slaves that connect to MaxScale. +This may either be the same as the server-id of the real master or can be chosen to be different if the slaves need to be aware of the proxy layer. +The real master server-id will be used if the option is not set. + +### `master_uuid` + +It is a requirement of replication that each slave have a unique UUID value. The MaxScale router will identify itself to the slaves using the uuid of the real master if this option is not set. + +### `master_version` + +The MaxScale router will identify itself to the slaves using the server version of the real master if this option is not set. + +### `master_hostname` + +The MaxScale router will identify itself to the slaves using the server hostname of the real master if this option is not set. + +### `user` + +This is the user name that MaxScale uses when it connects to the master. This user name must have the rights required for replication as with any other user that a slave uses for replication purposes. If the user parameter is not given in the router options then the same user as is used to retrieve the credential information will be used for the replication connection, i.e. the user in the service entry. + +The user that is used for replication, either defined using the user= option in the router options or using the username and password defined of the service must be granted replication privileges on the database server. + +``` + MariaDB> CREATE USER 'repl'@'maxscalehost' IDENTIFIED by 'password'; + MariaDB> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'maxscalehost'; +``` + +### `password` + +The password of the above user. If the password is not explicitly given then the password in the service entry will be used. For compatibility with other username and password definitions within the MaxScale configuration file it is also possible to use the parameter passwd=. + +### `heartbeat` + +This defines the value of the heartbeat interval in seconds for the connection to the master. MaxScale requests the master to ensure that a binlog event is sent at least every heartbeat period. If there are no real binlog events to send the master will sent a special heartbeat event. The default value for the heartbeat period is every 5 minutes. The current interval value is reported in the diagnostic output. + +### `burstsize` + +This parameter is used to define the maximum amount of data that will be sent to a slave by MaxScale when that slave is lagging behind the master. In this situation the slave is said to be in "catchup mode", this parameter is designed to both prevent flooding of that slave and also to prevent threads within MaxScale spending disproportionate amounts of time with slaves that are lagging behind the master. The burst size can be defined in Kb, Mb or Gb by adding the qualifier K, M or G to the number given. The default value of burstsize is 1Mb and will be used if burstsize is not given in the router options. + +### `mariadb10-compatibility` + +This parameter allows binlogrouter to replicate from a MariaDB 10.0 master server. GTID will not be used in the replication. + +``` +# Example +router_options=mariadb10-compatibility=1 +``` + +### `transaction_safety` + +This parameter is used to enable/disable incomplete transactions detection in binlog router. +When MaxScale starts an error message may appear if current binlog file is corrupted or an incomplete transaction is found. +During normal operations binlog events are not distributed to the slaves until a COMMIT is seen. +The default value is off, set transaction_safety=on to enable the incomplete transactions detection. + +### `send_slave_heartbeat` + +This defines whether (on | off) MaxSclale sends to the slave the heartbeat packet when there are no real binlog events to send. The default value if 'off', no heartbeat event is sent to slave server. If value is 'on' the interval value (requested by the slave during registration) is reported in the diagnostic output and the packect is send after the time interval without any event to send. + +A complete example of a service entry for a binlog router service would be as follows. +``` + [Replication] + type=service + router=binlogrouter + servers=masterdb + version_string=5.6.17-log + user=maxscale + passwd=Mhu87p2D + router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,filestem=mybin,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1,send_slave_heartbeat=1 +``` + +The minimum set of router options that must be given in the configuration are are server-id and master-id, default values may be used for all other options. diff --git a/Documentation/Routers/CLI.md b/Documentation/Routers/CLI.md new file mode 100644 index 000000000..83a31f46c --- /dev/null +++ b/Documentation/Routers/CLI.md @@ -0,0 +1,24 @@ +# CLI + +The command line interface as used by `maxadmin`. This is a variant of the debugcli that is built slightly differently so that it may be accessed by the client application `maxadmin`. The CLI requires the use of the `maxscaled` protocol. + +## Configuration + +There are two components to the definition required in order to run the command line interface to use with MaxAdmin; a service and a listener. + +The default entries required are shown below. + +``` +[CLI] +type=service +router=cli + +[CLI Listener] +type=listener +service=CLI +protocol=maxscaled +address=localhost +port=6603 +``` + +Note that this uses the default port of 6603 and confines the connections to localhost connections only. Remove the address= entry to allow connections from any machine on your network. Changing the port from 6603 will mean that you must allows pass a -p option to the MaxAdmin command. diff --git a/Documentation/Routers/Debug-CLI.md b/Documentation/Routers/Debug-CLI.md new file mode 100644 index 000000000..0e227e8a5 --- /dev/null +++ b/Documentation/Routers/Debug-CLI.md @@ -0,0 +1,55 @@ +# Debug CLI + +The **debugcli** router is a special kind of statement based router. Rather than direct the statements at an external data source they are handled internally. These statements are simple text commands and the results are the output of debug commands within MaxScale. The service and listener definitions for a debug cli service only differ from other services in that they require no backend server definitions. + +## Configuration + +The definition of the debug cli service is illustrated below + +``` +[Debug Service] +type=service +router=debugcli + +[Debug Listener] +type=listener +service=Debug Service +protocol=telnetd +port=4442 +``` + +Connections using the telnet protocol to port 4442 of the MaxScale host will result in a new debug CLI session. A default username and password are used for this module, new users may be created using the add user command. As soon as any users are explicitly created the default username will no longer continue to work. The default username is admin with a password of mariadb. + +The debugcli supports two modes of operation, `developer` and `user`. The mode is set via the `router_options` parameter. The user mode is more suited to end-users and administrators, whilst the develop mode is explicitly targeted to software developing adding or maintaining the MaxScale code base. Details of the differences between the modes can be found in the debugging guide for MaxScale. The default is `user` mode. The following service definition would enable a developer version of the debugcli. + +``` +[Debug Service] +type=service +router=debugcli +router_options=developer +``` + +It should be noted that both `user` and `developer` instances of debugcli may be defined within the same instance of MaxScale, however they must be defined as two distinct services, each with a distinct listener. + +``` +[Debug Service] +type=service +router=debugcli +router_options=developer + +[Debug Listener] +type=listener +service=Debug Service +protocol=telnetd +port=4442 + +[Admin Service] +type=service +router=debugcli + +[Admin Listener] +type=listener +service=Debug Service +protocol=telnetd +port=4242 +``` diff --git a/Documentation/Routers/ReadWriteSplit.md b/Documentation/Routers/ReadWriteSplit.md index 18a7aa3d4..9392422ef 100644 --- a/Documentation/Routers/ReadWriteSplit.md +++ b/Documentation/Routers/ReadWriteSplit.md @@ -205,3 +205,50 @@ Most imaginable reasons are related to replication lag but it could be possible ## Examples Examples of the readwritesplit router in use can be found in the [Tutorials](../Tutorials) folder. + +## Readwritesplit routing decisions + +Here is a small explanation which shows what kinds of queries are routed to which type of server. + +### Routing to Master + +Routing to master is important for data consistency and because majority of writes are written to binlog and thus become replicated to slaves. + +The following operations are routed to master: + +* write statements, +* all statements within an open transaction, +* stored procedure calls, and +* user-defined function calls. +* DDL statements (`DROP`|`CREATE`|`ALTER TABLE` … etc.) +* `EXECUTE` (prepared) statements +* all statements using temporary tables + +In addition to these, if the **readwritesplit** service is configured with the `max_slave_replication_lag` parameter, and if all slaves suffer from too much replication lag, then statements will be routed to the _Master_. (There might be other similar configuration parameters in the future which limit the number of statements that will be routed to slaves.) + +### Routing to Slaves + +The ability to route some statements to *Slave*s is important because it also decreases the load targeted to master. Moreover, it is possible to have multiple slaves to share the load in contrast to single master. + +Queries which can be routed to slaves must be auto committed and belong to one of the following group: + +* read-only database queries, +* read-only queries to system, or user-defined variables, +* `SHOW` statements, and +* system function calls. + +### Routing to every session backend + +A third class of statements includes those which modify session data, such as session system variables, user-defined variables, the default database, etc. We call them session commands, and they must be replicated as they affect the future results of read and write operations, so they must be executed on all servers that could execute statements on behalf of this client. + +Session commands include for example: + +* `SET` statements +* `USE `*``* +* system/user-defined variable assignments embedded in read-only statements, such as `SELECT (@myvar := 5)` +* `PREPARE` statements +* `QUIT`, `PING`, `STMT RESET`, `CHANGE USER`, etc. commands + +**NOTE: if variable assignment is embedded in a write statement it is routed to _Master_ only. For example, `INSERT INTO t1 values(@myvar:=5, 7)` would be routed to _Master_ only.** + +The router stores all of the executed session commands so that in case of a slave failure, a replacement slave can be chosen and the session command history can be repeated on that new slave. This means that the router stores each executed session command for the duration of the session. Applications that use long-running sessions might cause MaxScale to consume a growing amount of memory unless the sessions are closed. This can be solved by setting a connection timeout on the application side. diff --git a/Documentation/Tutorials/Galera-Cluster-Connection-Routing-Tutorial.md b/Documentation/Tutorials/Galera-Cluster-Connection-Routing-Tutorial.md index 0ebe57a2b..613903ecb 100644 --- a/Documentation/Tutorials/Galera-Cluster-Connection-Routing-Tutorial.md +++ b/Documentation/Tutorials/Galera-Cluster-Connection-Routing-Tutorial.md @@ -1,64 +1,16 @@ # Connection Routing with Galera Cluster -## Environment & Solution Space +# Environment & Solution Space -This document is designed as a quick introduction to setting up MaxScale in an environment in which you have a Galera Cluster within which you wish to balance connection across all the database nodes of the cluster that are active members of cluster. +The object of this tutorial is to have a system that has two ports available, one for write connections to the database cluster and the other for read connections to the database. -The process of setting and configuring MaxScale will be covered within this document. However the installation and configuration of the Galera Cluster will not be covered. +## Setting up MaxScale -This tutorial will assume the user is running from one of the binary distributions available and has installed this in the default location. Building from source code in GitHub is covered in guides elsewhere as is installing to non-default locations. +The first part of this tutorial is covered in [MaxScale Tutorial](MaxScale-Tutorial.md). Please read it and follow the instructions for setting up MaxScale with the type of cluster you want to use. -## Process +Once you have MaxScale installed and the database users created, we can create the configuration file for MaxScale. -The steps involved in creating a system from the binary distribution of MaxScale are: - -* Install the package relevant to your distribution - -* Create the required users in your MariaDB or MySQL Galera cluster - -* Create a MaxScale configuration file - -### Installation - -The precise installation process will vary from one distribution to another details of what to do with the RPM and DEB packages can be found on the download site when you select the distribution you are downloading from. The process involves setting up your package manager to include the MariaDB repositories and then running the package manager for your distribution, RPM or apt-get. - -Upon successful completion of the installation command you will have MaxScale installed and ready to be run but without a configuration. You must create a configuration file before you first run MaxScale. - -### Creating Database Users - -MaxScale needs to connect to the backend databases and run queries for two reasons; one to determine the current state of the database and the other to retrieve the user information for the database cluster. This may be done either using two separate usernames or with a single user. - -The first user required must be able to select data from the table mysql.user, to create this user follow the steps below. - -1. Connect to one of the nodes in your Galera cluster as the root user - -2. Create the user, substituting the username, password and host on which maxscale runs within your environment - -MariaDB [(none)]> create user '*username*'@'*maxscalehost*' identified by '*password*'; - -**Query OK, 0 rows affected (0.00 sec)** - -3. Grant select privileges on the mysql.user table - -MariaDB [(none)]> grant SELECT on mysql.user to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.03 sec)** - -Additionally, GRANT SELECT on the mysql.db table and SHOW DATABASES privileges are required in order to load databases name and grants suitable for database name authorization. - -MariaDB [(none)]> GRANT SELECT ON mysql.db TO 'username'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** - -MariaDB [(none)]> GRANT SHOW DATABASES ON *.* TO 'username'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** - -The second user is used to monitored the state of the cluster. This user, which may be the same username as the first, requires permissions to access the various sources of monitoring data within the information schema. No special permission need to be granted to the user in order to query the information schema. - -If you wish to use two different usernames for the two different roles of monitoring and collecting user information then create a different username using the first two steps from above. - -### Creating Your MaxScale Configuration +## Creating Your MaxScale Configuration MaxScale configuration is held in an ini file that is located in the file maxscale.cnf in the directory /etc, if you have installed in the default location then this file is available in /etc/maxscale.cnf. This is not created as part of the installation process and must be manually created. A template file does exist within the /usr/share/maxscale directory that may be use as a basis for your configuration. diff --git a/Documentation/Tutorials/Galera-Cluster-Read-Write-Splitting-Tutorial.md b/Documentation/Tutorials/Galera-Cluster-Read-Write-Splitting-Tutorial.md index d0f0b6594..17902554f 100644 --- a/Documentation/Tutorials/Galera-Cluster-Read-Write-Splitting-Tutorial.md +++ b/Documentation/Tutorials/Galera-Cluster-Read-Write-Splitting-Tutorial.md @@ -2,65 +2,15 @@ ## Environment & Solution Space -This document is designed as a quick introduction to setting up MaxScale in an environment in which you have a Galera Cluster which you wish to use as a single database node for update and one or more read only nodes. The object of this tutorial is to have a system that appears to the clients of MaxScale as if there is a single database behind MaxScale. MaxScale will split the statements such that write statements will be sent to only one server in the cluster and read statements will be balanced across the remainder of the servers. +The object of this tutorial is to have a system that appears to the clients of MaxScale as if there is a single database behind MaxScale. MaxScale will split the statements such that write statements will be sent to the current write-master server in the Galera cluster and read statements will be balanced across the rest of the nodes. -The reason for a configuration like this, with all the updates being directed to a single node within what is a multi-master cluster, is to prevent any possible conflict between updates that may run on multiple nodes. Galera is built to provide the mechanism for this situation, however issues have been known to occur when conflicting transactions are committed on multiple nodes. Some applications are unable to deal with the resulting errors that may be created in this situation. +## Setting up MaxScale -The process of setting and configuring MaxScale will be covered within this document. However the installation and configuration of the Galera Cluster will not be covered in this tutorial. +The first part of this tutorial is covered in [MaxScale Tutorial](MaxScale-Tutorial.md). Please read it and follow the instructions for setting up MaxScale with the type of cluster you want to use. -This tutorial will assume the user is running from one of the binary distributions available and has installed this in the default location. Building from source code in GitHub is covered in guides elsewhere as is installing to non-default locations. +Once you have MaxScale installed and the database users created, we can create the configuration file for MaxScale. -## Process - -The steps involved in creating a system from the binary distribution of MaxScale are: - -* Install the package relevant to your distribution - -* Create the required users in your Galera Cluster - -* Create a MaxScale configuration file - -### Installation - -The precise installation process will vary from one distribution to another details of what to do with the RPM and DEB packages can be found on the download site when you select the distribution you are downloading from. The process involves setting up your package manager to include the MariaDB repositories and then running the package manager for your distribution, RPM or apt-get. - -Upon successful completion of the installation command you will have MaxScale installed and ready to be run but without a configuration. You must create a configuration file before you first run MaxScale. - -### Creating Database Users - -MaxScale needs to connect to the backend databases and run queries for two reasons; one to determine the current state of the database and the other to retrieve the user information for the database cluster. This may be done either using two separate usernames or with a single user. - -The first user required must be able to select data from the table mysql.user, to create this user follow the steps below. - -1. Connect to Galera Cluster as the root user - -2. Create the user, substituting the username, password and host on which maxscale runs within your environment - -MariaDB [(none)]> create user '*username*'@'*maxscalehost*' identified by '*password*'; - -**Query OK, 0 rows affected (0.00 sec)** - -3. Grant select privileges on the mysql.user table. - -MariaDB [(none)]> grant SELECT on mysql.user to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.03 sec)** - -Additionally, GRANT SELECT on the mysql.db table and SHOW DATABASES privileges are required in order to load databases name and grants suitable for database name authorization. - -MariaDB [(none)]> GRANT SELECT ON mysql.db TO 'username'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** - -MariaDB [(none)]> GRANT SHOW DATABASES ON *.* TO 'username'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** - -The second user is used to monitored the state of the cluster. This user, which may be the same username as the first, requires permissions to access the various sources of monitoring data within the information schema. No special permission need to be granted to the user in order to query the information schema. - -If you wish to use two different usernames for the two different roles of monitoring and collecting user information then create a different username using the first two steps from above. - -### Creating Your MaxScale Configuration +## Creating Your MaxScale Configuration MaxScale configuration is held in an ini file that is located in the file maxscale.cnf in the directory /etc, if you have installed in the default location then this file is available in /etc/maxscale.cnf. This is not created as part of the installation process and must be manually created. A template file does exist within the /usr/share/maxscale directory that may be use as a basis for your configuration. diff --git a/Documentation/Tutorials/MaxScale-Tutorial.md b/Documentation/Tutorials/MaxScale-Tutorial.md new file mode 100644 index 000000000..1e7f37590 --- /dev/null +++ b/Documentation/Tutorials/MaxScale-Tutorial.md @@ -0,0 +1,111 @@ +# Setting up MaxScale + +This document is designed as a quick introduction to setting up MaxScale in an environment in which you have either a MySQL Master-Slave replication cluster with one master and multiple slave servers or a multi-node Galera cluster. The process of setting and configuring MaxScale will be covered within this document. + +The installation and configuration of the MySQL Replication or the Galera cluster will not be covered nor will any discussion of installation management tools to handle automated or semi-automated failover of the replication cluster. The [Setting Up Replication](https://mariadb.com/kb/en/mariadb/setting-up-replication/) article on the MariaDB knowledgebase can help you get started with replication clusters and the [Getting Started With Mariadb Galera Cluster](https://mariadb.com/kb/en/mariadb/getting-started-with-mariadb-galera-cluster/) article will help you set up a Galera cluster. + +This tutorial will assume the user is running from one of the binary distributions available and has installed this in the default location. Building from source code in GitHub is covered in the [Building from Source](../Getting-Started/Building-MaxScale-from-Source-Code.md) document. + +## Process + +The steps involved in setting up MaxScale are: + +* Install the package relevant to your distribution + +* Create the required users in your MariaDB or MySQL Replication cluster + +* Create a MaxScale configuration file + +## Installation + +The precise installation process will vary from one distribution to another details of what to do with the RPM and DEB packages can be found on the download site when you select the distribution you are downloading from. The process involves setting up your package manager to include the MariaDB repositories and then running the package manager for your distribution (usually yum or apt-get). + +Upon successful completion of the installation command you will have MaxScale installed and ready to be run but without a configuration. You must create a configuration file before you first run MaxScale which is covered in a later section. + +## Creating Database Users + +MaxScale needs to connect to the backend databases and run queries for two reasons; one to determine the current state of the database and the other to retrieve the user information for the database cluster. The first pair of credentials will be used by the monitor modules and the second is used by MaxScale itself. This may be done either using two separate usernames or with a single user. + +The first user required must be able to select data from the table mysql.user, to create this user follow the steps below. + +1. Connect to the current master server in your replication tree as the root user + +2. Create the user, substituting the username, password and host on which maxscale runs within your environment +``` +MariaDB [(none)]> create user '*username*'@'*maxscalehost*' identified by '*password*'; + +**Query OK, 0 rows affected (0.00 sec)** +``` +3. Grant select privileges on the mysql.user table. +``` +MariaDB [(none)]> grant SELECT on mysql.user to '*username*'@'*maxscalehost*'; + +**Query OK, 0 rows affected (0.03 sec)** +``` +Additionally, GRANT SELECT on the mysql.db table and SHOW DATABASES privileges are required in order to load databases name and grants suitable for database name authorization. +``` +MariaDB [(none)]> GRANT SELECT ON mysql.db TO 'username'@'maxscalehost'; + +**Query OK, 0 rows affected (0.00 sec)** + +MariaDB [(none)]> GRANT SHOW DATABASES ON *.* TO 'username'@'maxscalehost'; + +**Query OK, 0 rows affected (0.00 sec)** +``` +The second user is used to monitored the state of the cluster. This user, which may be the same username as the first, requires permissions to access the various sources of monitoring data. In order to monitor a replication cluster this user must be granted the roles REPLICATION SLAVE and REPLICATION CLIENT +``` +MariaDB [(none)]> grant REPLICATION SLAVE on *.* to '*username*'@'*maxscalehost*'; + +**Query OK, 0 rows affected (0.00 sec)** + +MariaDB [(none)]> grant REPLICATION CLIENT on *.* to '*username*'@'*maxscalehost*'; + +**Query OK, 0 rows affected (0.00 sec)** +``` +If you wish to use two different usernames for the two different roles of monitoring and collecting user information then create a different username using the first two steps from above. + +## Creating additional grants for users + +Because MaxScale is a proxy the backend databases will see all clients as if they were connecting from MaxScale's address. This usually requires users to create additional grants for MaxScale's hostname. The best way to describe this process is with an example. + +User `'jdoe'@'192.168.0.200` has the following grant on the cluster: `GRANT SELECT, INSERT, UPDATE, DELETE ON *.* TO 'jdoe'@'192.168.0.200'`. When the user connects directly to the server it will see it as `'jdoe'@'192.168.0.200` connecting to the server and it will match the grant for `'jdoe'@'192.168.0.200`. + +If MaxScale is at the address `192.168.0.101` and the user `jdoe` connects to this MaxScale, the backend server will see the connection as `'jdoe'@'192.168.0.101'`. Since the backend server has no grants for `'jdoe'@'192.168.0.101'`, the connection from MaxScale to the server will be refused. + +We can fix this by either creating a matching grant for user `jdoe` from the MaxScale address or by using a wildcard to cover both addresses. + +The quickest way to do this is by doing a SHOW GRANTS query: +``` +MariaDB [(none)]> SHOW GRANTS FOR 'jdoe'@'192.168.0.200'; ++-----------------------------------------------------------------------+ +| Grants for jdoe@192.168.0.200 | ++-----------------------------------------------------------------------+ +| GRANT SELECT, INSERT, UPDATE, DELETE ON *.* TO 'jdoe'@'192.168.0.200' | ++-----------------------------------------------------------------------+ +1 row in set (0.01 sec) +``` +Then creating the user `'jdoe'@'192.168.0.101'` and giving it the same grants: +``` +MariaDB [(none)]> CREATE USER 'jdoe'@'192.168.0.101'; +Query OK, 0 rows affected (0.00 sec) + +MariaDB [(none)]> GRANT SELECT, INSERT, UPDATE, DELETE ON *.* TO 'jdoe'@'192.168.0.101'; +Query OK, 0 rows affected (0.00 sec) +``` + +The other option is to use a wildcard grant like `GRANT SELECT, INSERT, UPDATE, DELETE ON *.* TO 'jdoe'@'%'`. This is more convenient but also less secure than having specific grants for both the client's address and MaxScale's address. + + +## Creating the configuration file + +The configuration file creation is covered in different tutorials. + +### Master-Slave cluster + +* [MySQL Replication Connection Routing Tutorial](MySQL-Replication-Connection-Routing-Tutorial.md) +* [MySQL Replication Read-Write Splitting Tutorial](MySQL-Replication-Read-Write-Splitting-Tutorial.md) + +### Galera cluster + +* [Galera Cluster Connection Routing Tutorial](Galera-Cluster-Connection-Routing-Tutorial.md) +* [Galera Cluster Read Write Splitting Tutorial](Galera-Cluster-Read-Write-Splitting-Tutorial.md) diff --git a/Documentation/Tutorials/MySQL-Replication-Connection-Routing-Tutorial.md b/Documentation/Tutorials/MySQL-Replication-Connection-Routing-Tutorial.md index 0140b7e00..276644551 100644 --- a/Documentation/Tutorials/MySQL-Replication-Connection-Routing-Tutorial.md +++ b/Documentation/Tutorials/MySQL-Replication-Connection-Routing-Tutorial.md @@ -1,78 +1,14 @@ -Getting Started With MariaDB MaxScale - -Connection Routing with MySQL Replication +# Connection Routing with Galera Cluster # Environment & Solution Space -This document is designed as a quick introduction to setting up MaxScale in an environment in which you have a MySQL Replication Cluster with one master and multiple slave servers. The object of this tutorial is to have a system that has two ports available, one for write connections to the database cluster and the other for read connections to the database. +The object of this tutorial is to have a system that has two ports available, one for write connections to the database cluster and the other for read connections to the database. -The process of setting and configuring MaxScale will be covered within this document. However the installation and configuration of the MySQL Replication subsystem will not be covered nor will any discussion of installation management tools to handle automated or semi-automated failover of the replication cluster. +## Setting up MaxScale -This tutorial will assume the user is running from one of the binary distributions available and has installed this in the default location. Building from source code in GitHub is covered in guides elsewhere as is installing to non-default locations. +The first part of this tutorial is covered in [MaxScale Tutorial](MaxScale-Tutorial.md). Please read it and follow the instructions for setting up MaxScale with the type of cluster you want to use. -# Process - -The steps involved in creating a system from the binary distribution of MaxScale are: - -* Install the package relevant to your distribution - -* Create the required users in your MariaDB or MySQL Replication cluster - -* Create a MaxScale configuration file - -## Installation - -The precise installation process will vary from one distribution to another details of what to do with the RPM and DEB packages can be found on the download site when you select the distribution you are downloading from. The process involves setting up your package manager to include the MariaDB repositories and then running the package manager for your distribution, RPM or apt-get. - -Upon successful completion of the installation command you will have MaxScale installed and ready to be run but without a configuration. You must create a configuration file before you first run MaxScale. - -## Creating Database Users - -MaxScale needs to connect to the backend databases and run queries for two reasons; one to determine the current state of the database and the other to retrieve the user information for the database cluster. This may be done either using two separate usernames or with a single user. - -The first user required must be able to select data from the table mysql.user, to create this user follow the steps below. - -1. Connect to the current master server in your replication tree as the root user - -2. Create the user, substituting the username, password and host on which maxscale runs within your environment - -``` -MariaDB [(none)]> create user '*username*'@'*maxscalehost*' identified by '*password*'; - -**Query OK, 0 rows affected (0.00 sec)** - -3. Grant select privileges on the mysql.user table - -MariaDB [(none)]> grant SELECT on mysql.user to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.03 sec)** -``` - -Additionally, GRANT SELECT on the mysql.db table and SHOW DATABASES privileges are required in order to load databases name and grants suitable for database name authorization. - -``` -MariaDB [(none)]> GRANT SELECT ON mysql.db TO '*username*'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** - -MariaDB [(none)]> GRANT SHOW DATABASES ON *.* TO '*username*'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** -``` - -The second user is used to monitored the state of the cluster. This user, which may be the same username as the first, requires permissions to access the various sources of monitoring data. In order to monitor a replication cluster this user must be granted the roles REPLICATION SLAVE and REPLICATION CLIENT - -``` -MariaDB [(none)]> grant REPLICATION SLAVE on *.* to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.00 sec)** - -MariaDB [(none)]> grant REPLICATION CLIENT on *.* to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.00 sec)** -``` - -If you wish to use two different usernames for the two different roles of monitoring and collecting user information then create a different username using the first two steps from above. +Once you have MaxScale installed and the database users created, we can create the configuration file for MaxScale. ## Creating Your MaxScale Configuration diff --git a/Documentation/Tutorials/MySQL-Replication-Read-Write-Splitting-Tutorial.md b/Documentation/Tutorials/MySQL-Replication-Read-Write-Splitting-Tutorial.md index 572d9482b..20fb38913 100644 --- a/Documentation/Tutorials/MySQL-Replication-Read-Write-Splitting-Tutorial.md +++ b/Documentation/Tutorials/MySQL-Replication-Read-Write-Splitting-Tutorial.md @@ -1,80 +1,14 @@ -Getting Started With MariaDB MaxScale +# Read/Write Splitting with MySQL Replication -Read/Write Splitting with MySQL Replication +## Environment & Solution Space -# Environment & Solution Space +The object of this tutorial is to have a system that appears to the clients of MaxScale as if there is a single database behind MaxScale. MaxScale will split the statements such that write statements will be sent to the current master server in the replication cluster and read statements will be balanced across the rest of the slave servers. -This document is designed as a quick introduction to setting up MaxScale in an environment in which you have a MySQL Replication Cluster with one master and multiple slave servers. The object of this tutorial is to have a system that appears to the clients of MaxScale as if there is a single database behind MaxScale. MaxScale will split the statements such that write statements will be sent to the current master server in the replication cluster and read statements will be balanced across a number of the slave statements. +## Setting up MaxScale -The process of setting and configuring MaxScale will be covered within this document. However the installation and configuration of the MySQL Replication subsystem will not be covered nor will any discussion of installation management tools to handle automated or semi-automated failover of the replication cluster. +The first part of this tutorial is covered in [MaxScale Tutorial](MaxScale-Tutorial.md). Please read it and follow the instructions for setting up MaxScale with the type of cluster you want to use. -This tutorial will assume the user is running from one of the binary distributions available and has installed this in the default location. Building from source code in GitHub is covered in guides elsewhere as is installing to non-default locations. - -# Process - -The steps involved in creating a system from the binary distribution of MaxScale are: - -* Install the package relevant to your distribution - -* Create the required users in your MariaDB or MySQL Replication cluster - -* Create a MaxScale configuration file - -## Installation - -The precise installation process will vary from one distribution to another details of what to do with the RPM and DEB packages can be found on the download site when you select the distribution you are downloading from. The process involves setting up your package manager to include the MariaDB repositories and then running the package manager for your distribution, RPM or apt-get. - -Upon successful completion of the installation command you will have MaxScale installed and ready to be run but without a configuration. You must create a configuration file before you first run MaxScale. - -## Creating Database Users - -MaxScale needs to connect to the backend databases and run queries for two reasons; one to determine the current state of the database and the other to retrieve the user information for the database cluster. This may be done either using two separate usernames or with a single user. - -The first user required must be able to select data from the table mysql.user, to create this user follow the steps below. - -1. Connect to the current master server in your replication tree as the root user - -2. Create the user, substituting the username, password and host on which maxscale runs within your environment - -``` -MariaDB [(none)]> create user '*username*'@'*maxscalehost*' identified by '*password*'; - -**Query OK, 0 rows affected (0.00 sec)** - -``` -3. Grant select privileges on the mysql.user table. - -``` -MariaDB [(none)]> grant SELECT on mysql.user to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.03 sec)** - -``` -Additionally, GRANT SELECT on the mysql.db table and SHOW DATABASES privileges are required in order to load databases name and grants suitable for database name authorization. - -``` -MariaDB [(none)]> GRANT SELECT ON mysql.db TO 'username'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** - -MariaDB [(none)]> GRANT SHOW DATABASES ON *.* TO 'username'@'maxscalehost'; - -**Query OK, 0 rows affected (0.00 sec)** - -``` -The second user is used to monitored the state of the cluster. This user, which may be the same username as the first, requires permissions to access the various sources of monitoring data. In order to monitor a replication cluster this user must be granted the roles REPLICATION SLAVE and REPLICATION CLIENT - -``` -MariaDB [(none)]> grant REPLICATION SLAVE on *.* to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.00 sec)** - -MariaDB [(none)]> grant REPLICATION CLIENT on *.* to '*username*'@'*maxscalehost*'; - -**Query OK, 0 rows affected (0.00 sec)** - -``` -If you wish to use two different usernames for the two different roles of monitoring and collecting user information then create a different username using the first two steps from above. +Once you have MaxScale installed and the database users created, we can create the configuration file for MaxScale. ## Creating Your MaxScale Configuration @@ -267,5 +201,5 @@ CLI | maxscaled | localhost | 6603 | Running ``` -MaxScale is now ready to start accepting client connections and routing them to the master or slaves within your cluster. Other configuration options are available that can alter the criteria used for routing, these include monitoring the replication lag within the cluster and routing only to slaves that are within a predetermined delay from the current master or using weights to obtain unequal balancing operations. These options may be found in the MaxScale Configuration Guide. More detail on the use of maxadmin can be found in the document "MaxAdmin - The MaxScale Administration & Monitoring Client Application". +MaxScale is now ready to start accepting client connections and routing them to the master or slaves within your cluster. Other configuration options are available that can alter the criteria used for routing, these include monitoring the replication lag within the cluster and routing only to slaves that are within a predetermined delay from the current master or using weights to obtain unequal balancing operations. These options may be found in the MaxScale Configuration Guide. More detail on the use of maxadmin can be found in the document [MaxAdmin - The MaxScale Administration & Monitoring Client Application](Administration-Tutorial.md). diff --git a/Documentation/Tutorials/Nagios-Plugins.md b/Documentation/Tutorials/Nagios-Plugins.md index bf91baaf3..7e62ea767 100644 --- a/Documentation/Tutorials/Nagios-Plugins.md +++ b/Documentation/Tutorials/Nagios-Plugins.md @@ -64,7 +64,7 @@ Assuming Nagios installed on a separated server and the plugins are in /usr/lib6 * Copy New commands and server1 definition (./nagios/plugins/maxscale_commands.cfg, server1.cfg) to /etc/nagios/objects/ on Nagios Server * Edit /etc/nagios/nagios.cfg on Nagios Server -and add (just after localhost.cfg or commnads.cfg) +and add (just after localhost.cfg or commands.cfg) ``` cfg_file=/etc/nagios/objects/maxscale_commands.cfg diff --git a/Documentation/Tutorials/RabbitMQ-And-Tee-Archiving.md b/Documentation/Tutorials/RabbitMQ-And-Tee-Archiving.md new file mode 100644 index 000000000..c4a478e37 --- /dev/null +++ b/Documentation/Tutorials/RabbitMQ-And-Tee-Archiving.md @@ -0,0 +1,377 @@ +# Data archiving with Mqfilter and Tee filters + +This tutorial gives a quick look into how you can combine various filters to create +systems for archiving data for analysis. The aim of this tutorial is to show +what can be done with MaxScale's filters rather than demonstrate a proven method +of archiving data. For this tutorial you will need two MariaDB/MySQL servers, one for +archiving the data and one for actual use, a RabbitMQ server and a MaxScale server. +For testing purposes some of these can locate on the same server but for actual +use, an HA solution is recommended. + +The use case for this tutorial is a production system with one main server where all +queries are routed and an archive server where only INSERT, UPDATE and DELETE statements +are routed. The queries routed to the archive servers are also transformed into a canonical +format and sent to a RabbitMQ broker for analysis. This setup allows us to control what we +send to the server and could possibly allow us to filter out DELETE statements completely, +making the archive server a true archive of all data. + +## Setting up MaxScale + +The installation of MaxScale is covered in the Installation chapter of the [MaxScale Tutorial](MaxScale-Tutorial.md). + +## Setting up the MariaDB/MySQL servers + +Since the archive server will not replicate from the main server, we don't need to +set up replication between the two. The only thing we need to do is to create the +users we will use for monitoring and authentication. + +The process of creating monitoring and authentication users for MaxScale is described +in the Creating Database Users section of the [MaxScale Tutorial](MaxScale-Tutorial.md). + +## Setting up RabbitMQ server + +To set up the RabbitMQ server, follow the instructions for your OS onthe [RabbitMQ website](https://www.rabbitmq.com/download.html). +Useful documentation about access rights can be found on the [Access Control](https://www.rabbitmq.com/access-control.html) +page and for UNIX systems the [`rabbitmqctl` manpage](https://www.rabbitmq.com/man/rabbitmqctl.1.man.html) +has all the needed commands to manage your installation of RabbitMQ. + +For this tutorial, we will use a RabbitMQ server installed on a CentOS 7 from +the RPM packages. Since CentOS 7 doesn't have the RabbitMQ server in the default +repositores, we will need two extra repositories: The EPEL repository and the Erlang repository. + +* [EPEL repositories](https://fedoraproject.org/wiki/EPEL) +* [Erlang repositories](https://www.erlang-solutions.com/resources/download.html) + +First we need the EPEL and Erlang repositories for CentOS 7 after which we will install the RabbitMQ server: + +``` +sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm +sudo yum -y install http://packages.erlang-solutions.com/erlang-solutions-1.0-1.noarch.rpm +sudo yum -y install https://www.rabbitmq.com/releases/rabbitmq-server/v3.5.6/rabbitmq-server-3.5.6-1.noarch.rpm +``` + +Next step is to start the RabbitMQ server and configure it. + +``` +sudo systemctl start rabbitmq-server +``` + +We need to create the `msguser` account which we will use to relay the messages +to the broker. This can be done with the `rabbitmqctl` command. + +``` +sudo rabbitmqctl add_user "msguser" "msgpwd" +sudo rabbitmqctl set_permissions "msguser" ".*" ".*" ".*" +``` + +This creates the `msguser` account with `msgpwd` as the password and grants read, write and configurations +permissions on the default virtual host `/`. + +## Configuration file creation + +The final step before testing is the creation of the maxscale.cnf file. We start off by adding the two +servers to the configuration file. + +``` +[production-1] +type=server +address=192.168.0.200 +port=3306 +protocol=MySQLBackend + +[archive-1] +type=server +address=192.168.0.201 +port=3000 +protocol=MySQLBackend +``` + +After we have defined the `production-1` and `archive-1` servers, we need a monitor +module for those servers. This module will detect if connectivity to the servers +is lost and notify MaxScale of the changed server states. + +``` +[MySQL Monitor] +type=monitor +module=mysqlmon +servers=production-1, archive-1 +user=maxuser +passwd=maxpwd +monitor_interval=5000 +``` + +The monitor will use the user `maxuser` with the password `maxpwd` to connect to +the servers and query them for their state. In the `servers` parameter we have +listed both of the `production-1` and `archive-1` servers. All objects in the +MaxScale configuration file are referred by their section names. Here the section +names of the servers are used in the `servers` parameter. The `monitor_interval` +parameter controls how often the monitor will poll the servers for status. For +this tutorial, we've set it to 5000 milliseconds. + +Next we configure the Production and Archive services. The Production service will +be the main access point to the system. + +``` +[Production] +type=service +router=readconnroute +servers=production-1 +user=maxuser +passwd=maxpwd +filters=Tee + +[Archive] +type=service +router=readconnroute +servers=archive-1 +user=maxuser +passwd=maxpwd +filters=MQ Filter +``` + +The `filters` parameters for the services refer to the filters we will be creating next. +The Production service will use the Tee filter to duplicate INSERT, UPDATE and DELETE +statements to the Archive service. The statements passed to the Archive service will +use the MQ Filter to send the canonic versions of the statements to the RabbitMQ broker. +The Production service will use the `production-1` server and the Archive service will +use the `archive-1` server. Both services user the `maxuser` user with the `maxpwd` password. + +Next we will configure the listeners for these two services. + +``` +[Production Listener] +type=listener +service=Production +protocol=MySQLClient +port=4000 + +[Archive Listener] +type=listener +service=Archive +protocol=MySQLClient +port=4001 +``` + +The `port` parameter controls which port the listener will listen on and where the client +connections should be made. The `service` parameter tells which listener belongs to which +service. + +After the serivces and their listeners are configured we will configure the two filters we'll use. We +begin with the Tee filter. + +``` +[Tee] +type=filter +module=tee +service=Archive +match=\(insert\)\|\(update\)\|\(delete\) +``` + +The `service` parameter controls which service we want to duplicate the statements to. +This needs to be set to the section name of the archive service: `Archive`. The `match` +parameter is a regular expression which, if matched, cause the statement to be duplicated. +Here we will want a regular expression which will match the INSERT, UPDATE and DELETE +statements. + +After the Tee filter is configured, we will configure the Mqfilter to communicate with +the RabbitMQ server we've set up. + +``` +[MQ Filter] +type=filter +module=mqfilter +hostname=192.168.0.201 +port=5672 +username=msguser +password=msgpwd +exchange=msg-ex-1 +queue=msg-queue-1 +key=MaxScale +``` + +The `hostname` is the address and the `port` is the port of the RabbitMQ server. For +`username` and `password` we will use the `msguser` and `msgpwd` credentials we +created earlier. The `exchange` is the name of the RabbitMQ exchange we're using and +the `key` is the key used for all the sent messages. The messages will be stored +in the `mxs-queue-1` queue. + +The last thing to add is the administration interface. + +``` +[MaxAdmin Service] +type=service +router=cli + +[MaxAdmin Listener] +type=listener +service=MaxAdmin Service +protocol=maxscaled +port=6603 +``` + +Now we have created the MaxScale configuration file and all we need to do is to save +it in `/etc/maxscale.cnf`, start MaxScale and test that it works. The testing will +be done in the next section. + +Here is the complete configuration file. + +``` +# The production and archive servers +[production-1] +type=server +address=192.168.0.200 +port=3306 +protocol=MySQLBackend + +[archive-1] +type=server +address=192.168.0.201 +port=3000 +protocol=MySQLBackend + +# MySQL server monitor +[MySQL Monitor] +type=monitor +module=mysqlmon +servers=production-1, archive-1 +user=maxuser +passwd=maxpwd +monitor_interval=5000 + +# Production service +[Production] +type=service +router=readconnroute +servers=production-1 +user=maxuser +passwd=maxpwd +filters=Tee + +# Archive service +[Archive] +type=service +router=readconnroute +servers=archive-1 +user=maxuser +passwd=maxpwd +filters=MQ Filter + +# Listeners for the services +[Production Listener] +type=listener +service=Production +protocol=MySQLClient +port=4000 + +[Archive Listener] +type=listener +service=Archive +protocol=MySQLClient +port=4001 + +# Tee filter to duplicate insert, update and delete +# statements to the archive server +[Tee] +type=filter +module=tee +service=Archive +match=\(insert\)\|\(update\)\|\(delete\) + +# Mqfilter to log the canonical versions of the archive +# server queries to a RabbitMQ broker +[MQ Filter] +type=filter +module=mqfilter +hostname=192.168.0.201 +port=5672 +username=msguser +password=msgpwd +exchange=msg-ex-1 +queue=msg-queue-1 +key=MaxScale + +# Administration interface setup +[MaxAdmin Service] +type=service +router=cli + +[MaxAdmin Listener] +type=listener +service=MaxAdmin Service +protocol=maxscaled +port=6603 +``` + +## Testing the setup + +Now that we have created the configuration file, prepared the RabbitMQ server +and the database servers we can start testing the setup. We do that by starting +MaxScale: + +``` +sudo systemctl start maxscale +``` + +We can see the state of the two servers with MaxAdmin: + +``` +maxadmin list servers + +Servers. +-------------------+-----------------+-------+-------------+-------------------- +Server | Address | Port | Connections | Status +-------------------+-----------------+-------+-------------+-------------------- +production-1 | 192.168.0.200 | 3306 | 0 | Running +archive-1 | 192.168.0.201 | 3000 | 0 | Running +-------------------+-----------------+-------+-------------+-------------------- +``` + +We can also check the queue status on the RabbitMQ server: + +``` +sudo rabbitmqctl list_queues + +Listing queues ... +``` + +If we create a connection on the Production service on port 4000 and execute +a set of data modifying statemets we should see an equal number of statements +being sent to the RabbitMQ server: + +``` +MariaDB [(none)]> insert into test.t1 values(1); +Query OK, 1 row affected (0.09 sec) + +MariaDB [(none)]> insert into test.t1 values(2); +Query OK, 1 row affected (0.07 sec) + +MariaDB [(none)]> update test.t1 set id = 0; +Query OK, 2 rows affected (0.05 sec) +Rows matched: 2 Changed: 2 Warnings: 0 +``` + +On the RabbitMQ server we can see the new queue `msg-queue-1` with three new messages in it. + +``` +sudo rabbitmqctl list_queues + +Listing queues ... +msg-queue-1 3 +``` + +We can also see the data on the archive server: + +``` +MariaDB [(none)]> select * from test.t1; ++------+ +| id | ++------+ +| 0 | +| 0 | ++------+ +2 rows in set (0.00 sec) +``` + +To read the data from the RabbitMQ, we can use the RabbitMQ Consumer tool +included in the MaxScale source. For a tutorial on how to use this tool, +please read [RabbitMQ Consumer Client](../Filters/RabbitMQ-Consumer-Client.md). diff --git a/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md b/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md index 3015f5642..674f2fa08 100644 --- a/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md +++ b/Documentation/Tutorials/Replication-Proxy-Binlog-Router-Tutorial.md @@ -1,62 +1,66 @@ # MaxScale as a Binlog Server MaxScale was designed as a highly configurable proxy that sits between a database layer and the clients of that database, the binlog router described here is somewhat different to that original concept, moving MaxScale down to play a role within the database layer itself. -In a traditional MySQL replication setup a single master server is created and a set of slaves MySQL instances are configured to pull the binlog files from that master to the slaves. There are some problems however in this setup; when the number of slaves servers starts to increase an increasing load is placed on the master to serve the binlogs to each slave. When a master server fails every slave server requires some action to be performed before a new server can become the master server. +In a traditional MySQL replication setup a single master server is created and a set of slaves MySQL instances are configured to pull the binlog files from that master to the slaves. There are some problems, however, in this setup; when the number of slaves grows an increasing load is placed on the master, to serve the binlogs to each slave. When the master server fails, some action must be performed on every slave server before a new server can become the master server. -Introducing a proxy layer between the master server and the slave servers can improve the situation by reducing the load on the master to simply serving the proxy layer rather than all of the slaves and the slaves only need to be aware of the proxy layer and not the real master server. Removing this requirement for the slaves to have knowledge of the master greatly simplifies the process of replacing a failed master within a replication environment. +Introducing a proxy layer between the master server and the slave servers can improve the situation, by reducing the load on the master to simply serving the proxy layer rather than all of the slaves. The slaves only need to be aware of the proxy layer and not of the real master server. Removing need for the slaves to have knowledge of the master, greatly simplifies the process of replacing a failed master within a replication environment. ## MariaDB/MySQL as a Binlog Server -The most obvious solution to the requirement for a proxy layer within a replication environment is to use a MariaDB or MySQL database instance. The database server is designed to allow this, since a slave server is able to be configured such that it will produce binary logs for updates it has itself received via replication from the master server. This is done with the log_slave_updates configuration option of the server. In this case the server is known as an intermediate master, it is both a slave to the real master and a master to the other slaves in the configuration. +The most obvious solution to the requirement for a proxy layer within a replication environment is to use a MariaDB or MySQL database instance. The database server is designed to allow this, since a slave server is able to be configured such that it will produce binary logs for updates it has itself received via replication from the master server. This is done with the *log_slave_updates* configuration option of the server. In this case the server is known as an intermediate master, it is simultanously a slave to the real master and a master to the other slaves in the configuration. -Using an intermediate master does not however solve all the problems and introduces some due to the way replication is implemented. A slave server reads the binary log data and creates a relay log from that binary log. This then provides a source of SQL statements which are executed within the slave in order to make the same changes to the databases on the slaves as were made on the master. If the log_slave_updates option has been enabled new binary log entries are created for the statements executed from the relay log. This means that the data in the binary log of the intermediate master is not a direct copy of the data that was received from the binary log of the real master. The resultant changes to the database will be the same, provided no updates have been performed on the intermediate master that did not originate on the real master, but the steps to achieve those changes may be different. In particular if group commit functionality is used, to allow multiple transactions to commit in parallel, these may well be different on the intermediate master. This can cause a reduction in the parallelism of the commits and a subsequent reduction in the performance of the slave servers. +Using an intermediate master does not, however, solve all the problems and introduces some new ones, due to the way replication is implemented. A slave server reads the binary log data and creates a relay log from that binary log. This log provides a source of SQL statements, which are executed within the slave in order to make the same changes to the databases on the slaves as were made on the master. If the *log_slave_updates* option has been enabled, new binary log entries are created for the statements executed from the relay log. + +The above means that the data in the binary log of the intermediate master is not a direct copy of the data that was received from the binary log of the real master. The resultant changes to the database will be the same, provided no updates have been performed on the intermediate master that did not originate on the real master, but the steps to achieve those changes may be different. In particular, if group commit functionality is used, to allow multiple transactions to commit in parallel, these may well be different on the intermediate master. This can cause a reduction in the parallelism of the commits and a subsequent reduction in the performance of the slave servers. This re-execution of the SQL statements also adds latency to the intermediate master solution, since the full process of parsing, optimization and execution must occur for every statement that is replicated from the master to the slaves must be performed in the intermediate master. This latency introduces lag in the replication chain, with a greater delay being introduced from the time a transaction is committed on the master until the data is available on the slaves. Use of an intermediate master does improve the process of failover of the master server, since the slaves are only aware of the intermediate master the process of promoting one of the existing slaves to become the new master only involves that slave and the intermediate master. A slave can become the new master as soon as all the changes from the intermediate master have been processed. The intermediate master then needs to be reset to the correct point in the binary log of the new master and replication can continue. -An added complexity that needs to be dealt with is the failure of the intermediate master itself. If this occurs then the same problem as described earlier exists, all slaves must be updated when a new intermediate master is created. If multiple intermediate masters are used there is also a restriction that slaves can not be moved from the failed intermediate master to another intermediate master due to the fact that the binlog on the different intermediate nodes are not guaranteed to be the same. +An added complexity that needs to be dealt with is the failure of the intermediate master itself. If this occurs then the same problem as described earlier exists, all slaves must be updated when a new intermediate master is created. If multiple intermediate masters are used, there is also a restriction that slaves can not be moved from the failed intermediate master to another intermediate master due to the fact that the binlog on the different intermediate nodes are not guaranteed to be the same. ## MaxScale's approach -MaxScale takes a much simpler approach to the process of being a Binlog Server. It acts as a slave to the real master and as a master to the slaves in the same way as an intermediate master does, however it does not implement any re-execution of the statements within the binary log. MaxScale creates a local cache of the binary logs it receives from the master and it will serve binary log events to the slaves from this cache of the master's binary log. This means that the slaves will always get binary log events that have a one-to-one correlation to those written by the master. Parallelism in the binary log events of the master is maintained in the events that are observed by the slaves. +MaxScale takes a much simpler approach to the process of being a Binlog Server. It acts as a slave to the real master and as a master to the slaves, in the same way as an intermediate master does. However, it does not implement any re-execution of the statements within the binary log. MaxScale creates a local cache of the binary logs it receives from the master and will serve binary log events to the slaves from this cache of the master's binary log. This means that the slaves will always get binary log events that have a one-to-one correlation to those written by the master. Parallelism in the binary log events of the master is maintained in the events that are observed by the slaves. -In the MaxScale approach the latency that is introduced is mostly the added network latency associated with adding the extra network hop. There is no appreciable processing performed at the MaxScale level, other than for managing the local cache of the binlog files. +In the MaxScale approach, the latency that is introduced is mostly the added network latency associated with adding the extra network hop. There is no appreciable processing performed at the MaxScale level, other than for managing the local cache of the binlog files. -In addition every MaxScale that is acting as a proxy of the master will have exactly the same binlog events as the master itself. This means that a slave can be moved between any of the MaxScale server or to the real master without the need to perform any special processing. The result is much simpler behavior for failure recovery and the ability to have a very simple, redundant proxy layer with slaves free to both between the proxies. +In addition, every MaxScale that is acting as a proxy of the master will have exactly the same binlog events as the master itself. This means that a slave can be moved between any of the MaxScale server or to the real master without a need to perform any special processing. The result is much simpler behavior for failure recovery and the ability to have a very simple, redundant proxy layer with slaves free to both between the proxies. # Configuring MaxScale as a Binlog Server Using MaxScale as a Binlog Server is much the same as using MaxScale as a proxy between the clients and the database servers. In this case the master server should be considered as the database backend and the slave servers as the clients of MaxScale. ## Service Configuration -As with any MaxScale configuration a good starting point is with the service definition with the maxscale.cnf file. The service requires a name which is the section name in the ini file, a type parameter with a value of service and the name of the router plugin that should be loaded. In the case of replication proxies this router name is binlogrouter. +As with any MaxScale configuration a good starting point is with the service definition with the *maxscale.cnf* file. The service requires a name which is the section name in the ini file, a type parameter with a value of service and the name of the router plugin that should be loaded. In the case of replication proxies this router name is *binlogrouter*. + +``` +[Replication] +type=service +router=binlogrouter ``` -[Replication] -type=service -router=binlogrouter -``` -Other standard service parameters need to be given in the configuration section that are used to retrieve the set of users from the backend (master) database, also a version string can be given such that the MaxScale instance will report this version string to the slave servers that connect to MaxScale. The master server entry must also be given. In the current implementation of the router only a single server can be given. +Other standard service parameters need to be given in the configuration section that are used to retrieve the set of users from the backend (master) database, also a version string can be given such that the MaxScale instance will report this version string to the slave servers that connect to MaxScale. + ``` [Replication] type=service router=binlogrouter -servers=masterdb version_string=5.6.17-log user=maxscale passwd=Mhu87p2D ``` -The user and passwd entries in the above example are used in order for MaxScale to populate the credential information that is required to allow the slaves to connect to MaxScale. This user should be configured in exactly the same way a for any other MaxScale service, i.e. the user needs access to the mysql.user table and the mysql.db table as well as having the ability to perform a SHOW DATABASES command. -The master server details are provided by a master.ini file located in binlog directory and could be changed via CHANGE MASTER TO command issued via MySQL connection to MaxScale, check Master setup section below for further details. +The *user* and *passwd* entries in the above example are used in order for MaxScale to populate the credential information that is required to allow the slaves to connect to MaxScale. This user should be configured in exactly the same way a for any other MaxScale service, i.e. the user needs access to the *mysql.user* table and the *mysql.db* table as well as having the ability to perform a *SHOW DATABASES* command. + +The master server details are currently provided by a **master.ini** file located in binlog directory and could be changed via *CHANGE MASTER TO* command issued via MySQL connection to MaxScale; refer to the Master setup section below for further details. In the current implementation of the router only a single server can be used. -The final configuration requirement is the router specific options. The binlog router requires a set of parameters to be passed, these are passed in the router_options parameter of the service definition as a comma separated list of name value pairs. +The final configuration requirement is the router specific options. The binlog router requires a set of parameters to be passed, these are passed, as a comma separated list of name value pairs, in the *router_options* parameter of the service definition.. ### binlogdir -This parameter allows the location that MaxScale uses to store binlog files to be set. If this parameter is not set to a directory name then MaxScale will store the binlog files in the directory /var/cache/maxscale/. -In the binlog dir there is also the 'cache' directory that contains data retrieved from the master dureing registration phase and the master.ini file wich contains the configuration of current configured master. +This parameter allows the location that MaxScale uses to store binlog files to be set. If this parameter is not set to a directory name then MaxScale will store the binlog files in the directory */var/cache/maxscale/*. +In the binlog dir there is also the 'cache' directory that contains data retrieved from the master during registration phase and the *master.ini* file wich contains the configuration of current configured master. ### uuid @@ -89,7 +93,7 @@ The MaxScale router will identify itself to the slaves using the server hostname This is the user name that MaxScale uses when it connects to the master. This user name must have the rights required for replication as with any other user that a slave uses for replication purposes. If the user parameter is not given in the router options then the same user as is used to retrieve the credential information will be used for the replication connection, i.e. the user in the service entry. -The user that is used for replication, either defined using the user= option in the router options or using the username and password defined of the service must be granted replication privileges on the database server. +The user that is used for replication, either defined using the *user=* option in the router options or using the username and password defined of the service must be granted replication privileges on the database server. ``` MariaDB> CREATE USER 'repl'@'maxscalehost' IDENTIFIED by 'password'; @@ -98,7 +102,7 @@ The user that is used for replication, either defined using the user= option in ### password -The password of the above user. If the password is not explicitly given then the password in the service entry will be used. For compatibility with other username and password definitions within the MaxScale configuration file it is also possible to use the parameter passwd=. +The password of the above user. If the password is not explicitly given then the password in the service entry will be used. For compatibility with other username and password definitions within the MaxScale configuration file it is also possible to use the parameter *passwd=*. ### heartbeat @@ -106,7 +110,7 @@ This defines the value of the heartbeat interval in seconds for the connection t ### send_slave_heartbeat -This defines whether (on | off) MaxSclale sends to the slave the heartbeat packet when there are no real binlog events to send. The default value if 'off', no heartbeat event is sent to slave server. If value is 'on' the interval value (requested by the slave during registration) is reported in the diagnostic output and the packect is send after the time interval without any event to send. +This defines whether (on | off) MaxScale sends to the slave the heartbeat packet when there are no real binlog events to send. The default value if 'off', no heartbeat event is sent to slave server. If value is 'on' the interval value (requested by the slave during registration) is reported in the diagnostic output and the packect is send after the time interval without any event to send. ### burstsize @@ -116,10 +120,11 @@ This parameter is used to define the maximum amount of data that will be sent to This parameter is used to enable/disable incomplete transactions detection in binlog router. When MaxScale starts an error message may appear if current binlog file is corrupted or an incomplete transaction is found. -During normal operations binlog events are not distributed to the slaves until a COMMIT is seen. -The default value is off, set transaction_safety=on to enable the incomplete transactions detection. +During normal operations binlog events are not distributed to the slaves until a *COMMIT* is seen. +The default value is off, set *transaction_safety=on* to enable the incomplete transactions detection. A complete example of a service entry for a binlog router service would be as follows. + ``` [Replication] type=service @@ -131,11 +136,11 @@ A complete example of a service entry for a binlog router service would be as fo router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,filestem=mybin,heartbeat=30,binlogdir=/var/binlogs,transaction_safety=1,master_version=5.6.19-common,master_hostname=common_server,master_uuid=xxx-fff-cccc-common,master-id=999,mariadb10-compatibility=1 ``` -The minimum set of router options that must be given in the configuration are are server-id and master-id, default values may be used for all other options. +The minimum set of router options that must be given in the configuration are are *server-id* and *master-id*, default values may be used for all other options. ## Listener Section -As per any service in MaxScale a listener section is required to define the address, port and protocol that is used to listen for incoming connections. Those incoming connections will originate from the slave servers or from a MySQL client in order to administer/configure the master server configuration via SQL commands such as STOP/START SLAVE and CHANGE MASTER TO ... +As per any service in MaxScale a listener section is required to define the address, port and protocol that is used to listen for incoming connections. Those incoming connections will originate from the slave servers or from a MySQL client in order to administer/configure the master server configuration via SQL commands such as *STOP/START SLAVE* and *CHANGE MASTER TO* ... [Replication Listener] type=listener @@ -143,7 +148,7 @@ As per any service in MaxScale a listener section is required to define the addr protocol=MySQLClient port=5308 -The protocol used by slaves for connection to MaxScale is the same MySQLClient protocol that is used for client applications to connect to databases, therefore the same MaxScale protocol module can be used. +The protocol used by slaves for connection to MaxScale is the same *MySQLClient* protocol that is used for client applications to connect to databases, therefore the same MaxScale protocol module can be used. # MaxScale replication diagnostics @@ -230,19 +235,19 @@ The binlog router module of MaxScale produces diagnostic output that can be view Binlog Router Plugin is compatible with MySQL 5.6 and MariaDB 5.5, the current default. -In order to use it with MySQL 5.6, the GTID_MODE setting must be OFF and connecting slaves mustn't use MASTER_AUTO_POSITION = 1 option. +In order to use it with MySQL 5.6, the *GTID_MODE* setting must be OFF and connecting slaves must not use *MASTER_AUTO_POSITION = 1* option. It’s also works with a MariaDB 10.0 setup (master and slaves) but slave connection must not include any GTID feature. -Binlog Router currently does not work for MySQL 5.5 due to missing @@global.binlog_checksum var. +Binlog Router currently does not work for MySQL 5.5 due to missing *@@global.binlog_checksum* variable. # Master server setup/change -In MaxScale ini file the server section for master is no longer required, same for servers=master_server in the service section. The mastet server setup is currently managed via CHANGE MASTER TO command issued in MySQL client connection to MaxScale or by providing a proper master.ini file in the binlogdir. +In the MaxScale ini file the server section for master is no longer required, same for *servers=master_server* in the service section. The master server setup is currently managed via *CHANGE MASTER TO* command issued in MySQL client connection to MaxScale or by providing a proper *master.ini* file in the *binlogdir*. -If MaxScale starts without master.ini there is no replication configured to any master and slaves cannot register to the router: the binlog router could be later configured via CHANGE MASTER TO and the master.ini file will be written. +If MaxScale starts without *master.ini* there is no replication configured to any master and slaves cannot register to the router: the binlog router could be later configured via *CHANGE MASTER TO* and the *master.ini* file will be written. -master.ini file example: +*master.ini* file example: [binlog_configuration] master_host=127.0.0.1 @@ -253,15 +258,15 @@ master.ini file example: Enabling replication from a master server requires: -CHANGE MASTER TO MASTER_HOST=‘$master_server’, MASTER_PORT=$master_port, MASTER_USER='repl', MASTER_PASSWORD=‘somepasswd’, MASTER_LOG_FILE=‘repl-bin.000159', MASTER_LOG_POS=4 + CHANGE MASTER TO MASTER_HOST=‘$master_server’, MASTER_PORT=$master_port, MASTER_USER='repl', MASTER_PASSWORD=‘somepasswd’, MASTER_LOG_FILE=‘repl-bin.000159', MASTER_LOG_POS=4 -It's possible to specify the desired MASTER_LOG_FILE but position must be 4 +It's possible to specify the desired *MASTER_LOG_FILE* but position must be 4 -The initfile option is nolonger available, filestem option also not available as the stem is automatically set by parsing MASTER_LOG_FILE +The initfile option is nolonger available, filestem option also not available as the stem is automatically set by parsing *MASTER_LOG_FILE*. ### Stop/start the replication -When router is configured and it's properly working it could be possible to stop/start replication: +When router is configured and it is properly working it is possible to stop/start replication: MariaDB> STOP SLAVE; ... @@ -269,11 +274,11 @@ When router is configured and it's properly working it could be possible to stop ... MariaDB> START SLAVE; -Connected or new slave connections are not affected: this STOP/START only controls the the connection to the master and the binlog events receiving. +Connected or new slave connections are not affected: this *STOP/START* only controls the the connection to the master and the binlog events receiving. ### Change the Master server configuration -When router is configured and it's properly working it could be possible to change the master parameters. +When router is configured and it is properly working it is possible to change the master parameters. First step is stop the replication from the master. MariaDB> STOP SLAVE; @@ -282,7 +287,7 @@ Next step is the master configuration MariaDB> CHANGE MASTER TO ... -A succesfull configuration change results in master.ini being updated. +A succesful configuration change results in *master.ini* being updated. Any error is reported in the MySQL and in log files @@ -295,9 +300,9 @@ The upported options are: MASTER_LOG_FILE MASTER_LOG_POS -There are some constraints related to MASTER_LOG_FILE and MASTER_LOG_POS: +There are some constraints related to *MASTER_LOG_FILE* and *MASTER_LOG_POS*: -MASTER_LOG_FILE could be changed to next binlog in sequence with MASTER_LOG_POS=4 or to current one at current position. +*MASTER_LOG_FILE* could be changed to next binlog in sequence with *MASTER_LOG_POS=4* or to current one at current position. Examples: @@ -305,21 +310,21 @@ Examples: MariaDB> CHANGE MASTER TO MASTER_LOG_FILE=‘mysql-bin.000003',MASTER_LOG_POS=8888 -This could be applied to current master_host/port or a new one. -If there is a master server maintenance and a slave is beeing promoted as master it should be checked that binlog file and position are valid: in case of any error replication stops and errors are reported via SHOW SLAVE STATUS and in error logs. +This could be applied to current master_host/port or a new one. +If there is a master server maintenance and a slave is beeing promoted as master it should be checked that binlog file and position are valid: in case of any error replication stops and errors are reported via *SHOW SLAVE STATUS* and in error logs. 2) Current binlog file is ‘mysql-bin.000099', position 1234 MariaDB> CHANGE MASTER TO MASTER_LOG_FILE=‘mysql-bin.000100',MASTER_LOG_POS=4 This could be applied with current master_host/port or a new one -If transaction safety option is on and the current binlog file contains an incomplete transaction it will be truncated to the position where transaction started. +If transaction safety option is on and the current binlog file contains an incomplete transaction it will be truncated to the position where transaction started. In such situation a proper message is reported in MySQL connection and with next START SLAVE binlog file truncation will occurr and MaxScale will request events from the master using the next binlog file at position 4. The above scenario might refer to a master crash/failure: - the new server that has just been promoted as master doesn't have last transaction events but it should have the new binlog file (the next in sequence). + the new server that has just been promoted as master doesn't have last transaction events but it should have the new binlog file (the next in sequence). Truncating the previous MaxScale binlog is safe as that incomplete transaction is lost. -It should be checked that current master or new one has the new bnlog file, in case of any error replication stops and errors are reported via SHOW SLAVE STATUS and in error logs. +It should be checked that current master or new one has the new binlog file, in case of any error replication stops and errors are reported via *SHOW SLAVE STATUS* and in error logs. MariaDB> START SLAVE; @@ -327,19 +332,19 @@ Check for any error in log files and with MariaDB> SHOW SLAVE STATUS; -In some situations replication state could be STOPPED and proper messages are displyed in error logs and in SHOW SLAVE STATUS +In some situations replication state could be *STOPPED* and proper messages are displyed in error logs and in *SHOW SLAVE STATUS*, -In order to resolve any mistake done with CHANGE MASTER TO MASTER_LOG_FILE / MASTER_LOG_POS, another administrative command would be helpful. +In order to resolve any mistake done with *CHANGE MASTER TO MASTER_LOG_FILE / MASTER_LOG_POS*, another administrative command would be helpful. MariaDB> RESET SLAVE; -This command removes master.ini file, blanks all master configuration in memory and sets binlog router in unconfigured state: a CHANHE MASTER TO command should be issued for the new configuration. +This command removes *master.ini* file, blanks all master configuration in memory and sets binlog router in unconfigured state: a *CHANGE MASTER TO* command should be issued for the new configuration. Note: existing binlog files are not touched by this command. ### Slave servers setup -Examples of CHANGE MASTER TO command issued on a slave server that wants to gets replication events from MaxScale binlog router: +Examples of *CHANGE MASTER TO* command issued on a slave server that wants to gets replication events from MaxScale binlog router: ``` CHANGE MASTER TO MASTER_HOST=‘$maxscale_IP’, MASTER_PORT=5308, MASTER_USER='repl', MASTER_PASSWORD=‘somepasswd’, MASTER_LOG_FILE=‘mysql-bin.000001' @@ -347,13 +352,13 @@ MASTER_LOG_FILE=‘mysql-bin.000001' CHANGE MASTER TO MASTER_HOST=‘$maxscale_IP’, MASTER_PORT=5308, MASTER_USER='repl', MASTER_PASSWORD=‘somepasswd’, MASTER_LOG_FILE=‘mysql-bin.000159', MASTER_LOG_POS=245 ``` -The latter example specifies a MASTER_LOG_POS for the selected MASTER_LOG_FILE +The latter example specifies a *MASTER_LOG_POS* for the selected *MASTER_LOG_FILE* Note: - - MASTER_LOG_FILE must be set to one of existing binlog files in MaxScale binlogdir + - *MASTER_LOG_FILE* must be set to one of existing binlog files in MaxScale binlogdir - - If MASTER_LOG_POS is not set with CHANGE MASTER TO it defaults to 4 + - If *MASTER_LOG_POS* is not set with *CHANGE MASTER TO* it defaults to 4 - Latest binlog file name and pos in MaxScale could be find via maxadmin output or from mysql client connected to MaxScale: diff --git a/Documentation/Upgrading/Upgrading-BinlogRouter-To-Maxscale-1.3.md b/Documentation/Upgrading/Upgrading-BinlogRouter-To-Maxscale-1.3.md new file mode 100644 index 000000000..8df925884 --- /dev/null +++ b/Documentation/Upgrading/Upgrading-BinlogRouter-To-Maxscale-1.3.md @@ -0,0 +1,27 @@ +# Upgrading Binlog Router to MaxScale to 1.3 + +This document describes upgrading the Binlog Router module to MaxScale version 1.3. + +## What's new + +The master server details are now provided with a **master.ini** file located in +the binlog directory and it can be changed using a CHANGE MASTER TO command issued +via a MySQL connection to MaxScale. + +This file, properly filled, is now mandatory and without it the binlog router +cannot connect to the master database. + +Before starting binlog router after MaxScale 1.3 upgrade, please add relevant +information to *master.ini*, example: + +``` +[binlog_configuration] +master_host=127.0.0.1 +master_port=3308 +master_user=repl +master_password=somepass +filestem=repl-bin +``` + +Additionally, the option ```servers=masterdb``` in the service definition is no +longer required. diff --git a/Documentation/Upgrading/Upgrading-To-MaxScale-1.3.md b/Documentation/Upgrading/Upgrading-To-MaxScale-1.3.md new file mode 100644 index 000000000..4c814b4a5 --- /dev/null +++ b/Documentation/Upgrading/Upgrading-To-MaxScale-1.3.md @@ -0,0 +1,11 @@ +# Upgrading MaxScale from 1.2 to 1.3 + +This document describes upgrading MaxScale from version 1.2.1 to 1.3. + +The major changes can be found in the [changelog](../Changelog.md) and more +detailed information in the [release notes](../Release-Notes/MaxScale-1.3.0-Release-Notes.md). + +## Installation + +Before starting the upgrade, we **strongly** recommend you back up your current +configuration file. diff --git a/Documentation/check_links.sh b/Documentation/check_links.sh new file mode 100755 index 000000000..3a7e70925 --- /dev/null +++ b/Documentation/check_links.sh @@ -0,0 +1,36 @@ +# +# This file is distributed as part of the MariaDB Corporation MaxScale. It is free +# software: you can redistribute it and/or modify it under the terms of the +# GNU General Public License as published by the Free Software Foundation, +# version 2. +# +# This program is distributed in the hope that it will be useful, but WITHOUT +# ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS +# FOR A PARTICULAR PURPOSE. See the GNU General Public License for more +# details. +# +# You should have received a copy of the GNU General Public License along with +# this program; if not, write to the Free Software Foundation, Inc., 51 +# Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. +# +# Copyright MariaDB Corporation Ab 2015 +# + +#!/bin/bash + +# Check that all links to Markdown format files point to existing local files + +homedir=$(pwd) + +for file in $(find . -name '*.md') +do + cd "$(dirname $file)" + for i in `grep -o '\[.*\]([^#].*[.]md)' $(basename $file)| sed -e 's/\[.*\](\(.*\))/\1/'` + do + if [ ! -f $i ] + then + echo "Link $i in $file is not correct!" + fi + done + cd "$homedir" +done