Merge branch 'origin/develop'
This commit is contained in:
Martin Brampton
@ -41,6 +41,11 @@
|
||||
- [Tee Filter](filters/Tee-Filter.md)
|
||||
- [Top N Filter](filters/Top-N-Filter.md)
|
||||
- [Firewall Filter](filters/Firewall-Filter.md)
|
||||
- [RabbitMQ Filter](filters/RabbitMQ-Filter.md)
|
||||
|
||||
## Utilities
|
||||
|
||||
- [RabbitMQ Consumer Client](filters/RabbitMQ-Consumer-Client.md)
|
||||
|
||||
## Design Documents
|
||||
|
||||
@ -55,3 +60,4 @@
|
||||
- [MaxScale 1.0 Release Notes](Release-Notes/MaxScale-1.0-Release-Notes.md)
|
||||
- [MaxScale 1.0.1 Release Notes](Release-Notes/MaxScale-1.0.1-Release-Notes.md)
|
||||
- [MaxScale 1.0.3 Release Notes](Release-Notes/MaxScale-1.0.3-Release-Notes.md)
|
||||
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@ -59,11 +59,16 @@ As with uuid, MaxScale must have a unique server-id for the connection it makes
|
||||
|
||||
### user
|
||||
|
||||
This is the user name that MaxScale uses when it connects to the master. This user name must have the rights required for replication as with any other user that a slave uses for replication purposes.
|
||||
This is the user name that MaxScale uses when it connects to the master. This user name must have the rights required for replication as with any other user that a slave uses for replication purposes. If the user parameter is not given in the router options then the same user as is used to retrieve the credential information will be used for the replication connection, i.e. the user in the service entry.
|
||||
|
||||
The user that is used for replication, either defined using the user= option in the router options or using the username and password defined of the service must be granted replication privileges on the database server.
|
||||
|
||||
MariaDB> CREATE USER 'repl'@'maxscalehost' IDENTIFIED by 'password';
|
||||
MariaDB> GRANT REPLICATION SLAVE ON *.* TO 'repl'@'maxscalehost';
|
||||
|
||||
### password
|
||||
|
||||
The password of the above user.
|
||||
The password of the above user. If the password is not explicitly given then the password in the service entry will be used. For compatibility with other username and password definitions within the MaxScale configuration file it is also possible to use the parameter passwd=.
|
||||
|
||||
### master-id
|
||||
|
||||
@ -75,7 +80,7 @@ This parameter is used to provide the stem of the file names that are used to st
|
||||
|
||||
### initialfile
|
||||
|
||||
This optional parameter allows for the administrator to define the number of the first binlog file to download. In normal circumstances MaxScale will use any existing binlog file to determine what to request from the master. If there are no files it will then ask for the binlog file with the index number defined in the initialfile parameter. If this parameter is not set then MaxScale will ask the master for binlog events from file 1.
|
||||
This optional parameter allows for the administrator to define the number of the first binlog file to download. If MaxScale has previously received binlogs it will use those existing binlog files to determine what to request from the master. If no files have been downloaded MaxScale will then ask for the binlog file with the index number defined in the initialfile parameter. If this parameter is not set then MaxScale will ask the master for binlog events from file 1.
|
||||
|
||||
### binlogdir
|
||||
|
||||
@ -87,19 +92,21 @@ This defines the value of the heartbeat interval in seconds for the connection t
|
||||
|
||||
### burstsize
|
||||
|
||||
This parameter is used to define the maximum amount of data that will be sent to a slave by MaxScale when that slave is lagging behind the master. In this situation the slave is said to be in "catchup mode", this parameter is designed to both prevent flooding of that slave and also to prevent threads within MaxScale spending disproportionate amounts of time with slaves that are lagging behind the master. The burst size can be defined in Kb, Mb or Gb by adding the qualifier K, M or G to the number given.
|
||||
This parameter is used to define the maximum amount of data that will be sent to a slave by MaxScale when that slave is lagging behind the master. In this situation the slave is said to be in "catchup mode", this parameter is designed to both prevent flooding of that slave and also to prevent threads within MaxScale spending disproportionate amounts of time with slaves that are lagging behind the master. The burst size can be defined in Kb, Mb or Gb by adding the qualifier K, M or G to the number given. The default value of burstsize is 1Mb and will be used if burstsize is not given in the router options.
|
||||
|
||||
A complete example of a service entry for a binlog router service would be as follows.
|
||||
|
||||
[Replication]
|
||||
type=service
|
||||
router=binlogrouter
|
||||
servers=maserdb
|
||||
servers=masterdb
|
||||
version_string=5.6.17-log
|
||||
router_options=uuid=f12fcb7f-b97b-11e3-bc5e-0401152c4c22,server-id=3,user=repl,password=slavepass,master-id=1,filestem=mybin,heartbeat=30,binlogdir=/home/mriddoch/binlogs
|
||||
user=maxscale
|
||||
passwd=Mhu87p2D
|
||||
|
||||
The minimum set of router options that must be given in the configuration are are server-id and aster-id, default values may be used for all other options.
|
||||
|
||||
## Listener Section
|
||||
|
||||
As per any service in MaxScale a listener section is required to define the address, port and protocol that is used to listen for incoming connections. In this case those incoming connections will originate from the slave servers.
|
||||
|
||||
38
filters/RabbitMQ-Consumer-Client.md
Normal file
38
filters/RabbitMQ-Consumer-Client.md
Normal file
@ -0,0 +1,38 @@
|
||||
#RabbitMQ Consumer Client
|
||||
|
||||
## Overview
|
||||
|
||||
This utility tool is used to read messages from a RabbitMQ broker sent by the [RabbitMQ Filter](RabbitMQ-Filter.md) and forward these messages into an SQL database as queries.
|
||||
|
||||
## Command Line Arguments
|
||||
|
||||
The **RabbitMQ Consumer Client** only has one command line argument.
|
||||
|
||||
| Command | Argument |
|
||||
|---------|-------------------------------------------------|
|
||||
| -c | Path to the folder containing the configuration file |
|
||||
|
||||
## Installation
|
||||
|
||||
To install the RabbitMQ Consumer Client you ca either use the provided packages or you can compile it from source code. The source code is included as a part of the MaxScale source code and can be found in the `rabbtmq_consumer` folder. Please refer to the [README](../../rabbitmq_consumer/README) in the folder for more detailed instructions about installation and configuration.
|
||||
|
||||
## Configuration
|
||||
|
||||
The consumer client requires that the `consumer.cnf` configuration file is either be present in the `etc` folder of the installation directory or in the folder specified by the `-c` argument.
|
||||
|
||||
The source broker, the destination database and the message log file can be configured into the separate `consumer.cnf` file.
|
||||
|
||||
| Option | Desctiption |
|
||||
|-----------|---------------------------------------------|
|
||||
| hostname | Hostname of the RabbitMQ server |
|
||||
| port | Port of the RabbitMQ server |
|
||||
| vhost | Virtual host location of the RabbitMQ server |
|
||||
| user | Username for the RabbitMQ server |
|
||||
| passwd | Password for the RabbitMQ server |
|
||||
| queue | Queue to consume from |
|
||||
| dbserver | Hostname of the SQL server |
|
||||
| dbport | Port of the SQL server |
|
||||
| dbname | Name of the SQL database to use |
|
||||
| dbuser | Database username |
|
||||
| dbpasswd | Database passwork |
|
||||
| logfile | Message log filename |
|
||||
61
filters/RabbitMQ-Filter.md
Normal file
61
filters/RabbitMQ-Filter.md
Normal file
@ -0,0 +1,61 @@
|
||||
#RabbitMQ Filter
|
||||
|
||||
## Overview
|
||||
This filter is designed to extract queries and transform them into a canonical form e.g. `INSERT INTO dabata.table VALUES ("John Doe", "Downtown",100,50.0);` turns into `INSERT INTO dabata.table VALUES ("?", "?",?,?);`. The filter pushes these canonized queries and their replies in to a RabbitMQ broker where they can later be retrieved. The retrieval can be done with your own application or the [RabbitMQ Consumer Client](RabbitMQ-Consumer-Client.md) utility tool, which reads the messages from the broker and sends the contents of those messages as SQL queries to a database.
|
||||
|
||||
## Configuration
|
||||
|
||||
The configuration block for the **mqfilter** filter requires the minimal filter options in it’s section within the MaxScale.cnf file, stored in $MAXSCALE_HOME/etc/MaxScale.cnf. Although the filter will start, it will use the default values which only work with a freshly installed RabbitMQ server and use its default values. This setup is mostly intednded for testing the filter.
|
||||
|
||||
The following is an example of a mqfilter configuration in the MaxScale.cnf file used for actual logging of queries to a RabbitMQ broker on a different host.
|
||||
|
||||
```
|
||||
[RabbitMQ]
|
||||
type=filter
|
||||
module=mqfilter
|
||||
hostname=192.168.122.100
|
||||
port=4000
|
||||
username=messageuser
|
||||
password=msgpwd
|
||||
exchange=msg-ex-1
|
||||
key=MaxScale
|
||||
logging_trigger=object,schema,source
|
||||
logging_strict=false
|
||||
logging_log_all=false
|
||||
logging_object=my1
|
||||
logging_schema=test
|
||||
logging_source_user=maxtest
|
||||
```
|
||||
|
||||
### Filter Options
|
||||
|
||||
The mqfilter filter does not support any filter options.
|
||||
|
||||
### Filter Parameters
|
||||
|
||||
The RabbitMQ filter has parameters to control which queries are logged based on either the attributes of the user or the query itself. These can be combined to to only log queries targeting a certain table in a certain database from a certain user from a certain network address.
|
||||
|
||||
|
||||
Option | Description | Accepted Values | Default |
|
||||
--------|-------------|-----------------|-------------
|
||||
logging_trigger | Set the logging level | `all, source, schema, object` | `all` |
|
||||
logging_strict | Sets whether to trigger when any of the parameters match or only if all parameters match | `true, false` | `false` |
|
||||
logging_log_all | Log only SELECT, UPDATE, DELETE and INSERT or all possible queries | `true, false` | `true` |
|
||||
logging_source_user | Comma-separated list of usernames to log | | |
|
||||
logging_source_host | Comma-separated list of hostnames to log | | |
|
||||
logging_schema | Comma-separated list of databases | | |
|
||||
logging_object | Comma-separated list of database objects |
|
||||
hostname | The server hostname where the messages are sent | | `localhost` |
|
||||
port | Port to send the messages to | | `5672` |
|
||||
username | Server login username | | `guest` |
|
||||
password | Server login password | | `guest` |
|
||||
vhost | The virtual host location on the server, where the messages are sent | | `/` |
|
||||
exchange | The name of the exchange | | `default_exchange` |
|
||||
exchange_type | The type of the exchange | `direct, fanout, topic, headers` | `direct` |
|
||||
key | The routing key used when sending messages to the exchange | | `key` |
|
||||
queue | The queue that will be bound to the used exchange | | |
|
||||
ssl_CA_cert | Path to the CA certificate in PEM format | | |
|
||||
ssl_client_cert | Path to the client cerificate in PEM format | | |
|
||||
ssl_client_key | Path to the client public key in PEM format | | |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user