hcq/MaxScale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

More format fixes.

Browse Source
This commit is contained in:
Markus Makela
2015-09-07 13:48:27 +03:00
parent 42fc2db65d
commit b8216aff4c
1 changed files with 64 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files

73
Documentation/Tutorials/RabbitMQ-Setup-And-MaxScale-Integration.md
View File

@ -29,11 +29,14 @@ systemctl start rabbitmq-server.service
To configure your RabbitMQ server, please refer to the RabbitMQ website: [http://www.rabbitmq.com/](http://www.rabbitmq.com/ RabbitMQ website). To configure your RabbitMQ server, please refer to the RabbitMQ website: [http://www.rabbitmq.com/](http://www.rabbitmq.com/ RabbitMQ website).
rabbitmqctl is a command line tool for managing a RabbitMQ broker. It performs all actions by connecting to one of the broker's nodes. rabbitmqctl is a command line tool for managing a RabbitMQ broker. It performs all actions by connecting to one of the broker's nodes.
``` ```
rabbitmqctl list_queues rabbitmqctl list_queues
rabbitmqctl list_queues | list_exchanges| cluster_status | list_bindings | list_connections | list_consumers | status rabbitmqctl list_queues | list_exchanges| cluster_status | list_bindings | list_connections | list_consumers | status
``` ```
Example output: Example output:
``` ```
[root@maxscale-02 MaxScale]# rabbitmqctl status [root@maxscale-02 MaxScale]# rabbitmqctl status
Status of node 'rabbit@maxscale-02' ... Status of node 'rabbit@maxscale-02' ...
@ -58,12 +61,15 @@ Example output:
x1 exchange q1 queue k1 [] x1 exchange q1 queue k1 []
...done. ...done.
``` ```
Interaction with the server may require stop & reset at some point: Interaction with the server may require stop & reset at some point:
``` ```
rabbitmqctl stop_app rabbitmqctl stop_app
rabbitmqctl reset rabbitmqctl reset
rabbitmqctl start_app rabbitmqctl start_app
``` ```
## Step 3 - Install and test the client libraries ## Step 3 - Install and test the client libraries
The selected library for MaxScale integration of RabbitMQ is: The selected library for MaxScale integration of RabbitMQ is:
@ -72,6 +78,7 @@ The selected library for MaxScale integration of RabbitMQ is:
### Manual software compilation ### Manual software compilation
To compile the RabbitMQ-C libraries manually: To compile the RabbitMQ-C libraries manually:
``` ```
git clone https://github.com/alanxz/rabbitmq-c.git git clone https://github.com/alanxz/rabbitmq-c.git
cd rabbitmq-c cd rabbitmq-c
@ -79,6 +86,7 @@ cmake -DCMAKE_INSTALL_PREFIX=/usr .
make make
make install make install
``` ```
Please note, this will install the packages to /usr. If you do not wish to install them to this location, provide a different value for the CMAKE_INSTALL_PREFIX variable. Please note, this will install the packages to /usr. If you do not wish to install them to this location, provide a different value for the CMAKE_INSTALL_PREFIX variable.
@ -87,14 +95,19 @@ Please note, this will install the packages to /usr. If you do not wish to insta
Check how to configure your distribution for the EPEL repository: [https://fedoraproject.org/wiki/EPEL](https://fedoraproject.org/wiki/EPEL EPEL) Check how to configure your distribution for the EPEL repository: [https://fedoraproject.org/wiki/EPEL](https://fedoraproject.org/wiki/EPEL EPEL)
Configure your repositories and install the software: Configure your repositories and install the software:
``` ```
yum install librabbitmq.x86_64 yum install librabbitmq.x86_64
``` ```
you might also like to install: you might also like to install:
``` ```
librabbitmq-tools.x86_64, librabbitmq-devel.x86_64 librabbitmq-tools.x86_64, librabbitmq-devel.x86_64
``` ```
Please note you may also install the rabbitmq server from the EPEL repository: Please note you may also install the rabbitmq server from the EPEL repository:
``` ```
yum install rabbitmq-server yum install rabbitmq-server
``` ```
@ -106,31 +119,42 @@ The required library librabbitmq-c is now installed and we continue with basic o
Please note, those example applications may not be included in the RPM library packages. Please note, those example applications may not be included in the RPM library packages.
#### Test 1 - create the exchange #### Test 1 - create the exchange
``` ```
[root@maxscale-02 examples]# ./amqp_exchange_declare [root@maxscale-02 examples]# ./amqp_exchange_declare
Usage: amqp_exchange_declare host port exchange exchangetype Usage: amqp_exchange_declare host port exchange exchangetype
``` ```
Declare the exchange: Declare the exchange:
``` ```
[root@maxscale-02 examples]# ./amqp_exchange_declare 127.0.0.1 5672 foo direct [root@maxscale-02 examples]# ./amqp_exchange_declare 127.0.0.1 5672 foo direct
``` ```
#### Test 2 - Listen to exchange with selected binding key #### Test 2 - Listen to exchange with selected binding key
``` ```
[root@maxscale-02 examples]# ./amqp_listen [root@maxscale-02 examples]# ./amqp_listen
Usage: amqp_listen host port exchange bindingkey Usage: amqp_listen host port exchange bindingkey
``` ```
Start the listener: Start the listener:
``` ```
[root@maxscale-02 examples]# ./amqp_listen 127.0.0.1 5672 foo k1 & [root@maxscale-02 examples]# ./amqp_listen 127.0.0.1 5672 foo k1 &
``` ```
#### Test 3 - Send a message … #### Test 3 - Send a message …
``` ```
[root@maxscale-02 examples]# ./amqp_sendstring [root@maxscale-02 examples]# ./amqp_sendstring
Usage: amqp_sendstring host port exchange routingkey messagebody Usage: amqp_sendstring host port exchange routingkey messagebody
```
[root@maxscale-02 examples]# ./amqp_sendstring 127.0.0.1 5672 foo k1 “This is a new message” [root@maxscale-02 examples]# ./amqp_sendstring 127.0.0.1 5672 foo k1 “This is a new message”
``` ```
... and watch the listener output ... and watch the listener output
``` ```
Delivery 1, exchange foo routingkey k1 Delivery 1, exchange foo routingkey k1
Content-type: text/plain Content-type: text/plain
@ -140,6 +164,7 @@ Content-type: text/plain
A new filter (mqfilter.c) is implemented in order to send messages to the rabbitmq server and a message consumer (rabbitmq_consumer/consumer.c) program will get messages and store them into a MySQL/MariaDB database. A new filter (mqfilter.c) is implemented in order to send messages to the rabbitmq server and a message consumer (rabbitmq_consumer/consumer.c) program will get messages and store them into a MySQL/MariaDB database.
A quick way to install MaxScale with the RabbitMQ filter is to go to the MaxScale source directory and run the following commands: A quick way to install MaxScale with the RabbitMQ filter is to go to the MaxScale source directory and run the following commands:
``` ```
mkdir build mkdir build
cd build cd build
@ -147,17 +172,23 @@ cmake .. -DBUILD_RABBITMQ=Y
make make
make install make install
``` ```
To build the RabbitMQ filter CMake needs an additional parameter: To build the RabbitMQ filter CMake needs an additional parameter:
``` ```
-DBUILD_RABBITMQ=Y -DBUILD_RABBITMQ=Y
``` ```
If the librabbitmq-c library is manually compiled it may be necessary to manually pass the location of the libraries and header files to CMake. If the librabbitmq-c library is manually compiled it may be necessary to manually pass the location of the libraries and header files to CMake.
Libraries: Libraries:
``` ```
-DRABBITMQ_LIBRARIES=<path to RabbitMQ-c libraries> -DRABBITMQ_LIBRARIES=<path to RabbitMQ-c libraries>
``` ```
Headers: Headers:
``` ```
-DRABBITMQ_HEADERS=<path to RabbitMQ-c headers> -DRABBITMQ_HEADERS=<path to RabbitMQ-c headers>
``` ```
@ -165,33 +196,44 @@ Headers:
Please note, Message Queue Consumer (consumer.c) also needs to be compiled with MySQL/MariaDB client libraries in addition to the RabbitMQ-c libraries. If you have your MySQL/MariaDB client libraries and headers in non-standard locations, you can pass them manually to CMake: Please note, Message Queue Consumer (consumer.c) also needs to be compiled with MySQL/MariaDB client libraries in addition to the RabbitMQ-c libraries. If you have your MySQL/MariaDB client libraries and headers in non-standard locations, you can pass them manually to CMake:
Libraries: Libraries:
``` ```
-DMYSQLCLIENT_LIBRARIES=<path to libraries> -DMYSQLCLIENT_LIBRARIES=<path to libraries>
``` ```
Headers: Headers:
``` ```
-DMYSQLCLIENT_HEADERS=<path to headers> -DMYSQLCLIENT_HEADERS=<path to headers>
``` ```
The message queue consumer must be also built as a separate task, it’s not built as part of MaxScale build system. To build it, run the following commands in the rabbitmq_consumer directory in the MaxScale source folder: The message queue consumer must be also built as a separate task, it’s not built as part of MaxScale build system. To build it, run the following commands in the rabbitmq_consumer directory in the MaxScale source folder:
``` ```
mkdir build mkdir build
cd build cd build
cmake .. cmake ..
make make
``` ```
To install it: To install it:
``` ```
make install make install
``` ```
To build packages: To build packages:
``` ```
make package make package
``` ```
This generates RPM or DEB packages based on your system. These packages can then be installed on remote systems for easy access to the data generated by the consumer client. This generates RPM or DEB packages based on your system. These packages can then be installed on remote systems for easy access to the data generated by the consumer client.
## Step 5 - Configure new applications ## Step 5 - Configure new applications
The new filter needs to be configured in maxscale.cnf. The new filter needs to be configured in maxscale.cnf.
``` ```
[Test Service] [Test Service]
type=service type=service
@ -214,6 +256,7 @@ logging_trigger=all
``` ```
Logging triggers define whether to log all or a subset of the incoming queries using these options: Logging triggers define whether to log all or a subset of the incoming queries using these options:
``` ```
# log only some elements or all # log only some elements or all
logging_trigger=[all,source,schema,object] logging_trigger=[all,source,schema,object]
@ -238,6 +281,7 @@ logging_schema=employees,orders,catalog
``` ```
Example: Example:
``` ```
logging_trigger=object,schema,source logging_trigger=object,schema,source
logging_strict=false logging_strict=false
@ -247,28 +291,29 @@ logging_schema=test
logging_source_user=maxtest logging_source_user=maxtest
``` ```
The logging result of the example is: The logging result of the example is:
``` ```
if user maxtest does something, it's logged if user maxtest does something, it's logged
and all queries in test schema are logged and all queries in test schema are logged
anything targeting my1 table is logged anything targeting my1 table is logged
SELECT NOW(), SELECT MD5(“xyz)” are not logged SELECT NOW(), SELECT MD5(“xyz)” are not logged
``` ```
Please note that if we want to log only the user ‘maxtest’ accessing the schema ‘test’ with target ‘my1’ the option logging_strict must be set to TRUE and if we want to include those selects without schema name the option logging_log_all must be set to TRUE. Please note that if we want to log only the user ‘maxtest’ accessing the schema ‘test’ with target ‘my1’ the option logging_strict must be set to TRUE and if we want to include those selects without schema name the option logging_log_all must be set to TRUE.
The mqfilter logs into the MaxScale TRACE log information about the matched logging triggers and the message delivering: The mqfilter logs into the MaxScale TRACE log information about the matched logging triggers and the message delivering:
``` ```
2014 09/03 06:22:04 Trigger is TRG_SOURCE: user: testuser = testuser 2014 09/03 06:22:04 Trigger is TRG_SOURCE: user: testuser = testuser
2014 09/03 06:22:04 Trigger is TRG_SCHEMA: test = test 2014 09/03 06:22:04 Trigger is TRG_SCHEMA: test = test
2014 09/03 06:22:04 Trigger is TRG_OBJECT: test.t1 = t1 2014 09/03 06:22:04 Trigger is TRG_OBJECT: test.t1 = t1
2014 09/03 06:22:04 Routing message to: 127.0.0.1:5672 / as guest/guest, exchange: x1<direct> key:k1 queue:q1 2014 09/03 06:22:04 Routing message to: 127.0.0.1:5672 / as guest/guest, exchange: x1<direct> key:k1 queue:q1
``` ```
The consumer application needs to be configured as well: The consumer application needs to be configured as well:
``` ```
#The options for the consumer are: #The options for the consumer are:
@ -300,22 +345,31 @@ dbname=mqpairs
dbuser=xxx dbuser=xxx
dbpasswd=yyy dbpasswd=yyy
``` ```
We may probably need to modify LD_LIBRARY_PATH before launching ‘consumer’: We may probably need to modify LD_LIBRARY_PATH before launching ‘consumer’:
``` ```
# export LD_LIBRARY_PATH=/packages/rabbitmq-c/rabbitmq-c/librabbitmq:/packages/mariadb_client-2.0.0-Linux/lib/mariadb:/usr/lib64 # export LD_LIBRARY_PATH=/packages/rabbitmq-c/rabbitmq-c/librabbitmq:/packages/mariadb_client-2.0.0-Linux/lib/mariadb:/usr/lib64
``` ```
and finally we can launch it: and finally we can launch it:
``` ```
# ./consumer # ./consumer
``` ```
If the consumer.cnf file is not in the same directory as the binary file is, you can provide the location of the folder that it is in by passing it the -c flag followed by the path: If the consumer.cnf file is not in the same directory as the binary file is, you can provide the location of the folder that it is in by passing it the -c flag followed by the path:
```
# ./consumer -c path/to/file # ./consumer -c path/to/file
```
and start maxScale as well and start maxScale as well
## Step 6 - Test the filter and check collected data ## Step 6 - Test the filter and check collected data
Assuming that MaxScale and the message consumer are successfully running let’s connect to the service with an active mqfilter: Assuming that MaxScale and the message consumer are successfully running let’s connect to the service with an active mqfilter:
``` ```
[root@maxscale-02 MaxScale]# mysql -h 127.0.0.1 -P 4506 -uxxx -pyyy [root@maxscale-02 MaxScale]# mysql -h 127.0.0.1 -P 4506 -uxxx -pyyy
... ...
@ -326,13 +380,12 @@ MariaDB [(none)]> select RAND(3), RAND(5);
| 0.9057697559760601 | 0.40613597483014313 | | 0.9057697559760601 | 0.40613597483014313 |
+--------------------+---------------------+ +--------------------+---------------------+
1 row in set (0.01 sec) 1 row in set (0.01 sec)
… …
MariaDB [(none)]> select RAND(3544), RAND(11); MariaDB [(none)]> select RAND(3544), RAND(11);
``` ```
we can check the consumer output in the terminal where it was started: we can check the consumer output in the terminal where it was started:
``` ```
-------------------------------------------------------------- --------------------------------------------------------------
Received: 1409671452|select @@version_comment limit ? Received: 1409671452|select @@version_comment limit ?
@ -362,9 +415,11 @@ MariaDB [mqpairs]> select * from pairs;
| 01050106010701080109010a010b010c10d | select RAND(?), RAND(?) | Columns: 2 | 2014-09-02 11:24:37 | 2014-09-02 11:24:37 | 1 | | 01050106010701080109010a010b010c10d | select RAND(?), RAND(?) | Columns: 2 | 2014-09-02 11:24:37 | 2014-09-02 11:24:37 | 1 |
+-------------------------------------+----------------------------------+------------+---------------------+---------------------+---------+ +-------------------------------------+----------------------------------+------------+---------------------+---------------------+---------+
7 rows in set (0.01 sec) 7 rows in set (0.01 sec)
``` ```
The filter send queries to the RabbitMQ server in the canonical format, i.e select RAND(?), RAND(?). The filter send queries to the RabbitMQ server in the canonical format, i.e select RAND(?), RAND(?).
The queries Message Queue Consumer application gets from the server are stored with a counter that quickly shows how many times that normalized query was received: The queries Message Queue Consumer application gets from the server are stored with a counter that quickly shows how many times that normalized query was received:
``` ```
| 01050106010701080109010a010b010c10d | select RAND(?), RAND(?) | Columns: 2 | 2014-09-02 11:24:37 | 2014-09-02 11:29:15 | 3 | | 01050106010701080109010a010b010c10d | select RAND(?), RAND(?) | Columns: 2 | 2014-09-02 11:24:37 | 2014-09-02 11:29:15 | 3 |
``` ```