Cleaned up filter documents and added clarification about service and filter interaction.
This commit is contained in:
Markus Makela
@ -44,6 +44,8 @@
|
||||
|
||||
## Filters
|
||||
|
||||
Here are detailed documents about the filters MaxScale offers. They contain configuration guides and example use cases. Before reading these,you should have read the filter tutorial so that you know how they work and how to configure them.
|
||||
|
||||
- [Query Log All](filters/Query-Log-All-Filter.md)
|
||||
- [Regex Filter](filters/Regex-Filter.md)
|
||||
- [Tee Filter](filters/Tee-Filter.md)
|
||||
|
||||
@ -12,6 +12,14 @@ The database firewall filter only requires minimal configuration in the MaxScale
|
||||
type=filter
|
||||
module=dbfwfilter
|
||||
rules=/home/user/rules.txt
|
||||
|
||||
[Firewalled Routing Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=DatabaseFirewall
|
||||
```
|
||||
|
||||
### Filter Options
|
||||
|
||||
@ -7,16 +7,19 @@ The Query Log All (QLA) filter is a filter module for MaxScale that is
|
||||
## Configuration
|
||||
|
||||
The configuration block for the QLA filter requires the minimal filter options in it's section within the MaxScale.cnf file, stored in $MAXSCALE_HOME/etc/MaxScale.cnf.
|
||||
|
||||
```
|
||||
[MyLogFilter]
|
||||
type=filter
|
||||
module=qlafilter
|
||||
```
|
||||
|
||||
## Filter Options
|
||||
|
||||
The QLA filter accepts one option value, this is the name that is used for the log files that are written. The file that is created appends the session number to the name given in the options entry. For example:
|
||||
|
||||
```
|
||||
options=/tmp/QueryLog
|
||||
```
|
||||
|
||||
would create log files /tmp/QueryLog.1 etc.
|
||||
|
||||
@ -30,7 +33,9 @@ The QLA filter accepts a number of optional parameters, these were in
|
||||
|
||||
The basename of the output file created for each session. A session index is added to the filename for each file written.
|
||||
|
||||
```
|
||||
filebase=/tmp/SqlQueryLog
|
||||
```
|
||||
|
||||
The filebase may also be set as the filter, the mechanism to set the filebase via the filter option is superseded by the parameter. If both are set the parameter setting will be used and the filter option ignored.
|
||||
|
||||
@ -38,7 +43,9 @@ The filebase may also be set as the filter, the mechanism to set th
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the QLA filter. The parameter value is a regular expression that is used to match against the SQL text. Only SQL statements that matches the text passed as the value of this parameter will be logged.
|
||||
|
||||
```
|
||||
match=select.*from.*customer.*where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both select, SELECT and any form of the word with upper or lowercase characters.
|
||||
|
||||
@ -46,7 +53,9 @@ All regular expressions are evaluated with the option to ignore the c
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the QLA filter. The parameter value is a regular expression that is used to match against the SQL text. SQL statements that match the text passed as the value of this parameter will be excluded from the log output.
|
||||
|
||||
```
|
||||
exclude=where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore an exclude option of select will exclude statements that contain both select, SELECT or any form of the word with upper or lowercase characters.
|
||||
|
||||
@ -54,13 +63,17 @@ All regular expressions are evaluated with the option to ignore the c
|
||||
|
||||
The optional source parameter defines an address that is used to match against the address from which the client connection to MaxScale originates. Only sessions that originate from this address will be logged.
|
||||
|
||||
```
|
||||
source=127.0.0.1
|
||||
```
|
||||
|
||||
### User
|
||||
|
||||
The optional user parameter defines a user name that is used to match against the user from which the client connection to MaxScale originates. Only sessions that are connected using this username are logged.
|
||||
|
||||
```
|
||||
user=john
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
@ -68,6 +81,7 @@ The optional user parameter defines a user name that is used to matc
|
||||
|
||||
Imagine you have observed an issue with a particular table and you want to determine if there are queries that are accessing that table but not using the primary key of the table. Let's assume the table name is PRODUCTS and the primary key is called PRODUCT_ID. Add a filter with the following definition:
|
||||
|
||||
```
|
||||
[ProductsSelectLogger]
|
||||
type=filter
|
||||
module=qlafilter
|
||||
@ -75,5 +89,13 @@ Imagine you have observed an issue with a particular table and you w
|
||||
exclude=WHERE.*PRODUCT_ID.*
|
||||
filebase=/var/logs/qla/SelectProducts
|
||||
|
||||
The result of then putting this filter into the service used by the application would be a log file of all select queries that mentioned the table but did not mention the PRODUCT_ID primary key in the predicates for the query.
|
||||
[Product Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=ProductsSelectLogger
|
||||
```
|
||||
|
||||
The result of then putting this filter into the service used by the application would be a log file of all select queries that mentioned the table but did not mention the PRODUCT_ID primary key in the predicates for the query.
|
||||
|
||||
@ -25,6 +25,14 @@ logging_log_all=false
|
||||
logging_object=my1
|
||||
logging_schema=test
|
||||
logging_source_user=maxtest
|
||||
|
||||
[RabbitMQ Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=RabbitMQ
|
||||
```
|
||||
|
||||
### Filter Options
|
||||
|
||||
@ -8,14 +8,21 @@ The tee filter is a filter module for MaxScale is a "plumbing" fitting in the Ma
|
||||
|
||||
The configuration block for the TEE filter requires the minimal filter parameters in it’s section within the MaxScale.cnf file, stored in $MAXSCALE_HOME/etc/MaxScale.cnf, that defines the filter to load and the service to send the duplicates to. Currently the tee filter does not support multi-statements.
|
||||
|
||||
```
|
||||
[DataMartFilter]
|
||||
|
||||
type=filter
|
||||
|
||||
module=tee
|
||||
|
||||
service=DataMart
|
||||
|
||||
[Data Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=DataMartFilter
|
||||
```
|
||||
|
||||
## Filter Options
|
||||
|
||||
The tee filter does not support any filter options.
|
||||
@ -28,7 +35,9 @@ The tee filter requires a mandatory parameter to define the service to replicate
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be replicated by the tee filter. The parameter value is a regular expression that is used to match against the SQL text. Only SQL statements that matches the text passed as the value of this parameter will be sent to the service defined in the filter section.
|
||||
|
||||
```
|
||||
match=insert.*into.*order*
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both insert, INSERT and any form of the word with upper or lowercase characters.
|
||||
|
||||
@ -36,7 +45,9 @@ All regular expressions are evaluated with the option to ignore the case of the
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be replicated by the tee filter. The parameter value is a regular expression that is used to match against the SQL text. SQL statements that match the text passed as the value of this parameter will be excluded from the replication stream.
|
||||
|
||||
```
|
||||
exclude=select
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore an exclude option of select will exclude statements that contain both select, SELECT or any form of the word with upper or lowercase characters.
|
||||
|
||||
@ -44,13 +55,17 @@ All regular expressions are evaluated with the option to ignore the case of the
|
||||
|
||||
The optional source parameter defines an address that is used to match against the address from which the client connection to MaxScale originates. Only sessions that originate from this address will be replicated.
|
||||
|
||||
```
|
||||
source=127.0.0.1
|
||||
```
|
||||
|
||||
### User
|
||||
|
||||
The optional user parameter defines a user name that is used to match against the user from which the client connection to MaxScale originates. Only sessions that are connected using this username are replicated.
|
||||
|
||||
```
|
||||
user=john
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
@ -60,69 +75,43 @@ Assume an order processing system that has a table called orders. You also have
|
||||
|
||||
Set up a service in MaxScale, called Orders, to communicate with the order processing system with the tee filter applied to it. Also set up a service to talk the datamart server, using the DataMart service. The tee filter would have as it’s service entry the DataMart service, by adding a match parameter of "insert into orders" would then result in all requests being sent to the order processing system, and insert statements that include the orders table being additionally sent to the datamart server.
|
||||
|
||||
```
|
||||
[Orders]
|
||||
|
||||
type=service
|
||||
|
||||
router=readconnroute
|
||||
|
||||
servers=server1, server2, server3, server4
|
||||
|
||||
user=massi
|
||||
|
||||
passwd=6628C50E07CCE1F0392EDEEB9D1203F3
|
||||
|
||||
filters=ReplicateOrders
|
||||
|
||||
[ReplicateOrders]
|
||||
|
||||
type=filter
|
||||
|
||||
module=tee
|
||||
|
||||
service=DataMart
|
||||
|
||||
match=insert[ ]*into[ ]*orders
|
||||
|
||||
[DataMart]
|
||||
|
||||
type=service
|
||||
|
||||
router=readconnroute
|
||||
|
||||
servers=datamartserver
|
||||
|
||||
user=massi
|
||||
|
||||
passwd=6628C50E07CCE1F0392EDEEB9D1203F3
|
||||
|
||||
filters=QLA_DataMart
|
||||
|
||||
[QLA_DataMart]
|
||||
|
||||
type=filter
|
||||
|
||||
module=qlafilter
|
||||
|
||||
options=/var/log/DataMart/InsertsLog
|
||||
|
||||
[Orders Listener]
|
||||
|
||||
type=listener
|
||||
|
||||
service=Orders
|
||||
|
||||
protocol=MySQLClient
|
||||
|
||||
port=4011
|
||||
|
||||
[DataMart Listener]
|
||||
|
||||
type=listener
|
||||
|
||||
service=DataMart
|
||||
|
||||
protocol=MySQLClient
|
||||
|
||||
port=4012
|
||||
|
||||
```
|
||||
|
||||
@ -8,12 +8,20 @@ The top filter is a filter module for MaxScale that monitors every SQL statement
|
||||
|
||||
The configuration block for the TOP filter requires the minimal filter options in it’s section within the MaxScale.cnf file, stored in $MAXSCALE_HOME/etc/MaxScale.cnf.
|
||||
|
||||
```
|
||||
[MyLogFilter]
|
||||
|
||||
type=filter
|
||||
|
||||
module=topfilter
|
||||
|
||||
[Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=MyLogFilter
|
||||
```
|
||||
|
||||
## Filter Options
|
||||
|
||||
The top filter does not support any filter options currently.
|
||||
@ -26,7 +34,9 @@ The top filter accepts a number of optional parameters.
|
||||
|
||||
The basename of the output file created for each session. A session index is added to the filename for each file written.
|
||||
|
||||
```
|
||||
filebase=/tmp/SqlQueryLog
|
||||
```
|
||||
|
||||
The filebase may also be set as the filter, the mechanism to set the filebase via the filter option is superseded by the parameter. If both are set the parameter setting will be used and the filter option ignored.
|
||||
|
||||
@ -34,7 +44,9 @@ The filebase may also be set as the filter, the mechanism to set the filebase vi
|
||||
|
||||
The number of SQL statements to store and report upon.
|
||||
|
||||
```
|
||||
count=30
|
||||
```
|
||||
|
||||
The default value for the number of statements recorded is 10.
|
||||
|
||||
@ -42,7 +54,9 @@ The default value for the number of statements recorded is 10.
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the top filter. The parameter value is a regular expression that is used to match against the SQL text. Only SQL statements that matches the text passed as the value of this parameter will be logged.
|
||||
|
||||
```
|
||||
match=select.*from.*customer.*where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both select, SELECT and any form of the word with upper or lowercase characters.
|
||||
|
||||
@ -50,7 +64,9 @@ All regular expressions are evaluated with the option to ignore the case of the
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the top filter. The parameter value is a regular expression that is used to match against the SQL text. SQL statements that match the text passed as the value of this parameter will be excluded from the log output.
|
||||
|
||||
```
|
||||
exclude=where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore an exclude option of select will exclude statements that contain both where, WHERE or any form of the word with upper or lowercase characters.
|
||||
|
||||
@ -58,13 +74,17 @@ All regular expressions are evaluated with the option to ignore the case of the
|
||||
|
||||
The optional source parameter defines an address that is used to match against the address from which the client connection to MaxScale originates. Only sessions that originate from this address will be logged.
|
||||
|
||||
```
|
||||
source=127.0.0.1
|
||||
```
|
||||
|
||||
### User
|
||||
|
||||
The optional user parameter defines a user name that is used to match against the user from which the client connection to MaxScale originates. Only sessions that are connected using this username will result in results being generated.
|
||||
|
||||
```
|
||||
user=john
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
@ -74,19 +94,15 @@ You have an order system and believe the updates of the PRODUCTS table is causin
|
||||
|
||||
Add a filter with the following definition;
|
||||
|
||||
```
|
||||
[ProductsUpdateTop20]
|
||||
|
||||
type=filter
|
||||
|
||||
module=topfilter
|
||||
|
||||
count=20
|
||||
|
||||
match=UPDATE.*PRODUCTS.*WHERE
|
||||
|
||||
exclude=UPDATE.*PRODUCTS_STOCK.*WHERE
|
||||
|
||||
filebase=/var/logs/top/ProductsUpdate
|
||||
```
|
||||
|
||||
Note the exclude entry, this is to prevent updates to the PRODUCTS_STOCK table from being included in the report.
|
||||
|
||||
@ -96,35 +112,38 @@ One of your applications servers is slower than the rest, you believe it is rela
|
||||
|
||||
Add a filter with the following definition;
|
||||
|
||||
```
|
||||
[SlowAppServer]
|
||||
|
||||
type=filter
|
||||
|
||||
module=topfilter
|
||||
|
||||
count=20
|
||||
|
||||
source=192.168.0.32
|
||||
|
||||
filebase=/var/logs/top/SlowAppServer
|
||||
```
|
||||
|
||||
In order to produce a comparison with an unaffected application server you can also add a second filter as a control.
|
||||
|
||||
```
|
||||
[ControlAppServer]
|
||||
|
||||
type=filter
|
||||
|
||||
module=topfilter
|
||||
|
||||
count=20
|
||||
|
||||
source=192.168.0.42
|
||||
|
||||
filebase=/var/logs/top/ControlAppServer
|
||||
```
|
||||
|
||||
In the router definition add both filters
|
||||
In the service definition add both filters
|
||||
|
||||
```
|
||||
[App Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=SlowAppServer | ControlAppServer
|
||||
```
|
||||
|
||||
You will then have two sets of logs files written, one which profiles the top 20 queries of the slow application server and another that gives you the top 20 queries of your control application server. These two sets of files can then be compared to determine what if anything is different between the two.
|
||||
|
||||
@ -132,6 +151,7 @@ You will then have two sets of logs files written, one which profiles the top 20
|
||||
|
||||
The following is an example report for a number of fictitious queries executed against the employees example database available for MySQL.
|
||||
|
||||
```
|
||||
-bash-4.1$ cat /var/logs/top/Employees-top-10.137
|
||||
|
||||
Top 10 longest running queries in session.
|
||||
@ -180,3 +200,4 @@ Total connection time 46.500 seconds
|
||||
|
||||
-bash-4.1$
|
||||
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user