Added template cnf file to /etc and renamed folders.
This commit is contained in:
Markus Makela
131
Filters/Database-Firewall-Filter.md
Normal file
131
Filters/Database-Firewall-Filter.md
Normal file
@ -0,0 +1,131 @@
|
||||
#Database Firewall filter
|
||||
|
||||
## Overview
|
||||
The database firewall filter is used to block queries that match a set of rules. It can be used to prevent harmful queries from reaching the backend database instances or to limit access to the database based on a more flexible set of rules compared to the traditional GRANT-based privilege system. Currently the filter does not support multi-statements.
|
||||
|
||||
## Configuration
|
||||
|
||||
The database firewall filter only requires minimal configuration in the maxscale.cnf file. The actual rules of the database firewall filter are located in a separate text file. The following is an example of a database firewall filter configuration in maxscale.cnf.
|
||||
|
||||
```
|
||||
[DatabaseFirewall]
|
||||
type=filter
|
||||
module=dbfwfilter
|
||||
rules=/home/user/rules.txt
|
||||
|
||||
[Firewalled Routing Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=DatabaseFirewall
|
||||
```
|
||||
|
||||
### Filter Options
|
||||
|
||||
The database firewall filter supports a single option, `ignorecase`. This will set the regular expression matching to case-insensitive mode.
|
||||
|
||||
### Filter Parameters
|
||||
|
||||
The database firewall filter has one mandatory parameter that defines the location of the rule file. This is the `rules` parameter and it expects an absolute path to the rule file.
|
||||
|
||||
## Rule syntax
|
||||
|
||||
The rules are defined by using the following syntax:
|
||||
|
||||
```
|
||||
rule NAME deny [wildcard | columns VALUE ... |
|
||||
regex REGEX | limit_queries COUNT TIMEPERIOD HOLDOFF |
|
||||
no_where_clause] [at_times VALUE...] [on_queries [select|update|insert|delete]]
|
||||
```
|
||||
|
||||
Rules always define a blocking action so the basic mode for the database firewall filter is to allow all queries that do not match a given set of rules. Rules are identified by their name and have a mandatory part and optional parts. You can add comments to the rule files by adding the `#` character at the beginning of the line.
|
||||
|
||||
The first step of defining a rule is to start with the keyword `rule` which identifies this line of text as a rule. The second token is identified as the name of the rule. After that the mandatory token `deny` is required to mark the start of the actual rule definition.
|
||||
|
||||
### Mandatory rule parameters
|
||||
|
||||
The database firewall filter's rules expect a single mandatory parameter for a rule. You can define multiple rules to cover situations where you would like to apply multiple mandatory rules to a query.
|
||||
|
||||
#### `wildcard`
|
||||
|
||||
This rule blocks all queries that use the wildcard character *.
|
||||
|
||||
#### `columns`
|
||||
|
||||
This rule expects a list of values after the `columns` keyword. These values are interpreted as column names and if a query targets any of these, it is blocked.
|
||||
|
||||
#### `regex`
|
||||
|
||||
This rule blocks all queries matching a regex enclosed in single or double quotes.
|
||||
|
||||
#### `limit_queries`
|
||||
|
||||
The limit_queries rule expects three parameters. The first parameter is the number of allowed queries during the time period. The second is the time period in seconds and the third is the amount of time for which the rule is considered active and blocking.
|
||||
|
||||
#### `no_where_clause`
|
||||
|
||||
This rule inspects the query and blocks it if it has no WHERE clause. For example, this would disallow a `DELETE FROM ...` query without a `WHERE` clause. This does not prevent wrongful usage of the `WHERE` clause e.g. `DELETE FROM ... WHERE 1=1`.
|
||||
|
||||
### Optional rule parameters
|
||||
|
||||
Each mandatory rule accepts one or more optional parameters. These are to be defined after the mandatory part of the rule.
|
||||
|
||||
#### `at_times`
|
||||
|
||||
This rule expects a list of time ranges that define the times when the rule in question is active. The time formats are expected to be ISO-8601 compliant and to be separated by a single dash (the - character). For example, to define the active period of a rule to be 5pm to 7pm, you would include `at times 17:00:00-19:00:00` in the rule definition. The rule uses local time to check if the rule is active and has a precision of one second.
|
||||
|
||||
#### `on_queries`
|
||||
|
||||
This limits the rule to be active only on certain types of queries.
|
||||
|
||||
### Applying rules to users
|
||||
|
||||
The `users` directive defines the users to which the rule should be applied.
|
||||
|
||||
`users NAME ... match [any|all|strict_all] rules RULE [,...]`
|
||||
|
||||
The first keyword is `users`, which identifies this line as a user definition line.
|
||||
|
||||
The second component is a list of user names and network addresses in the format *`user`*`@`*`0.0.0.0`*. The first part is the user name and the second part is the network address. You can use the `%` character as the wildcard to enable user name matching from any address or network matching for all users. After the list of users and networks the keyword match is expected.
|
||||
|
||||
After this either the keyword `any` `all` or `strict_all` is expected. This defined how the rules are matched. If `any` is used when the first rule is matched the query is considered blocked and the rest of the rules are skipped. If instead the `all` keyword is used all rules must match for the query to be blocked. The `strict_all` is the same as `all` but it checks the rules from left to right in the order they were listed. If one of these does not match, the rest of the rules are not checked. This could be useful in situations where you would for example combine `limit_queries` and `regex` rules. By using `strict_all` you can have the `regex` rule first and the `limit_queries` rule second. This way the rule only matches if the `regex` rule matches enough times for the `limit_queries` rule to match.
|
||||
|
||||
After the matching part comes the rules keyword after which a list of rule names is expected. This allows reusing of the rules and enables varying levels of query restriction.
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Use Case 1 - Prevent rapid execution of specific queries
|
||||
|
||||
To prevent the excessive use of a database we want to set a limit on the rate of queries. We only want to apply this limit to certain queries that cause unwanted behavior. To achieve this we can use a regular expression.
|
||||
|
||||
First we define the limit on the rate of queries. The first parameter for the rule sets the number of allowed queries to 10 queries and the second parameter sets the rate of sampling to 5 seconds. If a user executes queries faster than this, any further queries that match the regular expression are blocked for 60 seconds.
|
||||
|
||||
```
|
||||
rule limit_rate_of_queries deny limit_queries 10 5 60
|
||||
rule query_regex deny regex '.*select.*from.*user_data.*'
|
||||
```
|
||||
|
||||
To apply these rules we combine them into a single rule by adding a `users` line to the rule file.
|
||||
|
||||
```
|
||||
users %@% match all rules limit_rate_of_queries query_regex
|
||||
```
|
||||
|
||||
### Use Case 2 - Only allow deletes with a where clause
|
||||
|
||||
We have a table which contains all the managers of a company. We want to prevent accidental deletes into this table where the where clause is missing. This poses a problem, we don't want to require all the delete queries to have a where clause. We only want to prevent the data in the managers table from being deleted without a where clause.
|
||||
|
||||
To achieve this, we need two rules. The first rule defines that all delete operations must have a where clause. This rule alone does us no good so we need a second one. The second rule blocks all queries that match a regular expression.
|
||||
|
||||
```
|
||||
rule safe_delete deny no_where_clause on_queries delete
|
||||
rule managers_table deny regex '.*from.*managers.*'
|
||||
```
|
||||
|
||||
When we combine these two rules we get the result we want. To combine these two rules add the following line to the rule file.
|
||||
|
||||
```
|
||||
users %@% match all rules safe_delete managers_table
|
||||
```
|
||||
109
Filters/Query-Log-All-Filter.md
Normal file
109
Filters/Query-Log-All-Filter.md
Normal file
@ -0,0 +1,109 @@
|
||||
# Query Log All Filter
|
||||
|
||||
## Overview
|
||||
|
||||
The Query Log All (QLA) filter is a filter module for MaxScale that is able to log all query content on a per client session basis. Logs are written in a csv format file that lists the time submitted and the SQL statement text.
|
||||
|
||||
## Configuration
|
||||
|
||||
The configuration block for the QLA filter requires the minimal filter options in it's section within the maxscale.cnf file, stored in /etc/maxscale.cnf.
|
||||
```
|
||||
[MyLogFilter]
|
||||
type=filter
|
||||
module=qlafilter
|
||||
|
||||
[MyService]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=MyLogFilter
|
||||
```
|
||||
|
||||
## Filter Options
|
||||
|
||||
The QLA filter accepts one option value, this is the name that is used for the log files that are written. The file that is created appends the session number to the name given in the options entry. For example:
|
||||
|
||||
```
|
||||
options=/tmp/QueryLog
|
||||
```
|
||||
|
||||
would create log files /tmp/QueryLog.1 etc.
|
||||
|
||||
Note, this is included for backward compatibility with the version of the QLA filter that was provided in the initial filters implementation preview in 0.7 of MaxScale. The filebase parameter can now be used and will take precedence over the filter option.
|
||||
|
||||
## Filter Parameters
|
||||
|
||||
The QLA filter accepts a number of optional parameters, these were introduced in the 1.0 release of MaxScale.
|
||||
|
||||
### Filebase
|
||||
|
||||
The basename of the output file created for each session. A session index is added to the filename for each file written.
|
||||
|
||||
```
|
||||
filebase=/tmp/SqlQueryLog
|
||||
```
|
||||
|
||||
The filebase may also be set as the filter, the mechanism to set the filebase via the filter option is superseded by the parameter. If both are set the parameter setting will be used and the filter option ignored.
|
||||
|
||||
### Match
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the QLA filter. The parameter value is a regular expression that is used to match against the SQL text. Only SQL statements that matches the text passed as the value of this parameter will be logged.
|
||||
|
||||
```
|
||||
match=select.*from.*customer.*where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both select, SELECT and any form of the word with upper or lowercase characters.
|
||||
|
||||
### Exclude
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the QLA filter. The parameter value is a regular expression that is used to match against the SQL text. SQL statements that match the text passed as the value of this parameter will be excluded from the log output.
|
||||
|
||||
```
|
||||
exclude=where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore an exclude option of select will exclude statements that contain both select, SELECT or any form of the word with upper or lowercase characters.
|
||||
|
||||
### Source
|
||||
|
||||
The optional source parameter defines an address that is used to match against the address from which the client connection to MaxScale originates. Only sessions that originate from this address will be logged.
|
||||
|
||||
```
|
||||
source=127.0.0.1
|
||||
```
|
||||
|
||||
### User
|
||||
|
||||
The optional user parameter defines a user name that is used to match against the user from which the client connection to MaxScale originates. Only sessions that are connected using this username are logged.
|
||||
|
||||
```
|
||||
user=john
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1 - Query without primary key
|
||||
|
||||
Imagine you have observed an issue with a particular table and you want to determine if there are queries that are accessing that table but not using the primary key of the table. Let's assume the table name is PRODUCTS and the primary key is called PRODUCT_ID. Add a filter with the following definition:
|
||||
|
||||
```
|
||||
[ProductsSelectLogger]
|
||||
type=filter
|
||||
module=qlafilter
|
||||
match=SELECT.*from.*PRODUCTS .*
|
||||
exclude=WHERE.*PRODUCT_ID.*
|
||||
filebase=/var/logs/qla/SelectProducts
|
||||
|
||||
[Product Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=ProductsSelectLogger
|
||||
```
|
||||
|
||||
The result of then putting this filter into the service used by the application would be a log file of all select queries that mentioned the table but did not mention the PRODUCT_ID primary key in the predicates for the query.
|
||||
38
Filters/RabbitMQ-Consumer-Client.md
Normal file
38
Filters/RabbitMQ-Consumer-Client.md
Normal file
@ -0,0 +1,38 @@
|
||||
#RabbitMQ Consumer Client
|
||||
|
||||
## Overview
|
||||
|
||||
This utility tool is used to read messages from a RabbitMQ broker sent by the [RabbitMQ Filter](RabbitMQ-Filter.md) and forward these messages into an SQL database as queries.
|
||||
|
||||
## Command Line Arguments
|
||||
|
||||
The **RabbitMQ Consumer Client** only has one command line argument.
|
||||
|
||||
| Command | Argument |
|
||||
|---------|-------------------------------------------------|
|
||||
| -c | Path to the folder containing the configuration file |
|
||||
|
||||
## Installation
|
||||
|
||||
To install the RabbitMQ Consumer Client you ca either use the provided packages or you can compile it from source code. The source code is included as a part of the MaxScale source code and can be found in the `rabbtmq_consumer` folder. Please refer to the [README](../../rabbitmq_consumer/README) in the folder for more detailed instructions about installation and configuration.
|
||||
|
||||
## Configuration
|
||||
|
||||
The consumer client requires that the `consumer.cnf` configuration file is either be present in the `etc` folder of the installation directory or in the folder specified by the `-c` argument.
|
||||
|
||||
The source broker, the destination database and the message log file can be configured into the separate `consumer.cnf` file.
|
||||
|
||||
| Option | Description |
|
||||
|-----------|---------------------------------------------|
|
||||
| hostname | Hostname of the RabbitMQ server |
|
||||
| port | Port of the RabbitMQ server |
|
||||
| vhost | Virtual host location of the RabbitMQ server |
|
||||
| user | Username for the RabbitMQ server |
|
||||
| passwd | Password for the RabbitMQ server |
|
||||
| queue | Queue to consume from |
|
||||
| dbserver | Hostname of the SQL server |
|
||||
| dbport | Port of the SQL server |
|
||||
| dbname | Name of the SQL database to use |
|
||||
| dbuser | Database username |
|
||||
| dbpasswd | Database password |
|
||||
| logfile | Message log filename |
|
||||
69
Filters/RabbitMQ-Filter.md
Normal file
69
Filters/RabbitMQ-Filter.md
Normal file
@ -0,0 +1,69 @@
|
||||
#RabbitMQ Filter
|
||||
|
||||
## Overview
|
||||
This filter is designed to extract queries and transform them into a canonical form e.g. `INSERT INTO dabata.table VALUES ("John Doe", "Downtown",100,50.0);` turns into `INSERT INTO dabata.table VALUES ("?", "?",?,?);`. The filter pushes these canonized queries and their replies in to a RabbitMQ broker where they can later be retrieved. The retrieval can be done with your own application or the [RabbitMQ Consumer Client](RabbitMQ-Consumer-Client.md) utility tool, which reads the messages from the broker and sends the contents of those messages as SQL queries to a database.
|
||||
|
||||
## Configuration
|
||||
|
||||
The configuration block for the **mqfilter** filter requires the minimal filter options in it’s section within the maxscale.cnf file, stored in /etc/maxscale.cnf. Although the filter will start, it will use the default values which only work with a freshly installed RabbitMQ server and use its default values. This setup is mostly intended for testing the filter.
|
||||
|
||||
The following is an example of a mqfilter configuration in the maxscale.cnf file used for actual logging of queries to a RabbitMQ broker on a different host.
|
||||
|
||||
```
|
||||
[RabbitMQ]
|
||||
type=filter
|
||||
module=mqfilter
|
||||
hostname=192.168.122.100
|
||||
port=4000
|
||||
username=messageuser
|
||||
password=msgpwd
|
||||
exchange=msg-ex-1
|
||||
key=MaxScale
|
||||
logging_trigger=object,schema,source
|
||||
logging_strict=false
|
||||
logging_log_all=false
|
||||
logging_object=my1
|
||||
logging_schema=test
|
||||
logging_source_user=maxtest
|
||||
|
||||
[RabbitMQ Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=RabbitMQ
|
||||
```
|
||||
|
||||
### Filter Options
|
||||
|
||||
The mqfilter filter does not support any filter options.
|
||||
|
||||
### Filter Parameters
|
||||
|
||||
The RabbitMQ filter has parameters to control which queries are logged based on either the attributes of the user or the query itself. These can be combined to to only log queries targeting a certain table in a certain database from a certain user from a certain network address.
|
||||
|
||||
|
||||
Option | Description | Accepted Values | Default |
|
||||
--------|-------------|-----------------|-------------
|
||||
logging_trigger | Set the logging level | `all, source, schema, object` | `all` |
|
||||
logging_strict | Sets whether to trigger when any of the parameters match or only if all parameters match | `true, false` | `false` |
|
||||
logging_log_all | Log only SELECT, UPDATE, DELETE and INSERT or all possible queries | `true, false` | `true` |
|
||||
logging_source_user | Comma-separated list of usernames to log | | |
|
||||
logging_source_host | Comma-separated list of hostnames to log | | |
|
||||
logging_schema | Comma-separated list of databases | | |
|
||||
logging_object | Comma-separated list of database objects |
|
||||
hostname | The server hostname where the messages are sent | | `localhost` |
|
||||
port | Port to send the messages to | | `5672` |
|
||||
username | Server login username | | `guest` |
|
||||
password | Server login password | | `guest` |
|
||||
vhost | The virtual host location on the server, where the messages are sent | | `/` |
|
||||
exchange | The name of the exchange | | `default_exchange` |
|
||||
exchange_type | The type of the exchange | `direct, fanout, topic, headers` | `direct` |
|
||||
key | The routing key used when sending messages to the exchange | | `key` |
|
||||
queue | The queue that will be bound to the used exchange | | |
|
||||
ssl_CA_cert | Path to the CA certificate in PEM format | | |
|
||||
ssl_client_cert | Path to the client cerificate in PEM format | | |
|
||||
ssl_client_key | Path to the client public key in PEM format | | |
|
||||
|
||||
|
||||
106
Filters/Regex-Filter.md
Normal file
106
Filters/Regex-Filter.md
Normal file
@ -0,0 +1,106 @@
|
||||
Regex Filter
|
||||
|
||||
# Overview
|
||||
|
||||
The regex filter is a filter module for MaxScale that is able to rewrite query content using regular expression matches and text substitution.
|
||||
|
||||
# Configuration
|
||||
|
||||
The configuration block for the Regex filter requires the minimal filter options in it’s section within the maxscale.cnf file, stored in /etc/maxscale.cnf.
|
||||
|
||||
```
|
||||
[MyRegexFilter]
|
||||
type=filter
|
||||
module=regexfilter
|
||||
match=some string
|
||||
replace=replacement string
|
||||
|
||||
[MyService]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=MyRegexfilter
|
||||
```
|
||||
|
||||
## Filter Options
|
||||
|
||||
The regex filter accepts the options ignorecase or case. These define if the pattern text should take the case of the string it is matching against into consideration or not.
|
||||
|
||||
## Filter Parameters
|
||||
|
||||
The Regex filter requires two mandatory parameters to be defined.
|
||||
|
||||
### `match`
|
||||
|
||||
A parameter that can be used to match text in the SQL statement which should be replaced.
|
||||
|
||||
```
|
||||
match=TYPE[ ]*=
|
||||
```
|
||||
|
||||
If the filter option ignorecase is used all regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both type, TYPE and any form of the word with upper or lowercase characters.
|
||||
|
||||
### `replace`
|
||||
|
||||
The replace parameter defines the text that should replace the text in the SQL text which matches the match.
|
||||
|
||||
```
|
||||
replace=ENGINE =
|
||||
```
|
||||
|
||||
### `source`
|
||||
|
||||
The optional source parameter defines an address that is used to match against the address from which the client connection to MaxScale originates. Only sessions that originate from this address will have the match and replacement applied to them.
|
||||
|
||||
```
|
||||
source=127.0.0.1
|
||||
```
|
||||
|
||||
### `user`
|
||||
|
||||
The optional user parameter defines a user name that is used to match against the user from which the client connection to MaxScale originates. Only sessions that are connected using this username will have the match and replacement applied to them.
|
||||
|
||||
```
|
||||
user=john
|
||||
```
|
||||
|
||||
### `log_file`
|
||||
|
||||
The optional log_file parameter defines a log file in which the filter writes all queries that are not mached and maching queries with their replacement queries. All sessions will log to this file so this should only be used for diagnostic purposes.
|
||||
|
||||
```
|
||||
log_file=/tmp/regexfilter.log
|
||||
```
|
||||
|
||||
### `log_trace`
|
||||
|
||||
The optional log_trace parameter toggles the logging of non-matching and matching queries with their replacements into the trace log file. This is the preferred method of diagnosing the matching of queries since the trace log can be disabled mid-session if such a need rises.
|
||||
|
||||
```
|
||||
log_trace=true
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1 - Replace MySQL 5.1 create table syntax with that for later versions
|
||||
|
||||
MySQL 5.1 used the parameter TYPE = to set the storage engine that should be used for a table. In later versions this changed to be ENGINE =. Imagine you have an application that you can not change for some reason, but you wish to migrate to a newer version of MySQL. The regexfilter can be used to transform the create table statements into the form that could be used by MySQL 5.5
|
||||
|
||||
```
|
||||
[CreateTableFilter]
|
||||
type=filter
|
||||
module=regexfilter
|
||||
options=ignorecase
|
||||
match=TYPE[ ]*=
|
||||
replace=ENGINE=
|
||||
|
||||
[MyService]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=CreateTableFilter
|
||||
```
|
||||
117
Filters/Tee-Filter.md
Normal file
117
Filters/Tee-Filter.md
Normal file
@ -0,0 +1,117 @@
|
||||
TEE Filter
|
||||
|
||||
# Overview
|
||||
|
||||
The tee filter is a filter module for MaxScale is a "plumbing" fitting in the MaxScale filter toolkit. It can be used in a filter pipeline of a service to make a copy of requests from the client and dispatch a copy of the request to another service within MaxScale.
|
||||
|
||||
# Configuration
|
||||
|
||||
The configuration block for the TEE filter requires the minimal filter parameters in it’s section within the maxscale.cnf file, stored in /etc/maxscale.cnf, that defines the filter to load and the service to send the duplicates to. Currently the tee filter does not support multi-statements.
|
||||
|
||||
```
|
||||
[DataMartFilter]
|
||||
type=filter
|
||||
module=tee
|
||||
service=DataMart
|
||||
|
||||
[Data Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=DataMartFilter
|
||||
```
|
||||
|
||||
## Filter Options
|
||||
|
||||
The tee filter does not support any filter options.
|
||||
|
||||
## Filter Parameters
|
||||
|
||||
The tee filter requires a mandatory parameter to define the service to replicate statements to and accepts a number of optional parameters.
|
||||
|
||||
### Match
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be replicated by the tee filter. The parameter value is a regular expression that is used to match against the SQL text. Only SQL statements that matches the text passed as the value of this parameter will be sent to the service defined in the filter section.
|
||||
|
||||
```
|
||||
match=insert.*into.*order*
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both insert, INSERT and any form of the word with upper or lowercase characters.
|
||||
|
||||
### Exclude
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be replicated by the tee filter. The parameter value is a regular expression that is used to match against the SQL text. SQL statements that match the text passed as the value of this parameter will be excluded from the replication stream.
|
||||
|
||||
```
|
||||
exclude=select
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore an exclude option of select will exclude statements that contain both select, SELECT or any form of the word with upper or lowercase characters.
|
||||
|
||||
### Source
|
||||
|
||||
The optional source parameter defines an address that is used to match against the address from which the client connection to MaxScale originates. Only sessions that originate from this address will be replicated.
|
||||
|
||||
```
|
||||
source=127.0.0.1
|
||||
```
|
||||
|
||||
### User
|
||||
|
||||
The optional user parameter defines a user name that is used to match against the user from which the client connection to MaxScale originates. Only sessions that are connected using this username are replicated.
|
||||
|
||||
```
|
||||
user=john
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1 - Replicate all inserts into the orders table
|
||||
|
||||
Assume an order processing system that has a table called orders. You also have another database server, the datamart server, that requires all inserts into orders to be replicated to it. Deletes and updates are not however required.
|
||||
|
||||
Set up a service in MaxScale, called Orders, to communicate with the order processing system with the tee filter applied to it. Also set up a service to talk the datamart server, using the DataMart service. The tee filter would have as it’s service entry the DataMart service, by adding a match parameter of "insert into orders" would then result in all requests being sent to the order processing system, and insert statements that include the orders table being additionally sent to the datamart server.
|
||||
|
||||
```
|
||||
[Orders]
|
||||
type=service
|
||||
router=readconnroute
|
||||
servers=server1, server2, server3, server4
|
||||
user=massi
|
||||
passwd=6628C50E07CCE1F0392EDEEB9D1203F3
|
||||
filters=ReplicateOrders
|
||||
|
||||
[ReplicateOrders]
|
||||
type=filter
|
||||
module=tee
|
||||
service=DataMart
|
||||
match=insert[ ]*into[ ]*orders
|
||||
|
||||
[DataMart]
|
||||
type=service
|
||||
router=readconnroute
|
||||
servers=datamartserver
|
||||
user=massi
|
||||
passwd=6628C50E07CCE1F0392EDEEB9D1203F3
|
||||
filters=QLA_DataMart
|
||||
|
||||
[QLA_DataMart]
|
||||
type=filter
|
||||
module=qlafilter
|
||||
options=/var/log/DataMart/InsertsLog
|
||||
|
||||
[Orders Listener]
|
||||
type=listener
|
||||
service=Orders
|
||||
protocol=MySQLClient
|
||||
port=4011
|
||||
|
||||
[DataMart Listener]
|
||||
type=listener
|
||||
service=DataMart
|
||||
protocol=MySQLClient
|
||||
port=4012
|
||||
```
|
||||
203
Filters/Top-N-Filter.md
Normal file
203
Filters/Top-N-Filter.md
Normal file
@ -0,0 +1,203 @@
|
||||
Top Filter
|
||||
|
||||
# Overview
|
||||
|
||||
The top filter is a filter module for MaxScale that monitors every SQL statement that passes through the filter. It measures the duration of that statement, the time between the statement being sent and the first result being returned. The top N times are kept, along with the SQL text itself and a list sorted on the execution times of the query is written to a file upon closure of the client session.
|
||||
|
||||
# Configuration
|
||||
|
||||
The configuration block for the TOP filter requires the minimal filter options in it’s section within the maxscale.cnf file, stored in /etc/maxscale.cnf.
|
||||
|
||||
```
|
||||
[MyLogFilter]
|
||||
type=filter
|
||||
module=topfilter
|
||||
|
||||
[Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=MyLogFilter
|
||||
```
|
||||
|
||||
## Filter Options
|
||||
|
||||
The top filter does not support any filter options currently.
|
||||
|
||||
## Filter Parameters
|
||||
|
||||
The top filter accepts a number of optional parameters.
|
||||
|
||||
### Filebase
|
||||
|
||||
The basename of the output file created for each session. A session index is added to the filename for each file written.
|
||||
|
||||
```
|
||||
filebase=/tmp/SqlQueryLog
|
||||
```
|
||||
|
||||
The filebase may also be set as the filter, the mechanism to set the filebase via the filter option is superseded by the parameter. If both are set the parameter setting will be used and the filter option ignored.
|
||||
|
||||
### Count
|
||||
|
||||
The number of SQL statements to store and report upon.
|
||||
|
||||
```
|
||||
count=30
|
||||
```
|
||||
|
||||
The default value for the number of statements recorded is 10.
|
||||
|
||||
### Match
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the top filter. The parameter value is a regular expression that is used to match against the SQL text. Only SQL statements that matches the text passed as the value of this parameter will be logged.
|
||||
|
||||
```
|
||||
match=select.*from.*customer.*where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both select, SELECT and any form of the word with upper or lowercase characters.
|
||||
|
||||
### Exclude
|
||||
|
||||
An optional parameter that can be used to limit the queries that will be logged by the top filter. The parameter value is a regular expression that is used to match against the SQL text. SQL statements that match the text passed as the value of this parameter will be excluded from the log output.
|
||||
|
||||
```
|
||||
exclude=where
|
||||
```
|
||||
|
||||
All regular expressions are evaluated with the option to ignore the case of the text, therefore an exclude option of select will exclude statements that contain both where, WHERE or any form of the word with upper or lowercase characters.
|
||||
|
||||
### Source
|
||||
|
||||
The optional source parameter defines an address that is used to match against the address from which the client connection to MaxScale originates. Only sessions that originate from this address will be logged.
|
||||
|
||||
```
|
||||
source=127.0.0.1
|
||||
```
|
||||
|
||||
### User
|
||||
|
||||
The optional user parameter defines a user name that is used to match against the user from which the client connection to MaxScale originates. Only sessions that are connected using this username will result in results being generated.
|
||||
|
||||
```
|
||||
user=john
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1 - Heavily Contended Table
|
||||
|
||||
You have an order system and believe the updates of the PRODUCTS table is causing some performance issues for the rest of your application. You would like to know which of the many updates in your application is causing the issue.
|
||||
|
||||
Add a filter with the following definition;
|
||||
|
||||
```
|
||||
[ProductsUpdateTop20]
|
||||
type=filter
|
||||
module=topfilter
|
||||
count=20
|
||||
match=UPDATE.*PRODUCTS.*WHERE
|
||||
exclude=UPDATE.*PRODUCTS_STOCK.*WHERE
|
||||
filebase=/var/logs/top/ProductsUpdate
|
||||
```
|
||||
|
||||
Note the exclude entry, this is to prevent updates to the PRODUCTS_STOCK table from being included in the report.
|
||||
|
||||
### Example 2 - One Application Server is Slow
|
||||
|
||||
One of your applications servers is slower than the rest, you believe it is related to database access but you not not sure what is taking the time.
|
||||
|
||||
Add a filter with the following definition;
|
||||
|
||||
```
|
||||
[SlowAppServer]
|
||||
type=filter
|
||||
module=topfilter
|
||||
count=20
|
||||
source=192.168.0.32
|
||||
filebase=/var/logs/top/SlowAppServer
|
||||
```
|
||||
|
||||
In order to produce a comparison with an unaffected application server you can also add a second filter as a control.
|
||||
|
||||
```
|
||||
[ControlAppServer]
|
||||
|
||||
type=filter
|
||||
module=topfilter
|
||||
count=20
|
||||
source=192.168.0.42
|
||||
filebase=/var/logs/top/ControlAppServer
|
||||
```
|
||||
|
||||
In the service definition add both filters
|
||||
|
||||
```
|
||||
[App Service]
|
||||
type=service
|
||||
router=readconnrouter
|
||||
servers=server1
|
||||
user=myuser
|
||||
passwd=mypasswd
|
||||
filters=SlowAppServer | ControlAppServer
|
||||
```
|
||||
|
||||
You will then have two sets of logs files written, one which profiles the top 20 queries of the slow application server and another that gives you the top 20 queries of your control application server. These two sets of files can then be compared to determine what if anything is different between the two.
|
||||
|
||||
# Output Report
|
||||
|
||||
The following is an example report for a number of fictitious queries executed against the employees example database available for MySQL.
|
||||
|
||||
```
|
||||
-bash-4.1$ cat /var/logs/top/Employees-top-10.137
|
||||
|
||||
Top 10 longest running queries in session.
|
||||
|
||||
==========================================
|
||||
|
||||
Time (sec) | Query
|
||||
|
||||
-----------+-----------------------------------------------------------------
|
||||
|
||||
22.985 | select sum(salary), year(from_date) from salaries s, (select distinct year(from_date) as y1 from salaries) y where (makedate(y.y1, 1) between s.from_date and s.to_date) group by y.y1
|
||||
|
||||
5.304 | select d.dept_name as "Department", y.y1 as "Year", count(*) as "Count" from departments d, dept_emp de, (select distinct year(from_date) as y1 from dept_emp order by 1) y where d.dept_no = de.dept_no and (makedate(y.y1, 1) between de.from_date and de.to_date) group by y.y1, d.dept_name order by 1, 2
|
||||
|
||||
2.896 | select year(now()) - year(birth_date) as age, gender, avg(salary) as "Average Salary" from employees e, salaries s where e.emp_no = s.emp_no and ("1988-08-01" between from_date AND to_date) group by year(now()) - year(birth_date), gender order by 1,2
|
||||
|
||||
2.160 | select dept_name as "Department", sum(salary) / 12 as "Salary Bill" from employees e, departments d, dept_emp de, salaries s where e.emp_no = de.emp_no and de.dept_no = d.dept_no and ("1988-08-01" between de.from_date AND de.to_date) and ("1988-08-01" between s.from_date AND s.to_date) and s.emp_no = e.emp_no group by dept_name order by 1
|
||||
|
||||
0.845 | select dept_name as "Department", avg(year(now()) - year(birth_date)) as "Average Age", gender from employees e, departments d, dept_emp de where e.emp_no = de.emp_no and de.dept_no = d.dept_no and ("1988-08-01" between from_date AND to_date) group by dept_name, gender
|
||||
|
||||
0.668 | select year(hire_date) as "Hired", d.dept_name, count(*) as "Count" from employees e, departments d, dept_emp de where de.emp_no = e.emp_no and de.dept_no = d.dept_no group by d.dept_name, year(hire_date)
|
||||
|
||||
0.249 | select moves.n_depts As "No. of Departments", count(moves.emp_no) as "No. of Employees" from (select de1.emp_no as emp_no, count(de1.emp_no) as n_depts from dept_emp de1 group by de1.emp_no) as moves group by moves.n_depts order by 1
|
||||
|
||||
0.245 | select year(now()) - year(birth_date) as age, gender, count(*) as "Count" from employees group by year(now()) - year(birth_date), gender order by 1,2
|
||||
|
||||
0.179 | select year(hire_date) as "Hired", count(*) as "Count" from employees group by year(hire_date)
|
||||
|
||||
0.160 | select year(hire_date) - year(birth_date) as "Age", count(*) as Count from employees group by year(hire_date) - year(birth_date) order by 1
|
||||
|
||||
-----------+-----------------------------------------------------------------
|
||||
|
||||
Session started Wed Jun 18 18:41:03 2014
|
||||
|
||||
Connection from 127.0.0.1
|
||||
|
||||
Username massi
|
||||
|
||||
Total of 24 statements executed.
|
||||
|
||||
Total statement execution time 35.701 seconds
|
||||
|
||||
Average statement execution time 1.488 seconds
|
||||
|
||||
Total connection time 46.500 seconds
|
||||
|
||||
-bash-4.1$
|
||||
|
||||
```
|
||||
Reference in New Issue
Block a user