hcq/MaxScale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch '2.3' into develop

Browse Source
This commit is contained in:
Esa Korhonen
2019-05-21 11:18:40 +03:00
parent cf46004bd8 bb706394f6
commit 0af1f6e690
11 changed files with 146 additions and 246 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
Documentation/Filters/BinlogFilter.md
View File

@ -7,7 +7,7 @@ This filter was introduced in MariaDB MaxScale 2.3.0.
The `binlogfilter` can be combined with a `binlogrouter` service to selectively The `binlogfilter` can be combined with a `binlogrouter` service to selectively
replicate the binary log events to slave servers. replicate the binary log events to slave servers.
The filter uses two parameters, `match` and `exclude`, to decide which events The filter uses two parameters, *match* and *exclude*, to decide which events
are replicated. If a binlog event does not match or is excluded, the event is are replicated. If a binlog event does not match or is excluded, the event is
replaced with an empty data event. The empty event is always 35 bytes which replaced with an empty data event. The empty event is always 35 bytes which
translates to a space reduction in most cases. translates to a space reduction in most cases.
@ -18,33 +18,25 @@ that there are no ambiguities in the event filtering.
## Configuration ## Configuration
Both the `match` and `exclude` parameters are optional. If neither of them is ### `match` and `exclude`
defined, the filter does nothing and all events are replicated.
Both the *match* and *exclude* parameters are optional and work mostly as other
[typical regular expression parameters](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters).
If neither of them is defined, the filter does nothing and all events are replicated. This
filter does not accept regular expression options as a separate parameter, such settings
must be defined in the patterns themselves. See the
[PCRE2 api documentation](https://www.pcre.org/current/doc/html/pcre2api.html#SEC20) for
more information.
The two parameters are matched against the database and table name concatenated The two parameters are matched against the database and table name concatenated
with a period. For example, the string the patterns are matched against for the with a period. For example, the string the patterns are matched against for the
database `test` and table `t1` is `test.t1`. database `test` and table `t1` is `test.t1`.
For statement based replication, the pattern is matched against all the tables For statement based replication, the pattern is matched against all the tables
in the statements. If any of the tables matches the `match` pattern, the event in the statements. If any of the tables matches the *match* pattern, the event
is replicated. If any of the tables matches the `exclude` pattern, the event is is replicated. If any of the tables matches the *exclude* pattern, the event is
not replicated. not replicated.
### `match`
A [PCRE2 regular expression](../Getting-Started/Configuration-Guide.md#regular-expressions)
that is matched against the database and table name. If the pattern matches, the
event is replicated to the slave. If no `match` parameter is defined, all events
are considered to match.
### `exclude`
A [PCRE2 regular expression](../Getting-Started/Configuration-Guide.md#regular-expressions)
that is matched against the database and table name. If the pattern matches, the
event is excluded and is not replicated to the slave. If no `exclude` pattern is
defined, the event filtering is controlled completely by the `match` parameter.
## Example Configuration ## Example Configuration
With the following configuration, only events belonging to database `customers` With the following configuration, only events belonging to database `customers`

38
Documentation/Filters/CCRFilter.md
View File

@ -35,22 +35,6 @@ comment. The `match`-comment typically has no effect, since write queries by
default trigger the filter anyway. It can be used to override an ignore-type default trigger the filter anyway. It can be used to override an ignore-type
regular expression that would othewise prevent triggering. regular expression that would othewise prevent triggering.
## Filter Options
The CCR filter accepts the following options.
|Option |Description |
|-----------|--------------------------------------------|
|ignorecase |Use case-insensitive matching (default) |
|case |Use case-sensitive matching |
|extended |Use extended regular expression syntax (ERE)|
To use multiple filter options, list them in a comma-separated list.
```
options=case,extended
```
## Filter Parameters ## Filter Parameters
The CCR filter has no mandatory parameters. The CCR filter has no mandatory parameters.
@ -86,27 +70,17 @@ the counter reaches zero, the statements are routed normally. If a new data
modifying SQL statement is processed, the counter is reset to the value of modifying SQL statement is processed, the counter is reset to the value of
_count_. _count_.
### `match` ### `match`, `ignore` and `options`
An optional parameter that can be used to control which statements trigger the These [regular expression settings](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters)
statement re-routing. The parameter value is a regular expression that is used control which statements trigger statement re-routing. Only non-SELECT statements are
to match against the SQL text. Only non-SELECT statements are inspected. If this inspected. For CCRFilter, the *exclude*-parameter is instead named *ignore*, yet works
parameter is defined, *only* matching SQL-queries will trigger the filter similarly.
(assuming no ccr hint comments in the query).
``` ```
match=.*INSERT.* match=.*INSERT.*
```
### `ignore`
An optional parameter that can be used to control which statements don't trigger
the statement re-routing. This does the opposite of the _match_ parameter. The
parameter value is a regular expression that is used to match against the SQL
text. Only non-SELECT statements are inspected.
```
ignore=.*UPDATE.* ignore=.*UPDATE.*
options=case,extended
``` ```
## Example Configuration ## Example Configuration

9
Documentation/Filters/Database-Firewall-Filter.md
View File

@ -318,10 +318,11 @@ rule examplerule match not_function columns ssn
#### `regex` #### `regex`
This rule blocks all queries matching a regex enclosed in single or double This rule blocks all queries matching the regular expression. The regex string expects a
quotes. The regex string expects a PCRE2 syntax regular expression. For more PCRE2 syntax regular expression. For more information about PCRE2 syntax, read the
information about the PCRE2 syntax, read the [PCRE2 [PCRE2 documentation](http://www.pcre.org/current/doc/html/pcre2syntax.html). Unlike
documentation](http://www.pcre.org/current/doc/html/pcre2syntax.html). typical MaxScale regex parameters, the value should be enclosed in single or double
quotes, not in `/.../`. Any compilation options must be included in the pattern itself.
##### Example ##### Example

45
Documentation/Filters/Named-Server-Filter.md
View File

@ -50,23 +50,29 @@ filters=NamedServerFilter
## Filter Parameters ## Filter Parameters
The NamedServerFilter requires two mandatory parameters. NamedServerFilter requires at least one *matchXY* - *targetXY* pair.
### `matchXY` ### `matchXY`, `options`
Regular expression the SQL-query is matched against. XY must be a number in the *matchXY* defines a
range 01 - 25. Each *match* setting must have a similarly indexed *target* [PCRE2 regular expression](../Getting-Started/Configuration-Guide.md#regular-expressions)
setting. against which the incoming SQL query is matched. *XY* must be a number in the range
01 - 25. Each *match*-setting pairs with a similarly indexed *target*- setting. If one is
defined, the other must be defined as well. If a query matches the pattern, the filter
attaches a routing hint defined by the *target*-setting to the query. The
*options*-parameter affects how the patterns are compiled as
[usual](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters).
``` ```
match01=^SELECT match01=^SELECT
options=case,extended
``` ```
### `targetXY` ### `targetXY`
This is the hint which will be attached to the queries matching the regex. If a The hint which is attached to the queries matching the regular expression defined by
compatible router is used in the service the query will be routed accordingly. *matchXY*. If a compatible router is used in the service the query will be routed
The target can be one of the following: accordingly. The target can be one of the following:
* a server name (adds a `HINT_ROUTE_TO_NAMED_SERVER` hint) * a server name (adds a `HINT_ROUTE_TO_NAMED_SERVER` hint)
* a list of server names, comma-separated (adds several * a list of server names, comma-separated (adds several
@ -115,26 +121,7 @@ names is simply left as is and routed straight through.
user=john user=john
``` ```
## Filter Options ## Additional remarks
The named server filter accepts the following options.
|Option |Description |
|----------|--------------------------------------------|
|ignorecase|Use case-insensitive matching (default) |
|case |Use case-sensitive matching |
|extended |Ignore white space and # comments |
To use multiple filter options, list them in a comma-separated list.
```
options=case,extended
```
**Note:** The *ignorecase* and *case* options are mutually exclusive and only
one of them should be used.
## Notes
The maximum number of accepted *match* - *target* pairs may be higher and can The maximum number of accepted *match* - *target* pairs may be higher and can
change if other features are added to the filter. A minimum of 25 is guaranteed change if other features are added to the filter. A minimum of 25 is guaranteed
@ -143,7 +130,7 @@ for now.
In the configuration, the indexed match and target settings may be in any order In the configuration, the indexed match and target settings may be in any order
and may skip numbers. During SQL-query matching, however, the regexes are tested and may skip numbers. During SQL-query matching, however, the regexes are tested
in ascending order: match01, match02, match03 and so on. As soon as a match is in ascending order: match01, match02, match03 and so on. As soon as a match is
found for a qiven query, the routing hints are written and the packet is found for a given query, the routing hints are written and the packet is
forwarded to the next filter or router. Any possibly remaining match regexes are forwarded to the next filter or router. Any possibly remaining match regexes are
ignored. This means the *match* - *target* pairs should be indexed in priority ignored. This means the *match* - *target* pairs should be indexed in priority
order, or, if priority is not a factor, in order of decreasing match order, or, if priority is not a factor, in order of decreasing match

39
Documentation/Filters/Query-Log-All-Filter.md
View File

@ -23,28 +23,6 @@ password=mypasswd
filters=MyLogFilter filters=MyLogFilter
``` ```
## Filter Options
The QLA filter accepts the following options.
Option | Description
-------| -----------
ignorecase | Use case-insensitive matching
case | Use case-sensitive matching
extended | Use extended regular expression syntax (ERE)
To use multiple filter options, list them in a comma-separated list. If no
options are given, default will be used. Multiple options can be enabled
simultaneously.
```
options=case,extended
```
**Note**: older the version of the QLA filter in 0.7 of MariaDB MaxScale used
the `options` to define the location of the log files. This functionality is not
supported anymore and the `filebase` parameter should be used instead.
## Filter Parameters ## Filter Parameters
The QLA filter has one mandatory parameter, `filebase`, and a number of optional The QLA filter has one mandatory parameter, `filebase`, and a number of optional
@ -60,25 +38,18 @@ added to the filename for each written session file. For unified log files,
filebase=/tmp/SqlQueryLog filebase=/tmp/SqlQueryLog
``` ```
The filebase may also be set as the filter option. If both option and parameter ### `match`, `exclude` and `options`
are set, the parameter setting will be used and the filter option ignored.
### `match` and `exclude` These
[regular expression settings](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters)
These optional parameters limit logging on a query level. The parameter values limit which queries are logged.
are regular expressions which are matched against the SQL query text. Only SQL
statements that match the regular expression in *match* but do not match the
*exclude* expression are logged.
``` ```
match=select.*from.*customer.*where match=select.*from.*customer.*where
exclude=^insert exclude=^insert
options=case,extended
``` ```
*match* is checked before *exclude*. If *match* is empty, all queries are
considered matching. If *exclude* is empty, no query is exluded. If both are
empty, all queries are logged.
### `user` and `source` ### `user` and `source`
These optional parameters limit logging on a session level. If `user` is These optional parameters limit logging on a session level. If `user` is

22
Documentation/Filters/Regex-Filter.md
View File

@ -28,27 +28,29 @@ password=mypasswd
filters=MyRegexfilter filters=MyRegexfilter
``` ```
## Filter Options
The Regex filter accepts the options ignorecase or case. These define if the pattern text should take the case of the string it is matching against into consideration or not.
## Filter Parameters ## Filter Parameters
The Regex filter requires two mandatory parameters to be defined. The Regex filter has two mandatory parameters: *match* and *replace*.
### `match` ### `match`, `options`
A parameter that can be used to match text in the SQL statement which should be replaced. *match* is a
[PCRE2 regular expression](../Getting-Started/Configuration-Guide.md#regular-expressions)
which defines the text in the SQL statements that is replaced.
The *options*-parameter affects how the patterns are compiled as
[usual](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters).
Regex filter does not support the `extended`-option.
``` ```
match=TYPE[ ]*= match=TYPE[ ]*=
options=case
``` ```
If the filter option ignorecase is used all regular expressions are evaluated with the option to ignore the case of the text, therefore a match option of select will match both type, TYPE and any form of the word with upper or lowercase characters.
### `replace` ### `replace`
The replace parameter defines the text that should replace the text in the SQL text which matches the match. This is the text that should replace the part of the SQL-query matching the pattern
defined in *match*.
``` ```
replace=ENGINE = replace=ENGINE =

38
Documentation/Filters/Tee-Filter.md
View File

@ -36,46 +36,14 @@ filters=DataMartFilter
The tee filter requires a mandatory parameter to define the service to replicate The tee filter requires a mandatory parameter to define the service to replicate
statements to and accepts a number of optional parameters. statements to and accepts a number of optional parameters.
### `match` ### `match`, `exclude` and `options`
An optional parameter used to limit the queries that will be replicated by the These [regular expression settings](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters)
tee filter. The parameter value is a PCRE2 regular expression that is used to limit the queries replicated by the tee filter.
match against the SQL text. Only SQL statements that match the text passed as
the value of this parameter will be sent to the service defined in the filter
section.
``` ```
match=/insert.*into.*order*/ match=/insert.*into.*order*/
```
### `exclude`
An optional parameter used to limit the queries that will be replicated by the
tee filter. The parameter value is a PCRE2 regular expression that is used to
match against the SQL text. Any SQL statements that match the text passed as the
value of this parameter will be excluded from the replication stream.
```
exclude=/select.*from.*t1/ exclude=/select.*from.*t1/
```
If both `match` and `exclude` parameters are defined, `exclude` takes
precedence.
### `options`
The options parameter controls the regular expression options. The following
options are accepted.
|Option |Description |
|----------|--------------------------------------------|
|ignorecase|Use case-insensitive matching |
|case |Use case-sensitive matching |
|extended |Use extended regular expression syntax (ERE)|
To use multiple filter options, list them in a comma-separated list.
```
options=case,extended options=case,extended
``` ```

45
Documentation/Filters/Top-N-Filter.md
View File

@ -9,7 +9,7 @@ Table of Contents
* [Filter Parameters](#filter-parameters) * [Filter Parameters](#filter-parameters)
* [filebase](#filebase) * [filebase](#filebase)
* [count](#count) * [count](#count)
* [match](#match) * [match, exclude and options](#match-exclude-and-options)
* [exclude](#exclude) * [exclude](#exclude)
* [source](#source) * [source](#source)
* [user](#user) * [user](#user)
@ -44,22 +44,6 @@ password=mypasswd
filters=MyLogFilter filters=MyLogFilter
``` ```
### Filter Options
The top filter accepts the following options.
|Option |Description |
|----------|--------------------------------------------|
|ignorecase|Use case-insensitive matching |
|case |Use case-sensitive matching |
|extended |Use extended regular expression syntax (ERE)|
To use multiple filter options, list them in a comma-separated list.
```
options=case,extended
```
### Filter Parameters ### Filter Parameters
The top filter has one mandatory parameter, `filebase`, and a number of optional The top filter has one mandatory parameter, `filebase`, and a number of optional
@ -88,36 +72,17 @@ count=30
The default value for the number of statements recorded is 10. The default value for the number of statements recorded is 10.
#### `match` #### `match`, `exclude` and `options`
An optional parameter that can be used to limit the queries that will be logged These [regular expression settings](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters)
by the top filter. The parameter value is a regular expression that is used to limit the queries logged by the top filter.
match against the SQL text. Only SQL statements that matches the text passed as
the value of this parameter will be logged.
``` ```
match=select.*from.*customer.*where match=select.*from.*customer.*where
```
All regular expressions are evaluated with the option to ignore the case of the
text, therefore a match option of select will match both select, SELECT and any
form of the word with upper or lowercase characters.
#### `exclude`
An optional parameter that can be used to limit the queries that will be logged
by the top filter. The parameter value is a regular expression that is used to
match against the SQL text. SQL statements that match the text passed as the
value of this parameter will be excluded from the log output.
```
exclude=where exclude=where
options=case,extended
``` ```
All regular expressions are evaluated with the option to ignore the case of the
text, therefore an exclude option of select will exclude statements that contain
both where, WHERE or any form of the word with upper or lowercase characters.
#### `source` #### `source`
The optional source parameter defines an address that is used to match against The optional source parameter defines an address that is used to match against

65
Documentation/Getting-Started/Configuration-Guide.md
View File

@ -279,14 +279,63 @@ _Not_ providing an explicit unit has been deprecated in MaxScale 2.4.
#### Regular Expressions #### Regular Expressions
When a regular expression (regex) parameter is accepted, the pattern string Many modules have settings which accept a regular expression. In most cases, these
should be enclosed in slashes e.g. `match=/^select/` defines the pattern settings are named either *match* or *exclude*, and are used to filter users or queries.
`^select`. The slashes allow whitespace to be read from the ends of the regex MaxScale uses the [PCRE2-library](https://www.pcre.org/current/doc/html/) for matching
string contrary to a normal string parameter and are removed before compiling regular expressions.
the pattern. For backwards compatibility, the slashes are not yet mandatory.
Omitting them is, however, deprecated and will be rejected in the next release When writing a regular expression (regex) type parameter to a MaxScale configuration file,
of MaxScale. Currently, *QLAFilter* accepts parameters in regular expression the pattern string should be enclosed in slashes e.g. `^select` -> `match=/^select/`. This
form. clarifies where the pattern begins and ends, even if it includes whitespace. Without
slashes the configuration loader trims the pattern from the ends. The slashes are removed
before compiling the pattern. For backwards compatibility, the slashes are not yet
mandatory. Omitting them is, however, deprecated and will be rejected in a future release
of MaxScale. Currently, *binlogfilter*, *ccrfilter*, *qlafilter*, *tee* and *avrorouter*
accept parameters in this type of regular expression form. Some other modules may not
handle the slashes yet correctly.
PCRE2 supports a complicated regular expression
[syntax](https://www.pcre.org/current/doc/html/pcre2syntax.html). MaxScale typically uses
regular expressions simply, only checking whether the pattern and subject match at some
point. For example, using the QLAFilter and setting `match=/SELECT/` causes the filter to
accept any query with the text "SELECT" somewhere within. To force the pattern to only
match at the beginning of the query, set `match=/^SELECT/`. To only match the end, set
`match=/SELECT$/`.
Modules which accept regular expression parameters also often accept options which affect
how the patterns are compiled. Typically, this setting is called *options* and accepts
values such as `ignorecase`, `case` and `extended`. `ignorecase` causes the regular
expression matcher to ignore letter case, and is often on by default. `extended` ignores
whitespace in the pattern. `case` turns on case-sensitive matching. These settings can
also be defined in the pattern itself, so they can be used even in modules without
pattern compilation settings. The pattern settings are `(?i)` for `ignorecase` and `(?x)`
for `extended`. See the
[PCRE2 api documentation](https://www.pcre.org/current/doc/html/pcre2api.html#SEC20)
for more information.
##### Standard regular expression settings for filters
Many filters use the settings *match*, *exclude* and *options*. Since these settings are
used in a similar way across these filters, the settings are explained here. The
documentation of the filters link here and describe any exceptions to this
generalized explanation.
These settings typically limit the queries the filter module acts on. *match* and
*exclude* define PCRE2 regular expression patterns while *options* affects how both of the
patterns are compiled. *options* works as explained above, accepting the values
`ignorecase`, `case` and `extended`, with `ignorecase` being the default.
The queries are matched as they arrive to the filter on their way to a routing module. If
*match* is defined, the filter only acts on queries matching that pattern. If *match* is
not defined, all queries are considered to match.
If *exclude* is defined, the filter only acts on queries not matching that pattern. If
*exclude* is not defined, nothing is excluded.
If both are defined, the query needs to match *match* but not match *exclude*.
Even if a filter does not act on a query, the query is not lost. The query is simply
passed on to the next module in the processing chain as if the filter was not there.
### Global Settings ### Global Settings

54
Documentation/Routers/Avrorouter.md
View File

@ -74,39 +74,11 @@ _deflate_. These are the mandatory compression algorithms required by the
Avro specification. For more information about the compression types, Avro specification. For more information about the compression types,
refer to the [Avro specification](https://avro.apache.org/docs/current/spec.html#Required+Codecs). refer to the [Avro specification](https://avro.apache.org/docs/current/spec.html#Required+Codecs).
#### `match` #### `match` and `exclude`
Only process events for tables that match this PCRE2 regular expression. See These [regular expression settings](../Getting-Started/Configuration-Guide.md#standard-regular-expression-settings-for-filters)
[Regular Expressions](../Getting-Started/Configuration-Guide.md#regular-expressions) filter events for processing depending on table names. Avrorouter does not support the
for more information about regular expressions. *options*-parameter for regular expressions.
This parameter was added in MaxScale 2.2.14.
#### `exclude`
Ignore events for tables that match this PCRE2 regular expression. This can be
combined with the `match` parameter to implement table event filtering.
This parameter was added in MaxScale 2.2.14.
**Note:** Since the 2.1 version of MaxScale, all of the router options can also
be defined as parameters.
```
[replication-router]
type=service
router=binlogrouter
router_options=server-id=4000,binlogdir=/var/lib/mysql,filestem=binlog
user=maxuser
password=maxpwd
[avro-router]
type=service
router=avrorouter
binlogdir=/var/lib/mysql
filestem=binlog
avrodir=/var/lib/maxscale
```
### Router Options ### Router Options
@ -172,6 +144,24 @@ currently, if used with Avrorouter, the option `mariadb10_master_gtid` must be
set to off in the Binlog Server configuration in order to correclty read the set to off in the Binlog Server configuration in order to correclty read the
binlog files. binlog files.
##### Example configuration
```
[replication-router]
type=service
router=binlogrouter
router_options=server-id=4000,binlogdir=/var/lib/mysql,filestem=binlog
user=maxuser
password=maxpwd
[avro-router]
type=service
router=avrorouter
binlogdir=/var/lib/mysql
filestem=binlog
avrodir=/var/lib/maxscale
```
#### Avro file options #### Avro file options
These options control how large the Avro file data blocks can get. These options control how large the Avro file data blocks can get.

5
Documentation/Routers/SchemaRouter.md
View File

@ -123,8 +123,9 @@ List of databases to ignore when checking for duplicate databases.
### `ignore_databases_regex` ### `ignore_databases_regex`
Regular expression that is matched against database names when checking for A
duplicate databases. [PCRE2 regular expression](../Getting-Started/Configuration-Guide.md#regular-expressions)
that is matched against database names when checking for duplicate databases.
### `preferred_server` ### `preferred_server`