hcq/MaxScale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch 'develop' into MAX-324

Browse Source
This commit is contained in:
Markus Makela
2015-02-20 04:45:48 +02:00
parent 02e11b96b0 3924f70d6b
commit ea55572ab8
28 changed files with 857 additions and 423 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
Documentation/About/About-MaxScale.md Normal file
View File

@ -0,0 +1,23 @@
# About MaxScale
The MariaDB Corporation **MaxScale** is an intelligent proxy that allows forwarding of database statements to one or more database servers using complex rules, which can be based on a semantic understanding of the database statements and the roles of the various servers within the backend cluster of databases.
MaxScale is designed to provide load balancing and high availability functionality transparently to the applications. In addition it provides a highly scalable and flexibile architecture, with plugin components to support different protocols and routing decisions.
MaxScale is implemented in C so as to operate speedily. It also makes extensive use of the asynchronous I/O capabilities of the Linux operating system. The epoll system is used to provide the event driven framework for the input and output via sockets. Similar features in Windows® could be used in future development of MaxScale.
Many of the services provided by MaxScale are implemented as external shared object modules which can be loaded at runtime. These modules support a fixed interface, communicating the entry points via a structure consisting of a set of function pointers. This structure is called the "module object". Additional modules can be created to work with MaxScale.
One group of modules provides support for protocols, both for clients that communicate with MaxScale and for backend servers. The code that routes the queries to the backend servers is also loaded as external shared objects and they are referred to as routing modules. Another group of modules work on data as it passes through MaxScale, and they are known as filters.
A Google Group exists for MaxScale that can be used to discuss ideas, issues and communicate with the MaxScale community:
Send email to [maxscale@googlegroups.com](mailto:maxscale@googlegroups.com)
or use the [forum](http://groups.google.com/forum/#!forum/maxscale) interface
Bugs can be reported in the MariaDB Corporation bugs database
[bug.mariadb.com](http://bugs.mariadb.com)
## Installing MaxScale
Information about installing MaxScale, either from a repository or by building from source code, is included in the guide [Getting Started with MaxScale](/Documentation/Getting-Started/Getting-Started-With-MaxScale.md).
The same guide also provides basic information on running MaxScale. More detailed information about configuring MaxScale is given in the [Configuration Guide](/Documentation/Getting-Started/Configuration-Guide.md).

2
Documentation/Documentation-Contents.md
View File

@ -4,6 +4,7 @@
## About MaxScale
- [About MaxScale](About/About-MaxScale.md)
- [Release Notes 1.0.4](About/MaxScale-1.0.4-Release-Notes.md)
- [Limitations](About/Limitations.md)
- [COPYRIGHT](About/COPYRIGHT.md)
@ -13,6 +14,7 @@
## Getting Started
- [Getting Started with MaxScale](Getting-Started/Getting-Started-With-MaxScale.md)
- [Building MaxScale from Source Code](Getting-Started/Building-MaxScale-from-Source-Code.md)
- [Configuration Guide](Getting-Started/Configuration-Guide.md)
## Reference

190
Documentation/Getting-Started/Building-MaxScale-from-Source-Code.md Normal file
View File

@ -0,0 +1,190 @@
# Building MaxScale from Source Code
You will need a number of tools and libraries in order to achieve this.
* cmake version 2.8.12 or later
* gcc recommended version 4.4.7 or later
* libaio
* MariaDB Develop libraries version 5.5.38 or later
* libedit 2.11 or later (used by the MaxAdmin tool)
The full list of dependencies for the most common distros is provided in the next section.
### All RHEL, CentOS and Fedora versions:
gcc gcc-c++ ncurses-devel bison glibc-devel cmake libgcc perl make libtool
openssl-devel libaio libaio-devel librabbitmq-devel
In addition, if you wish to to build an RPM package include:
rpm-build
#### RHEL 6, 7, CentOS 6, 7, Fedora:
libedit-devel
#### RHEL 7, CentOS 7:
mariadb-devel mariadb-embedded-devel
#### RHEL 5, 7, CentOS 5, 6, Fedora 19, 20
MariaDB-devel MariaDB-server
#### Fedora 19, 20
systemtap-sdt-devel
### All Ubuntu and Debian versions:
build-essential libssl-dev libaio-dev ncurses-dev bison
cmake perl libtool librabbitmq-dev
If you want to build a DEB package, you will also need:
dpkg-dev
#### Ubuntu 14.04 or later, Debian 8 (Jessie) or later
libmariadbclient-dev libmariadbd-dev
#### Earlier versions of Ubuntu or Debian
For these, you will need to obtain the MariaDB embedded library. It has to be manually extracted from the tarball. But first ascertain what version of glibc is installed. Run the command:
dpkg -l | grep libc6
which will show the version number. If the version is less than 2.14 you should obtain the library from:
[https://downloads.mariadb.org/interstitial/mariadb-5.5.41/bintar-linux-x86_64/mariadb-5.5.41-linux-x86_64.tar.gz](https://downloads.mariadb.org/interstitial/mariadb-5.5.41/bintar-linux-x86_64/mariadb-5.5.41-linux-x86_64.tar.gz).
Otherwise, from:
[https://downloads.mariadb.org/interstitial/mariadb-5.5.41/bintar-linux-glibc_214-x86_64/mariadb-5.5.41-linux-glibc_214-x86_64.tar.gz](https://downloads.mariadb.org/interstitial/mariadb-5.5.41/bintar-linux-glibc_214-x86_64/mariadb-5.5.41-linux-glibc_214-x86_64.tar.gz)
The suggested location for extracting the tarball is /usr so the operation can be done by the following commands:
cd /usr
tar -xzvf /path/to/mariadb.library.tar.gz
where /path/to/mariadb.library.tar.gz is replaced by the actual path and name of the downloaded tarball.
### OpenSUSE
(mariadb-devel package, build fails????)
The packages required are:
gcc gcc-c++ ncurses-devel bison glibc-devel cmake libgcc_s1 perl
make libtool libopenssl-devel libaio libaio-devel
libedit-devel librabbitmq-devel
MariaDB-devel MariaDB-client MariaDB-server
(if zypper ask which MariaDB client should be installed 'MariaDB-client' or 'mariadb-client'
please select 'MariaDB-client')
##Obtaining the MaxScale Source Code
Now clone the GitHub project to your machine either via the web interface, your favorite graphical interface or the git command line
$ git clone https://github.com/mariadb-corporation/MaxScale
Cloning into 'MaxScale'...
remote: Counting objects: 16228, done.
...
Change directory to the MaxScale directory, create a build directory and change directory to that build directory
$ cd MaxScale
$ mkdir build
$ cd build
The next step is to run the cmake command to build the Makefile you need to compile Maxscale. There are a number of options you may give to configure cmake and point it to the various packages it requires. In this example we will assume the MariaDB developer packages have been installed as described above and set all the options required to locate these, along with options to build the unit tests and configure the installation target directory.
If you run into any trouble while configuring CMake, you can always remove the
'CMakeCache.txt' file to clear CMake's internal cache. This resets all values to their
defaults and can be used to fix a 'stuck' configuration of CMake. This is also a good
reason why you should always build into a separate directory, because you can safely
wipe the build directory clean without the danger of deleting important files when
something goes wrong. Building 'out-of-source' also allows you to have multiple
configurations of MaxScale at the same time.
The default values that CMake uses can be found in the 'macros.cmake' file.
If you wish to change these, edit the 'macros.cmake' file or define the
variables manually at configuration time.
To display all CMake variables with their descriptions:
cmake -LH <path to source>
When you are ready to run cmake:
$ cmake -DMYSQL\_DIR=/usr/mariadb-5.5.41-linux-x86_64/include/mysql \
-DEMBEDDED\_LIB=/usr/mariadb-5.5.41-linux-x86\_64/lib/libmysqld.a \
-DMYSQLCLIENT\_LIBRARIES=/usr/mariadb-5.5.41-linux-x86_64/lib/libmysqlclient.so \
-DERRMSG=/usr/mariadb-5.5.41-linux-x86\_64/share/english/errmsg.sys \
-DINSTALL\_DIR=/home/maxscale/MaxScale -DBUILD_TESTS=Y \
-DINSTALL\_SYSTEM\_FILES=N \
-DBUILD_BINLOG=Y .. -DBUILD_RABBITMQ=N/
<pre>
-- CMake version: 2.8.12.2
-- The C compiler identification is GNU 4.4.7
-- The CXX compiler identification is GNU 4.4.7
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Library was found at: /lib64/libaio.so
-- Library was found at: /usr/lib64/libssl.so
-- Library was found at: /usr/lib64/libcrypt.so
-- Library was found at: /usr/lib64/libcrypto.so
-- Library was found at: /usr/lib64/libz.so
-- Library was found at: /usr/lib64/libm.so
-- Library was found at: /usr/lib64/libdl.so
-- Library was found at: /usr/lib64/librt.so
-- Library was found at: /usr/lib64/libpthread.so
-- Using errmsg.sys found at: /home/maxscale/usr/share/mysql/english/errmsg.sys
-- Using embedded library: /home/mpinto/usr/lib64/libmysqld.a
-- Valgrind found: /usr/bin/valgrind
-- Found dynamic MySQL client library: /home/maxscale/usr/lib64/libmysqlclient.so
-- Found static MySQL client library: /usr/lib/libmysqlclient.a
-- C Compiler supports: -Werror=format-security
-- Linking against: /home/mpinto/usr/lib64/libmysqlclient.so
-- Installing MaxScale to: /usr/local/maxscale/
-- Generating RPM packages
-- Found Doxygen: /usr/bin/doxygen (found version "1.6.1")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/maxscale/develop/build
Once the cmake command is complete simply run make to build the MaxScale binaries.
$ make
<pre>
**Scanning dependencies of target utils**
[ 1%] Building CXX object utils/CMakeFiles/utils.dir/skygw_utils.cc.o
**Linking CXX static library libutils.a**
[ 1%] Built target utils
**Scanning dependencies of target log_manager**
[ 2%] Building CXX object log_manager/CMakeFiles/log_manager.dir/log_manager.cc.o
...
</pre>
After the completion of the make process the installation can be achieved by running the make install target.
$ make install
...
This will result in an installation being created which is identical to that which would be achieved by installing the binary package. The only difference is that init.d scripts aren't installed and the RabbitMQ components are not built.
By default, MaxScale installs to '/usr/local/skysql/maxscale' and places init.d scripts and ldconfig files into their folders. Change the CMAKE_INSTALL_PREFIX variable to your desired installation directory and set INSTALL_SYSTEM_FILES=N to prevent the init.d script and ldconfig file installation.
Other useful targets for Make are `documentation`, which generates the Doxygen documentation, and `uninstall` which uninstall MaxScale binaries after an install.
## Running the MaxScale testsuite
To run "make testall" you need to have four mysqld servers running on localhost. It assumes a master-slave replication setup with one slave and three slaves.
The ports to which these servers are listening and the credentials to use for testing
can be specified in the 'macros.cmake' file.
On the master full privileges on the databases "test" and "FOO" are needed, on the saves SELECT permissions on test.* should be sufficient.
When you run the 'make testall' target after configuring the build with CMake a local version of MaxScale is installed into the build folder. After this a MaxScale instance is started and the test set is executed.
After testing has finished you can find a full testlog generated by CTest in Testing/Temporary/ directory and MaxScale's log files in the log/ directory of the build root.

108
Documentation/Getting-Started/Getting-Started-With-MaxScale.md
View File

@ -30,98 +30,7 @@ Upon successful completion of the installation process you have a version of Max
## Building MaxScale From Source Code
Alternatively you may download the MaxScale source and build your own binaries. You will need a number of tools and libraries in order to achieve this.
* cmake version 2.8.12 or later
* gcc recommended version 4.4.7 or later
* libaio
* MariaDB Develop libraries version 5.5.38 or later
* libedit 2.11 or later (used by MaxAdmin tool)
First clone the GitHub project to your machine either via the web interface, your favorite graphical interface or the git command line
$ git clone https://github.com/mariadb-corporation/MaxScale
Cloning into 'MaxScale'...
remote: Counting objects: 16228, done.
...
Change directory to the MaxScale directory, create a build directory and change directory to that build directory
$ cd MaxScale
$ mkdir build
$ cd build
The next step is to run the cmake command to build the Makefile you need to compile Maxscale. There are a number of options you may give to configure cmake and point it to the various packages it requires. These are documented in the MaxScale README file, in this example we will assume the MariaDB developer packages have been installed in a non-standard location and set all the options required to locate these, along with options to build the unit tests and configure the installation target directory.
$ cmake -DMYSQL\_DIR=/usr/mariadb-5.5.41-linux-x86_64/include/mysql \
-DEMBEDDED\_LIB=/usr/mariadb-5.5.41-linux-x86\_64/lib/libmysqld.a \
-DMYSQLCLIENT\_LIBRARIES=/usr/mariadb-5.5.41-linux-x86_64/lib/libmysqlclient.so \
-DERRMSG=/usr/mariadb-5.5.41-linux-x86\_64/share/english/errmsg.sys \
-DINSTALL\_DIR=/home/maxscale/MaxScale -DBUILD_TESTS=Y \
-DINSTALL\_SYSTEM\_FILES=N \
-DBUILD_BINLOG=Y ../
-- CMake version: 2.8.12.2
-- The C compiler identification is GNU 4.4.7
-- The CXX compiler identification is GNU 4.4.7
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Library was found at: /lib64/libaio.so
-- Library was found at: /usr/lib64/libssl.so
-- Library was found at: /usr/lib64/libcrypt.so
-- Library was found at: /usr/lib64/libcrypto.so
-- Library was found at: /usr/lib64/libz.so
-- Library was found at: /usr/lib64/libm.so
-- Library was found at: /usr/lib64/libdl.so
-- Library was found at: /usr/lib64/librt.so
-- Library was found at: /usr/lib64/libpthread.so
-- Using errmsg.sys found at: /home/maxscale/usr/share/mysql/english/errmsg.sys
-- Using embedded library: /home/mpinto/usr/lib64/libmysqld.a
-- Valgrind found: /usr/bin/valgrind
-- Found dynamic MySQL client library: /home/maxscale/usr/lib64/libmysqlclient.so
-- Found static MySQL client library: /usr/lib/libmysqlclient.a
-- C Compiler supports: -Werror=format-security
-- Linking against: /home/mpinto/usr/lib64/libmysqlclient.so
-- Installing MaxScale to: /usr/local/maxscale/
-- Generating RPM packages
-- Found Doxygen: /usr/bin/doxygen (found version "1.6.1")
-- Configuring done
-- Generating done
-- Build files have been written to: /home/maxscale/develop/build
-bash-4.1$ make depend
-bash-4.1$ make
Once the cmake command is complete simply run make to build the MaxScale binaries.
$ make
**Scanning dependencies of target utils**
[ 1%] Building CXX object utils/CMakeFiles/utils.dir/skygw_utils.cc.o
**Linking CXX static library libutils.a**
[ 1%] Built target utils
**Scanning dependencies of target log_manager**
[ 2%] Building CXX object log_manager/CMakeFiles/log_manager.dir/log_manager.cc.o
...
After the completion of the make process the installation can be achieved by running the make install target.
$ make install
...
This will result in an installation being created which is identical to that which would be achieved by installing the binary package.
Alternatively you may download the MaxScale source and build your own binaries. To do this, refer to the separate document [Building MaxScale from Source Code](/Documentation/Getting-Started/Building-MaxScale-from-Source-Code.md)
## Configuring MaxScale
@ -147,8 +56,21 @@ It is also possible to use the Read/Write Splitter with Galera. Although it is n
As well as the four major configuration choices outlined above there are also other configurations sub-options that may be mixed with those to provide a variety of different configuration and functionality. The MaxScale filter concept allows the basic configurations to be built upon in a large variety of ways. A separate filter tutorial is available that discusses the concept and gives some examples of ways to use filters.
## Running MaxScale
MaxScale consists of a core executable and a number of modules that implement
the different protocols and routing algorithms. These modules are built as
shared objects that are loaded on demand. In order for MaxScale to find these
modules it will search using a predescribed search path. The rules are:
1. Look in the current directory for the module
2. Look in $MAXSCALE_HOME/modules
3. Look in /usr/local/skysql/maxscale/modules
Configuration is read by default from the file \$MAXSCALE_HOME/etc/MaxScale.cnf, /etc/MaxScale.cnf. An example file is included in in the installation and can be found in the etc/ folder within the MaxScale installation. The default value of MAXSCALE_HOME can be overriden by using the -c flag on the command line. This should be immediately followed by the path to the MaxScale home directory. The -f flag can be used on the command line to set the name and the location of the configuration file. Without path expression the file is read from \$MAXSCALE_HOME/etc directory.
## Administration Of MaxScale
There are various administration tasks that may be done with MaxScale, a client command, maxadmin, is available that will interact with a running MaxScale and allow the status of MaxScale to be monitored and give some control of the MaxScale functionality. There is a separate reference guide for the maxadmin utility and also a short administration tutorial that covers the common administration tasks that need to be done with MaxScale.

16
Documentation/filters/Firewall-Filter.md
View File

@ -19,7 +19,7 @@ The firewall filter does not support any filter options.
### Filter Parameters
The firewall filter has one mandatory parameter that defines the location of the rule file. This is the 'rules' parameter and it expects an absolute path to the rule file.
The firewall filter has one mandatory parameter that defines the location of the rule file. This is the `rules` parameter and it expects an absolute path to the rule file.
## Rule syntax
@ -31,7 +31,7 @@ The rules are defined by using the following syntax.
Rules always define a blocking action so the basic mode for the firewall filter is to allow all queries that do not match a given set of rules. Rules are identified by their name and have a mandatory part and optional parts.
The first step of defining a rule is to start with the keyword 'rule' which identifies this line of text as a rule. The second token is identified as the name of the rule. After that the mandatory token 'deny' is required to mark the start of the actual rule definition.
The first step of defining a rule is to start with the keyword `rule` which identifies this line of text as a rule. The second token is identified as the name of the rule. After that the mandatory token `deny` is required to mark the start of the actual rule definition.
### Mandatory rule parameters
@ -43,7 +43,7 @@ This rule blocks all queries that use the wildcard character *.
#### Columns
This rule expects a list of values after the 'columns' keyword. These values are interpreted as column names and if a query targets any of these, it is blocked.
This rule expects a list of values after the `columns` keyword. These values are interpreted as column names and if a query targets any of these, it is blocked.
#### Regex
@ -63,7 +63,7 @@ Each mandatory rule accepts one or more optional parameters. These are to be def
#### At_times
This rule expects a list of time ranges that define the times when the rule in question is active. The time formats are expected to be ISO-8601 compliant and to be separated by a single dash (the - character). For example defining the active period of a rule to be 17:00 to 19:00 you would add 'at times 17:00:00-19:00:00' to the end of the rule.
This rule expects a list of time ranges that define the times when the rule in question is active. The time formats are expected to be ISO-8601 compliant and to be separated by a single dash (the - character). For example defining the active period of a rule to be 17:00 to 19:00 you would add `at times 17:00:00-19:00:00` to the end of the rule.
#### On_queries
@ -73,9 +73,11 @@ This limits the rule to be active only on certain types of queries.
To apply the defined rules to users use the following syntax.
`users NAME ... match [any|all] rules RULE ...`
`users NAME ... match [any|all|strict_all] rules RULE ...`
The first keyword is users which identifies this line as a user definition line. After this a list of user names and network addresses in the format 'user@0.0.0.0' is expected. The first part is the user name and the second part is the network address. You can use the '%' character as the wildcard to enable user name matching from any address or network matching for all users. After the list of users and networks the keyword match is expected. After this either the keyword 'any' or 'all' is expected. This defined how the rules are matched. If 'any' is used when the first rule is matched the query is considered blocked and the rest of the rules are skipped. If instead the 'all' keyword is used all rules must match for the query to be blocked.
The first keyword is users which identifies this line as a user definition line. After this a list of user names and network addresses in the format `user@0.0.0.0` is expected. The first part is the user name and the second part is the network address. You can use the `%` character as the wildcard to enable user name matching from any address or network matching for all users. After the list of users and networks the keyword match is expected.
After this either the keyword `any` `all` or `strict_all` is expected. This defined how the rules are matched. If `any` is used when the first rule is matched the query is considered blocked and the rest of the rules are skipped. If instead the `all` keyword is used all rules must match for the query to be blocked. The `strict_all` is the same as `all` but it checks the rules from left to right in the order they were listed. If one of these does not match, the rest of the rules are not checked. This could be usedful in situations where you would for example combine `limit_queries` and `regex` rules. By using `strict_all` you can have the `regex` rule first and the `limit_queries` rule second. This way the rule only matches if the `regex` rule matches enough times for the `limit_queries` rule to match.
After the matching part comes the rules keyword after which a list of rule names is expected. This allows reusing of the rules and enables varying levels of query restriction.
@ -105,4 +107,4 @@ This can be achieved by creating two rules. One that blocks the usage of the wil
We want to prevent accidental deletes into the managers table where the where clause is missing. This poses a problem, we don't want to require all the delete queries to have a where clause. We only want to prevent the data in the managers table from being deleted without a where clause.
To achieve this, we need two rules. The first rule can be seen on line 5 in the example rule file. This defines that all delete operations must have a where clause. This rule alone does us no good so we need a second one. The second rule is defined on line 6 and it blocks all queries that match the provided regular expression. When we combine these two rules we get the result we want. You can see the application of these rules on line 9 of the example rule file. The usage of the 'all' matching mode requires that all the rules must match for the query to be blocked. This in effect combines the two rules into a more complex rule.
To achieve this, we need two rules. The first rule can be seen on line 5 in the example rule file. This defines that all delete operations must have a where clause. This rule alone does us no good so we need a second one. The second rule is defined on line 6 and it blocks all queries that match the provided regular expression. When we combine these two rules we get the result we want. You can see the application of these rules on line 9 of the example rule file. The usage of the `all` and `strict_all` matching mode requires that all the rules must match for the query to be blocked. This in effect combines the two rules into a more complex rule.