hcq/MaxScale
hcq/MaxScale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge branch '2.3' into develop

Browse Source
This commit is contained in:
Markus Mäkelä 2019-04-01 11:31:49 +03:00
commit e659c6cc73
No known key found for this signature in database
GPG Key ID: 72D48FCE664F7B19
15 changed files with 200 additions and 73 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
Documentation/Filters/Masking.md
View File

@ -87,6 +87,15 @@ SELECT revealed_ssn FROM cheat;
```
to get access to the cleartext version of a masked field `ssn`.
From MaxScale 2.3.5 onwards, the masking filter will, if any of the
`prevent_function_usage`, `check_user_variables`, `check_unions` or
`check_subqueries` parameters is set to true, block statements that
cannot be fully parsed.
Please see the configuration parameter
[require_fully_parsed](#require_fully_parsed)
for how to change the default behaviour.
## Limitations
The masking filter can _only_ be used for masking columns of the following
@ -186,6 +195,26 @@ prevent_function_usage=false
```
The default value is `true`.
#### `require_fully_parsed`
This optional parameter specifies how the masking filter should
behave in case any of `prevent_function_usage`, `check_user_variables`,
`check_unions` or `check_subqueries` is true and it encounters a
statement that cannot be fully parsed,
If true, then statements that cannot be fully parsed (due to a
parser limitation) will be blocked.
```
require_fully_parsed=false
```
The default value is `true`.
Note that if this parameter is set to false, then `prevent_function_usage`,
`check_user_variables`, `check_unions` and `check_subqueries` are rendered
less effective, as it with a statement that can not be fully parsed may be
possible to bypass the protection that they are intended to provide.
#### `check_user_variables`
This optional parameter specifies how the masking filter should

59
Documentation/Release-Notes/MaxScale-2.3.5-Release-Notes.md Normal file
View File

@ -0,0 +1,59 @@
# MariaDB MaxScale 2.3.5 Release Notes
Release 2.3.5 is a GA release.
This document describes the changes in release 2.3.5, when compared to the
previous release in the same series.
For any problems you encounter, please consider submitting a bug
report on [our Jira](https://jira.mariadb.org/projects/MXS).
## Bug fixes
* [MXS-2410](https://jira.mariadb.org/browse/MXS-2410) Hangup delivered to wrong DCB
* [MXS-2409](https://jira.mariadb.org/browse/MXS-2409) Schemarouter crashes if PREPARE or EXECUTE is malformed
* [MXS-2403](https://jira.mariadb.org/browse/MXS-2403) Masking filter should check subqueries
* [MXS-2402](https://jira.mariadb.org/browse/MXS-2402) Masking filter should check unions
* [MXS-2398](https://jira.mariadb.org/browse/MXS-2398) Recognize MariaDB specific executable comments
* [MXS-2396](https://jira.mariadb.org/browse/MXS-2396) Masking filter should examine user variables
* [MXS-2394](https://jira.mariadb.org/browse/MXS-2394) global setting "substitute_variables" is rejected as unknown global parameter
* [MXS-2393](https://jira.mariadb.org/browse/MXS-2393) Masking filter should check parse result of query classification.
* [MXS-2392](https://jira.mariadb.org/browse/MXS-2392) Masking filter should examine statement being prepared
* [MXS-2390](https://jira.mariadb.org/browse/MXS-2390) Masking and DBFW filters should reject statement prepared from variable
* [MXS-2389](https://jira.mariadb.org/browse/MXS-2389) Fix executable comment handling
* [MXS-2379](https://jira.mariadb.org/browse/MXS-2379) JSON Interface not work with Maxscale 2.3
* [MXS-2374](https://jira.mariadb.org/browse/MXS-2374) Binlogfilter can break replication if last event is ignored
* [MXS-2373](https://jira.mariadb.org/browse/MXS-2373) Generated configs for filters does not include module
* [MXS-2370](https://jira.mariadb.org/browse/MXS-2370) Query timeout warning message does not print reason of timeout
* [MXS-2368](https://jira.mariadb.org/browse/MXS-2368) maxctrl requires password on command line and cannot change user password
* [MXS-2365](https://jira.mariadb.org/browse/MXS-2365) Wrong classification of queued queries in readwritesplit
* [MXS-2359](https://jira.mariadb.org/browse/MXS-2359) LIKE clause in SHOW TABLES is ignored by schemarouter
* [MXS-2357](https://jira.mariadb.org/browse/MXS-2357) maxctrl documentation for alter service, include use_sql_variables_in
* [MXS-2355](https://jira.mariadb.org/browse/MXS-2355) MaxScale does not let mysql client 8.0.15 to connect with password
* [MXS-2342](https://jira.mariadb.org/browse/MXS-2342) maxadmin commands hang when master pod deleted after failover occurs
* [MXS-2337](https://jira.mariadb.org/browse/MXS-2337) schemarouter in 2.3.4 doesn't show all tables from all backends
* [MXS-2326](https://jira.mariadb.org/browse/MXS-2326) Routing hints are ignored when reconnection is required
* [MXS-2325](https://jira.mariadb.org/browse/MXS-2325) Disabled events are enabled on promoted slave upon failover
* [MXS-2323](https://jira.mariadb.org/browse/MXS-2323) Connections to servers in maintenance aren't closed
* [MXS-2292](https://jira.mariadb.org/browse/MXS-2292) Allow PAM user and group mapping to work with more specific host than '%'
* [MXS-1991](https://jira.mariadb.org/browse/MXS-1991) Why MariaDBMon complain about replication_user and replication_password?
## Known Issues and Limitations
There are some limitations and known issues within this version of MaxScale.
For more information, please refer to the [Limitations](../About/Limitations.md) document.
## Packaging
RPM and Debian packages are provided for supported the Linux distributions.
Packages can be downloaded [here](https://mariadb.com/downloads/mariadb-tx/maxscale).
## Source Code
The source code of MaxScale is tagged at GitHub with a tag, which is identical
with the version of MaxScale. For instance, the tag of version X.Y.Z of MaxScale
is `maxscale-X.Y.Z`. Further, the default branch is always the latest GA version
of MaxScale.
The source code is available [here](https://github.com/mariadb-corporation/MaxScale).

21
Documentation/Routers/ReadWriteSplit.md
View File

@ -438,23 +438,10 @@ failure of a master node without any visible effects to the client.
If no replacement node becomes available before the timeout controlled by
`delayed_retry_timeout` is exceeded, the client connection is closed.
Not all transactions can be safely replayed. Only when the following criteria
are met, the transaction can be safely replayed.
* Transaction contains only data modification (`INSERT`, `UPDATE`, `DELETE`
etc.) or `SELECT ... FOR UPDATE` statements.
* The replacement server where the transaction is applied returns results
identical to the original partial transaction.
If the results from the replacement server are not identical when the transaction is
replayed, the client connection is closed. This means that any transaction with a server
specific result (e.g. `NOW()`, `@@server_id`) cannot be replayed successfully.
Performing MVCC reads (`SELECT` queries without `FOR UPDATE` or `LOCK IN SHARE MODE`)
with transaction replay is discouraged. If such statements are executed
but the results of each reply are identical, the transaction is replayed but the results
are not guaranteed to be consistent on the database level.
If the results from the replacement server are not identical when the
transaction is replayed, the client connection is closed. This means that any
transaction with a server specific result (e.g. `NOW()`, `@@server_id`) cannot
be replayed successfully but it will still be attempted.
### `transaction_replay_max_size`

1
maxscale-system-test/cnf/maxscale.cnf.template.masking_mysqltest
View File

@ -71,6 +71,7 @@ socket=default
type=filter
module=masking
rules=/###access_homedir###/masking_rules.json
require_fully_parsed=false
[server1]
type=server

6
maxscale-system-test/masking_auto_firewall.cpp
View File

@ -121,13 +121,13 @@ void run(TestConnections& test)
test_one(test, "select 1 UNION select * FROM masking_auto_firewall", Expect::FAILURE);
// This SHOULD succeed as a masked column is not used in the statment.
test_one(test, "select * FROM (select b from masking_auto_firewall)", Expect::SUCCESS);
test_one(test, "select * FROM (select b from masking_auto_firewall) tbl", Expect::SUCCESS);
// This SHOULD succeed as a masked column is not used in the statment.
test_one(test, "select * FROM (select a as b from masking_auto_firewall)", Expect::FAILURE);
test_one(test, "select * FROM (select a as b from masking_auto_firewall) tbl", Expect::FAILURE);
// This SHOULD succeed as '*' is used in the statment.
test_one(test, "select * FROM (select * from masking_auto_firewall)", Expect::FAILURE);
test_one(test, "select * FROM (select * from masking_auto_firewall) tbl", Expect::FAILURE);
}
}

40
query_classifier/qc_sqlite/qc_sqlite.cc
View File

@ -2250,24 +2250,44 @@ public:
m_type_mask = (QUERY_TYPE_WRITE | QUERY_TYPE_COMMIT);
m_operation = QUERY_OP_DROP;
if (what == MXS_DROP_SEQUENCE)
switch (what)
{
const char* zDatabase = NULL;
char database[pDatabase ? pDatabase->n + 1 : 1];
if (pDatabase)
case MXS_DROP_DATABASE:
{
#ifdef TODO_SPECIFIC_OP_FOR_DROP_DATABASE_ADDED
// TODO: As there is only QUERY_OP_DROP, you can't be fully
// TODO: certain what a returned database actually refers to
// TODO: so better not to provide a name until there is a
// TODO: specific op.
char database[pDatabase->n + 1];
strncpy(database, pDatabase->z, pDatabase->n);
database[pDatabase->n] = 0;
zDatabase = database;
update_database_names(database);
#endif
}
break;
char table[pName->n + 1];
strncpy(table, pName->z, pName->n);
table[pName->n] = 0;
case MXS_DROP_SEQUENCE:
{
const char* zDatabase = NULL;
char database[pDatabase ? pDatabase->n + 1 : 1];
update_names(zDatabase, table, NULL, NULL);
if (pDatabase)
{
strncpy(database, pDatabase->z, pDatabase->n);
database[pDatabase->n] = 0;
zDatabase = database;
}
char table[pName->n + 1];
strncpy(table, pName->z, pName->n);
table[pName->n] = 0;
update_names(zDatabase, table, NULL, NULL);
}
break;
}
}

8
query_classifier/qc_sqlite/sqlite-src-3110100/src/parse.y
View File

@ -615,7 +615,7 @@ columnid(A) ::= nm(X). {
// TODO: However, if not here then rules such as CAST need to be modified.
BINARY
/*CASCADE*/ CAST CLOSE COLUMNKW COLUMNS COMMENT CONCURRENT /*CONFLICT*/
DATA /*DATABASE*/ DEALLOCATE DEFERRED /*DESC*/ /*DETACH*/ DUMPFILE
DATA DATABASE DEALLOCATE DEFERRED /*DESC*/ /*DETACH*/ DUMPFILE
/*EACH*/ END ENGINE ENUM EXCLUSIVE /*EXPLAIN*/
FIRST FLUSH /*FOR*/ FORMAT
GLOBAL
@ -2861,6 +2861,12 @@ eq_opt ::= EQ.
default_opt ::= .
default_opt ::= DEFAULT.
////////////////////////// DROP DATABASE statement /////////////////////////////////////
//
cmd ::= DROP DATABASE ifexists id(X). {
maxscaleDrop(pParse, MXS_DROP_DATABASE, &X, NULL);
}
//////////////////////// CALL statement ////////////////////////////////////
//
cmd ::= call.

1
query_classifier/qc_sqlite/sqlite-src-3110100/src/sqliteInt.h
View File

@ -4094,6 +4094,7 @@ int sqlite3DbstatRegister(sqlite3*);
typedef enum mxs_drop
{
MXS_DROP_DATABASE,
MXS_DROP_FUNCTION,
MXS_DROP_SEQUENCE,
} mxs_drop_t;

4
query_classifier/qc_sqlite/sqlite-src-3110100/tool/mkkeywordhash.c
View File

@ -216,8 +216,10 @@ static Keyword aKeywordTable[] = {
{ "CURRENT_TIMESTAMP","TK_CTIME_KW", ALWAYS },
#ifdef MAXSCALE
{ "DATA", "TK_DATA", ALWAYS },
#endif
{ "DATABASE", "TK_DATABASE", ALWAYS },
#else
{ "DATABASE", "TK_DATABASE", ATTACH },
#endif
#ifdef MAXSCALE
{ "DATABASES", "TK_DATABASES_KW", ALWAYS },
{ "DEALLOCATE", "TK_DEALLOCATE", ALWAYS },

6
server/core/service.cc
View File

@ -111,12 +111,12 @@ Service* service_alloc(const char* name, const char* router, MXS_CONFIG_PARAMETE
dcb_enable_session_timeouts();
}
// Store router, used when service is serialized
service_add_parameter(service, CN_ROUTER, router);
// Store parameters in the service
service_add_parameters(service, params);
// Store router, used when service is serialized
service_replace_parameter(service, CN_ROUTER, router);
service->router_instance = router_api->createInstance(service, params);
if (service->router_instance == NULL)

7
server/modules/filter/binlogfilter/binlogfiltersession.cc
View File

@ -405,10 +405,11 @@ static bool should_skip_query(const BinlogConfig& config, const std::string& sql
qc_free_table_names(names, n);
}
// Also check for the default database in case the query has no tables in it
if (!rval && should_skip(config, db))
// Also check for the default database in case the query has no tables in it. The dot at the end is
// required to distinct database names from table names.
if (n == 0)
{
rval = true;
rval = should_skip(config, db + '.');
}
gwbuf_free(buf);

6
server/modules/filter/masking/maskingfilter.cc
View File

@ -139,6 +139,12 @@ extern "C" MXS_MODULE* MXS_CREATE_MODULE()
Config::check_subqueries_default,
MXS_MODULE_OPT_NONE,
},
{
Config::require_fully_parsed_name,
MXS_MODULE_PARAM_BOOL,
Config::require_fully_parsed_default,
MXS_MODULE_OPT_NONE,
},
{MXS_END_MODULE_PARAMS}
}
};

63
server/modules/filter/masking/maskingfilterconfig.cc
View File

@ -17,21 +17,23 @@
namespace
{
const char config_name_large_payload[] = "large_payload";
const char config_name_rules[] = "rules";
const char config_name_warn_type_mismatch[] = "warn_type_mismatch";
const char config_value_abort[] = "abort";
const char config_value_ignore[] = "ignore";
const char config_value_never[] = "never";
const char config_value_always[] = "always";
const char config_name_check_subqueries[] = "check_subqueries";
const char config_name_check_unions[] = "check_unions";
const char config_name_check_user_variables[] = "check_user_variables";
const char config_name_large_payload[] = "large_payload";
const char config_name_prevent_function_usage[] = "prevent_function_usage";
const char config_check_user_variables[] = "check_user_variables";
const char config_check_unions[] = "check_unions";
const char config_check_subqueries[] = "check_subqueries";
const char config_name_require_fully_parsed[] = "require_fully_parsed";
const char config_name_rules[] = "rules";
const char config_name_warn_type_mismatch[] = "warn_type_mismatch";
const char config_value_abort[] = "abort";
const char config_value_always[] = "always";
const char config_value_ignore[] = "ignore";
const char config_value_never[] = "never";
const char config_value_true[] = "true";
}
/*
@ -63,10 +65,9 @@ const char* MaskingFilterConfig::rules_name = config_name_rules;
* PARAM warn_type_mismatch
*/
// static
const char* MaskingFilterConfig::warn_type_mismatch_name = config_name_warn_type_mismatch;
const char* MaskingFilterConfig::warn_type_mismatch_default = config_value_never;
// static
const MXS_ENUM_VALUE MaskingFilterConfig::warn_type_mismatch_values[] =
{
{config_value_never, MaskingFilterConfig::WARN_NEVER },
@ -74,46 +75,36 @@ const MXS_ENUM_VALUE MaskingFilterConfig::warn_type_mismatch_values[] =
{NULL}
};
// static
const char* MaskingFilterConfig::warn_type_mismatch_default = config_value_never;
/*
* PARAM prevent_function_usage
*/
// static
const char* MaskingFilterConfig::prevent_function_usage_name = config_name_prevent_function_usage;
// static
const char* MaskingFilterConfig::prevent_function_usage_default = config_value_true;
/*
* PARAM check_user_variables
*/
// static
const char* MaskingFilterConfig::check_user_variables_name = config_check_user_variables;
// static
const char* MaskingFilterConfig::check_user_variables_name = config_name_check_user_variables;
const char* MaskingFilterConfig::check_user_variables_default = config_value_true;
/*
* PARAM check_unions
*/
// static
const char* MaskingFilterConfig::check_unions_name = config_check_unions;
// static
const char* MaskingFilterConfig::check_unions_name = config_name_check_unions;
const char* MaskingFilterConfig::check_unions_default = config_value_true;
/*
* PARAM check_subqueries
*/
// static
const char* MaskingFilterConfig::check_subqueries_name = config_check_subqueries;
// static
const char* MaskingFilterConfig::check_subqueries_name = config_name_check_subqueries;
const char* MaskingFilterConfig::check_subqueries_default = config_value_true;
/*
* PARAM require_fully_parsed
*/
const char* MaskingFilterConfig::require_fully_parsed_name = config_name_require_fully_parsed;
const char* MaskingFilterConfig::require_fully_parsed_default = config_name_require_fully_parsed;
/*
* MaskingFilterConfig
@ -164,3 +155,9 @@ bool MaskingFilterConfig::get_check_subqueries(const MXS_CONFIG_PARAMETER* pPara
{
return pParams->get_bool(check_subqueries_name);
}
// static
bool MaskingFilterConfig::get_require_fully_parsed(const MXS_CONFIG_PARAMETER* pParams)
{
return pParams->get_bool(require_fully_parsed_name);
}

16
server/modules/filter/masking/maskingfilterconfig.hh
View File

@ -54,6 +54,9 @@ public:
static const char* check_subqueries_name;
static const char* check_subqueries_default;
static const char* require_fully_parsed_name;
static const char* require_fully_parsed_default;
MaskingFilterConfig(const char* zName, const MXS_CONFIG_PARAMETER* pParams)
: m_name(zName)
, m_large_payload(get_large_payload(pParams))
@ -63,6 +66,7 @@ public:
, m_check_user_variables(get_check_user_variables(pParams))
, m_check_unions(get_check_unions(pParams))
, m_check_subqueries(get_check_subqueries(pParams))
, m_require_fully_parsed(get_require_fully_parsed(pParams))
{
}
@ -110,6 +114,11 @@ public:
return m_check_subqueries;
}
bool require_fully_parsed() const
{
return m_require_fully_parsed;
}
void set_large_payload(large_payload_t l)
{
m_large_payload = l;
@ -144,6 +153,11 @@ public:
m_check_subqueries = b;
}
void set_require_fully_parsed(bool b)
{
m_require_fully_parsed = b;
}
bool is_parsing_needed() const
{
return prevent_function_usage() || check_user_variables() || check_unions() || check_subqueries();
@ -156,6 +170,7 @@ public:
static bool get_check_user_variables(const MXS_CONFIG_PARAMETER* pParams);
static bool get_check_unions(const MXS_CONFIG_PARAMETER* pParams);
static bool get_check_subqueries(const MXS_CONFIG_PARAMETER* pParams);
static bool get_require_fully_parsed(const MXS_CONFIG_PARAMETER* pParams);
private:
std::string m_name;
@ -166,4 +181,5 @@ private:
bool m_check_user_variables;
bool m_check_unions;
bool m_check_subqueries;
bool m_require_fully_parsed;
};

6
server/modules/filter/masking/maskingfiltersession.cc
View File

@ -127,7 +127,8 @@ bool MaskingFilterSession::check_textual_query(GWBUF* pPacket)
{
bool rv = false;
if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED)
if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED
|| !m_filter.config().require_fully_parsed())
{
if (qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_PREPARE_NAMED_STMT))
{
@ -165,7 +166,8 @@ bool MaskingFilterSession::check_binary_query(GWBUF* pPacket)
{
bool rv = false;
if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED)
if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED
|| !m_filter.config().require_fully_parsed())
{
rv = check_query(pPacket);
}