diff --git a/Documentation/Filters/Masking.md b/Documentation/Filters/Masking.md index c055b696a..05a763015 100644 --- a/Documentation/Filters/Masking.md +++ b/Documentation/Filters/Masking.md @@ -87,6 +87,15 @@ SELECT revealed_ssn FROM cheat; ``` to get access to the cleartext version of a masked field `ssn`. +From MaxScale 2.3.5 onwards, the masking filter will, if any of the +`prevent_function_usage`, `check_user_variables`, `check_unions` or +`check_subqueries` parameters is set to true, block statements that +cannot be fully parsed. + +Please see the configuration parameter +[require_fully_parsed](#require_fully_parsed) +for how to change the default behaviour. + ## Limitations The masking filter can _only_ be used for masking columns of the following @@ -186,6 +195,26 @@ prevent_function_usage=false ``` The default value is `true`. +#### `require_fully_parsed` + +This optional parameter specifies how the masking filter should +behave in case any of `prevent_function_usage`, `check_user_variables`, +`check_unions` or `check_subqueries` is true and it encounters a +statement that cannot be fully parsed, + +If true, then statements that cannot be fully parsed (due to a +parser limitation) will be blocked. +``` +require_fully_parsed=false +``` + +The default value is `true`. + +Note that if this parameter is set to false, then `prevent_function_usage`, +`check_user_variables`, `check_unions` and `check_subqueries` are rendered +less effective, as it with a statement that can not be fully parsed may be +possible to bypass the protection that they are intended to provide. + #### `check_user_variables` This optional parameter specifies how the masking filter should diff --git a/Documentation/Release-Notes/MaxScale-2.3.5-Release-Notes.md b/Documentation/Release-Notes/MaxScale-2.3.5-Release-Notes.md new file mode 100644 index 000000000..e6e7eb34c --- /dev/null +++ b/Documentation/Release-Notes/MaxScale-2.3.5-Release-Notes.md @@ -0,0 +1,59 @@ +# MariaDB MaxScale 2.3.5 Release Notes + +Release 2.3.5 is a GA release. + +This document describes the changes in release 2.3.5, when compared to the +previous release in the same series. + +For any problems you encounter, please consider submitting a bug +report on [our Jira](https://jira.mariadb.org/projects/MXS). + +## Bug fixes + +* [MXS-2410](https://jira.mariadb.org/browse/MXS-2410) Hangup delivered to wrong DCB +* [MXS-2409](https://jira.mariadb.org/browse/MXS-2409) Schemarouter crashes if PREPARE or EXECUTE is malformed +* [MXS-2403](https://jira.mariadb.org/browse/MXS-2403) Masking filter should check subqueries +* [MXS-2402](https://jira.mariadb.org/browse/MXS-2402) Masking filter should check unions +* [MXS-2398](https://jira.mariadb.org/browse/MXS-2398) Recognize MariaDB specific executable comments +* [MXS-2396](https://jira.mariadb.org/browse/MXS-2396) Masking filter should examine user variables +* [MXS-2394](https://jira.mariadb.org/browse/MXS-2394) global setting "substitute_variables" is rejected as unknown global parameter +* [MXS-2393](https://jira.mariadb.org/browse/MXS-2393) Masking filter should check parse result of query classification. +* [MXS-2392](https://jira.mariadb.org/browse/MXS-2392) Masking filter should examine statement being prepared +* [MXS-2390](https://jira.mariadb.org/browse/MXS-2390) Masking and DBFW filters should reject statement prepared from variable +* [MXS-2389](https://jira.mariadb.org/browse/MXS-2389) Fix executable comment handling +* [MXS-2379](https://jira.mariadb.org/browse/MXS-2379) JSON Interface not work with Maxscale 2.3 +* [MXS-2374](https://jira.mariadb.org/browse/MXS-2374) Binlogfilter can break replication if last event is ignored +* [MXS-2373](https://jira.mariadb.org/browse/MXS-2373) Generated configs for filters does not include module +* [MXS-2370](https://jira.mariadb.org/browse/MXS-2370) Query timeout warning message does not print reason of timeout +* [MXS-2368](https://jira.mariadb.org/browse/MXS-2368) maxctrl requires password on command line and cannot change user password +* [MXS-2365](https://jira.mariadb.org/browse/MXS-2365) Wrong classification of queued queries in readwritesplit +* [MXS-2359](https://jira.mariadb.org/browse/MXS-2359) LIKE clause in SHOW TABLES is ignored by schemarouter +* [MXS-2357](https://jira.mariadb.org/browse/MXS-2357) maxctrl documentation for alter service, include use_sql_variables_in +* [MXS-2355](https://jira.mariadb.org/browse/MXS-2355) MaxScale does not let mysql client 8.0.15 to connect with password +* [MXS-2342](https://jira.mariadb.org/browse/MXS-2342) maxadmin commands hang when master pod deleted after failover occurs +* [MXS-2337](https://jira.mariadb.org/browse/MXS-2337) schemarouter in 2.3.4 doesn't show all tables from all backends +* [MXS-2326](https://jira.mariadb.org/browse/MXS-2326) Routing hints are ignored when reconnection is required +* [MXS-2325](https://jira.mariadb.org/browse/MXS-2325) Disabled events are enabled on promoted slave upon failover +* [MXS-2323](https://jira.mariadb.org/browse/MXS-2323) Connections to servers in maintenance aren't closed +* [MXS-2292](https://jira.mariadb.org/browse/MXS-2292) Allow PAM user and group mapping to work with more specific host than '%' +* [MXS-1991](https://jira.mariadb.org/browse/MXS-1991) Why MariaDBMon complain about replication_user and replication_password? + +## Known Issues and Limitations + +There are some limitations and known issues within this version of MaxScale. +For more information, please refer to the [Limitations](../About/Limitations.md) document. + +## Packaging + +RPM and Debian packages are provided for supported the Linux distributions. + +Packages can be downloaded [here](https://mariadb.com/downloads/mariadb-tx/maxscale). + +## Source Code + +The source code of MaxScale is tagged at GitHub with a tag, which is identical +with the version of MaxScale. For instance, the tag of version X.Y.Z of MaxScale +is `maxscale-X.Y.Z`. Further, the default branch is always the latest GA version +of MaxScale. + +The source code is available [here](https://github.com/mariadb-corporation/MaxScale). diff --git a/Documentation/Routers/ReadWriteSplit.md b/Documentation/Routers/ReadWriteSplit.md index e8e893510..e1d122161 100644 --- a/Documentation/Routers/ReadWriteSplit.md +++ b/Documentation/Routers/ReadWriteSplit.md @@ -438,23 +438,10 @@ failure of a master node without any visible effects to the client. If no replacement node becomes available before the timeout controlled by `delayed_retry_timeout` is exceeded, the client connection is closed. -Not all transactions can be safely replayed. Only when the following criteria -are met, the transaction can be safely replayed. - -* Transaction contains only data modification (`INSERT`, `UPDATE`, `DELETE` - etc.) or `SELECT ... FOR UPDATE` statements. - -* The replacement server where the transaction is applied returns results - identical to the original partial transaction. - -If the results from the replacement server are not identical when the transaction is -replayed, the client connection is closed. This means that any transaction with a server -specific result (e.g. `NOW()`, `@@server_id`) cannot be replayed successfully. - -Performing MVCC reads (`SELECT` queries without `FOR UPDATE` or `LOCK IN SHARE MODE`) -with transaction replay is discouraged. If such statements are executed -but the results of each reply are identical, the transaction is replayed but the results -are not guaranteed to be consistent on the database level. +If the results from the replacement server are not identical when the +transaction is replayed, the client connection is closed. This means that any +transaction with a server specific result (e.g. `NOW()`, `@@server_id`) cannot +be replayed successfully but it will still be attempted. ### `transaction_replay_max_size` diff --git a/maxscale-system-test/cnf/maxscale.cnf.template.masking_mysqltest b/maxscale-system-test/cnf/maxscale.cnf.template.masking_mysqltest index dd4893e4d..bba0510f9 100644 --- a/maxscale-system-test/cnf/maxscale.cnf.template.masking_mysqltest +++ b/maxscale-system-test/cnf/maxscale.cnf.template.masking_mysqltest @@ -71,6 +71,7 @@ socket=default type=filter module=masking rules=/###access_homedir###/masking_rules.json +require_fully_parsed=false [server1] type=server diff --git a/maxscale-system-test/masking_auto_firewall.cpp b/maxscale-system-test/masking_auto_firewall.cpp index 554d1addb..fd7e7a64d 100644 --- a/maxscale-system-test/masking_auto_firewall.cpp +++ b/maxscale-system-test/masking_auto_firewall.cpp @@ -121,13 +121,13 @@ void run(TestConnections& test) test_one(test, "select 1 UNION select * FROM masking_auto_firewall", Expect::FAILURE); // This SHOULD succeed as a masked column is not used in the statment. - test_one(test, "select * FROM (select b from masking_auto_firewall)", Expect::SUCCESS); + test_one(test, "select * FROM (select b from masking_auto_firewall) tbl", Expect::SUCCESS); // This SHOULD succeed as a masked column is not used in the statment. - test_one(test, "select * FROM (select a as b from masking_auto_firewall)", Expect::FAILURE); + test_one(test, "select * FROM (select a as b from masking_auto_firewall) tbl", Expect::FAILURE); // This SHOULD succeed as '*' is used in the statment. - test_one(test, "select * FROM (select * from masking_auto_firewall)", Expect::FAILURE); + test_one(test, "select * FROM (select * from masking_auto_firewall) tbl", Expect::FAILURE); } } diff --git a/query_classifier/qc_sqlite/qc_sqlite.cc b/query_classifier/qc_sqlite/qc_sqlite.cc index d8d45e265..549d9def4 100644 --- a/query_classifier/qc_sqlite/qc_sqlite.cc +++ b/query_classifier/qc_sqlite/qc_sqlite.cc @@ -2250,24 +2250,44 @@ public: m_type_mask = (QUERY_TYPE_WRITE | QUERY_TYPE_COMMIT); m_operation = QUERY_OP_DROP; - if (what == MXS_DROP_SEQUENCE) + switch (what) { - const char* zDatabase = NULL; - char database[pDatabase ? pDatabase->n + 1 : 1]; - - if (pDatabase) + case MXS_DROP_DATABASE: { +#ifdef TODO_SPECIFIC_OP_FOR_DROP_DATABASE_ADDED + // TODO: As there is only QUERY_OP_DROP, you can't be fully + // TODO: certain what a returned database actually refers to + // TODO: so better not to provide a name until there is a + // TODO: specific op. + char database[pDatabase->n + 1]; strncpy(database, pDatabase->z, pDatabase->n); database[pDatabase->n] = 0; - zDatabase = database; + update_database_names(database); +#endif } + break; - char table[pName->n + 1]; - strncpy(table, pName->z, pName->n); - table[pName->n] = 0; + case MXS_DROP_SEQUENCE: + { + const char* zDatabase = NULL; + char database[pDatabase ? pDatabase->n + 1 : 1]; - update_names(zDatabase, table, NULL, NULL); + if (pDatabase) + { + strncpy(database, pDatabase->z, pDatabase->n); + database[pDatabase->n] = 0; + + zDatabase = database; + } + + char table[pName->n + 1]; + strncpy(table, pName->z, pName->n); + table[pName->n] = 0; + + update_names(zDatabase, table, NULL, NULL); + } + break; } } diff --git a/query_classifier/qc_sqlite/sqlite-src-3110100/src/parse.y b/query_classifier/qc_sqlite/sqlite-src-3110100/src/parse.y index e23ef97d6..67088e537 100644 --- a/query_classifier/qc_sqlite/sqlite-src-3110100/src/parse.y +++ b/query_classifier/qc_sqlite/sqlite-src-3110100/src/parse.y @@ -615,7 +615,7 @@ columnid(A) ::= nm(X). { // TODO: However, if not here then rules such as CAST need to be modified. BINARY /*CASCADE*/ CAST CLOSE COLUMNKW COLUMNS COMMENT CONCURRENT /*CONFLICT*/ - DATA /*DATABASE*/ DEALLOCATE DEFERRED /*DESC*/ /*DETACH*/ DUMPFILE + DATA DATABASE DEALLOCATE DEFERRED /*DESC*/ /*DETACH*/ DUMPFILE /*EACH*/ END ENGINE ENUM EXCLUSIVE /*EXPLAIN*/ FIRST FLUSH /*FOR*/ FORMAT GLOBAL @@ -2861,6 +2861,12 @@ eq_opt ::= EQ. default_opt ::= . default_opt ::= DEFAULT. +////////////////////////// DROP DATABASE statement ///////////////////////////////////// +// +cmd ::= DROP DATABASE ifexists id(X). { + maxscaleDrop(pParse, MXS_DROP_DATABASE, &X, NULL); +} + //////////////////////// CALL statement //////////////////////////////////// // cmd ::= call. diff --git a/query_classifier/qc_sqlite/sqlite-src-3110100/src/sqliteInt.h b/query_classifier/qc_sqlite/sqlite-src-3110100/src/sqliteInt.h index 2a19b2f77..fdc3217a6 100644 --- a/query_classifier/qc_sqlite/sqlite-src-3110100/src/sqliteInt.h +++ b/query_classifier/qc_sqlite/sqlite-src-3110100/src/sqliteInt.h @@ -4094,6 +4094,7 @@ int sqlite3DbstatRegister(sqlite3*); typedef enum mxs_drop { + MXS_DROP_DATABASE, MXS_DROP_FUNCTION, MXS_DROP_SEQUENCE, } mxs_drop_t; diff --git a/query_classifier/qc_sqlite/sqlite-src-3110100/tool/mkkeywordhash.c b/query_classifier/qc_sqlite/sqlite-src-3110100/tool/mkkeywordhash.c index 18017915c..675c543c4 100644 --- a/query_classifier/qc_sqlite/sqlite-src-3110100/tool/mkkeywordhash.c +++ b/query_classifier/qc_sqlite/sqlite-src-3110100/tool/mkkeywordhash.c @@ -216,8 +216,10 @@ static Keyword aKeywordTable[] = { { "CURRENT_TIMESTAMP","TK_CTIME_KW", ALWAYS }, #ifdef MAXSCALE { "DATA", "TK_DATA", ALWAYS }, -#endif + { "DATABASE", "TK_DATABASE", ALWAYS }, +#else { "DATABASE", "TK_DATABASE", ATTACH }, +#endif #ifdef MAXSCALE { "DATABASES", "TK_DATABASES_KW", ALWAYS }, { "DEALLOCATE", "TK_DEALLOCATE", ALWAYS }, diff --git a/server/core/service.cc b/server/core/service.cc index 67a4b7461..e32748d0e 100644 --- a/server/core/service.cc +++ b/server/core/service.cc @@ -111,12 +111,12 @@ Service* service_alloc(const char* name, const char* router, MXS_CONFIG_PARAMETE dcb_enable_session_timeouts(); } - // Store router, used when service is serialized - service_add_parameter(service, CN_ROUTER, router); - // Store parameters in the service service_add_parameters(service, params); + // Store router, used when service is serialized + service_replace_parameter(service, CN_ROUTER, router); + service->router_instance = router_api->createInstance(service, params); if (service->router_instance == NULL) diff --git a/server/modules/filter/binlogfilter/binlogfiltersession.cc b/server/modules/filter/binlogfilter/binlogfiltersession.cc index 10c84bc9c..e825cceb4 100644 --- a/server/modules/filter/binlogfilter/binlogfiltersession.cc +++ b/server/modules/filter/binlogfilter/binlogfiltersession.cc @@ -405,10 +405,11 @@ static bool should_skip_query(const BinlogConfig& config, const std::string& sql qc_free_table_names(names, n); } - // Also check for the default database in case the query has no tables in it - if (!rval && should_skip(config, db)) + // Also check for the default database in case the query has no tables in it. The dot at the end is + // required to distinct database names from table names. + if (n == 0) { - rval = true; + rval = should_skip(config, db + '.'); } gwbuf_free(buf); diff --git a/server/modules/filter/masking/maskingfilter.cc b/server/modules/filter/masking/maskingfilter.cc index d1c56cdc2..73ba4cf1e 100644 --- a/server/modules/filter/masking/maskingfilter.cc +++ b/server/modules/filter/masking/maskingfilter.cc @@ -139,6 +139,12 @@ extern "C" MXS_MODULE* MXS_CREATE_MODULE() Config::check_subqueries_default, MXS_MODULE_OPT_NONE, }, + { + Config::require_fully_parsed_name, + MXS_MODULE_PARAM_BOOL, + Config::require_fully_parsed_default, + MXS_MODULE_OPT_NONE, + }, {MXS_END_MODULE_PARAMS} } }; diff --git a/server/modules/filter/masking/maskingfilterconfig.cc b/server/modules/filter/masking/maskingfilterconfig.cc index 16831914e..4498da53a 100644 --- a/server/modules/filter/masking/maskingfilterconfig.cc +++ b/server/modules/filter/masking/maskingfilterconfig.cc @@ -17,21 +17,23 @@ namespace { -const char config_name_large_payload[] = "large_payload"; -const char config_name_rules[] = "rules"; -const char config_name_warn_type_mismatch[] = "warn_type_mismatch"; - -const char config_value_abort[] = "abort"; -const char config_value_ignore[] = "ignore"; -const char config_value_never[] = "never"; -const char config_value_always[] = "always"; - +const char config_name_check_subqueries[] = "check_subqueries"; +const char config_name_check_unions[] = "check_unions"; +const char config_name_check_user_variables[] = "check_user_variables"; +const char config_name_large_payload[] = "large_payload"; const char config_name_prevent_function_usage[] = "prevent_function_usage"; -const char config_check_user_variables[] = "check_user_variables"; -const char config_check_unions[] = "check_unions"; -const char config_check_subqueries[] = "check_subqueries"; +const char config_name_require_fully_parsed[] = "require_fully_parsed"; +const char config_name_rules[] = "rules"; +const char config_name_warn_type_mismatch[] = "warn_type_mismatch"; + + +const char config_value_abort[] = "abort"; +const char config_value_always[] = "always"; +const char config_value_ignore[] = "ignore"; +const char config_value_never[] = "never"; const char config_value_true[] = "true"; + } /* @@ -63,10 +65,9 @@ const char* MaskingFilterConfig::rules_name = config_name_rules; * PARAM warn_type_mismatch */ -// static const char* MaskingFilterConfig::warn_type_mismatch_name = config_name_warn_type_mismatch; +const char* MaskingFilterConfig::warn_type_mismatch_default = config_value_never; -// static const MXS_ENUM_VALUE MaskingFilterConfig::warn_type_mismatch_values[] = { {config_value_never, MaskingFilterConfig::WARN_NEVER }, @@ -74,46 +75,36 @@ const MXS_ENUM_VALUE MaskingFilterConfig::warn_type_mismatch_values[] = {NULL} }; -// static -const char* MaskingFilterConfig::warn_type_mismatch_default = config_value_never; - /* * PARAM prevent_function_usage */ - -// static const char* MaskingFilterConfig::prevent_function_usage_name = config_name_prevent_function_usage; - -// static const char* MaskingFilterConfig::prevent_function_usage_default = config_value_true; /* * PARAM check_user_variables */ -// static -const char* MaskingFilterConfig::check_user_variables_name = config_check_user_variables; - -// static +const char* MaskingFilterConfig::check_user_variables_name = config_name_check_user_variables; const char* MaskingFilterConfig::check_user_variables_default = config_value_true; /* * PARAM check_unions */ -// static -const char* MaskingFilterConfig::check_unions_name = config_check_unions; - -// static +const char* MaskingFilterConfig::check_unions_name = config_name_check_unions; const char* MaskingFilterConfig::check_unions_default = config_value_true; /* * PARAM check_subqueries */ -// static -const char* MaskingFilterConfig::check_subqueries_name = config_check_subqueries; - -// static +const char* MaskingFilterConfig::check_subqueries_name = config_name_check_subqueries; const char* MaskingFilterConfig::check_subqueries_default = config_value_true; +/* + * PARAM require_fully_parsed + */ +const char* MaskingFilterConfig::require_fully_parsed_name = config_name_require_fully_parsed; +const char* MaskingFilterConfig::require_fully_parsed_default = config_name_require_fully_parsed; + /* * MaskingFilterConfig @@ -164,3 +155,9 @@ bool MaskingFilterConfig::get_check_subqueries(const MXS_CONFIG_PARAMETER* pPara { return pParams->get_bool(check_subqueries_name); } + +// static +bool MaskingFilterConfig::get_require_fully_parsed(const MXS_CONFIG_PARAMETER* pParams) +{ + return pParams->get_bool(require_fully_parsed_name); +} diff --git a/server/modules/filter/masking/maskingfilterconfig.hh b/server/modules/filter/masking/maskingfilterconfig.hh index 4388a413d..c3771aa6e 100644 --- a/server/modules/filter/masking/maskingfilterconfig.hh +++ b/server/modules/filter/masking/maskingfilterconfig.hh @@ -54,6 +54,9 @@ public: static const char* check_subqueries_name; static const char* check_subqueries_default; + static const char* require_fully_parsed_name; + static const char* require_fully_parsed_default; + MaskingFilterConfig(const char* zName, const MXS_CONFIG_PARAMETER* pParams) : m_name(zName) , m_large_payload(get_large_payload(pParams)) @@ -63,6 +66,7 @@ public: , m_check_user_variables(get_check_user_variables(pParams)) , m_check_unions(get_check_unions(pParams)) , m_check_subqueries(get_check_subqueries(pParams)) + , m_require_fully_parsed(get_require_fully_parsed(pParams)) { } @@ -110,6 +114,11 @@ public: return m_check_subqueries; } + bool require_fully_parsed() const + { + return m_require_fully_parsed; + } + void set_large_payload(large_payload_t l) { m_large_payload = l; @@ -144,6 +153,11 @@ public: m_check_subqueries = b; } + void set_require_fully_parsed(bool b) + { + m_require_fully_parsed = b; + } + bool is_parsing_needed() const { return prevent_function_usage() || check_user_variables() || check_unions() || check_subqueries(); @@ -156,6 +170,7 @@ public: static bool get_check_user_variables(const MXS_CONFIG_PARAMETER* pParams); static bool get_check_unions(const MXS_CONFIG_PARAMETER* pParams); static bool get_check_subqueries(const MXS_CONFIG_PARAMETER* pParams); + static bool get_require_fully_parsed(const MXS_CONFIG_PARAMETER* pParams); private: std::string m_name; @@ -166,4 +181,5 @@ private: bool m_check_user_variables; bool m_check_unions; bool m_check_subqueries; + bool m_require_fully_parsed; }; diff --git a/server/modules/filter/masking/maskingfiltersession.cc b/server/modules/filter/masking/maskingfiltersession.cc index e77e612db..243a6a3e0 100644 --- a/server/modules/filter/masking/maskingfiltersession.cc +++ b/server/modules/filter/masking/maskingfiltersession.cc @@ -127,7 +127,8 @@ bool MaskingFilterSession::check_textual_query(GWBUF* pPacket) { bool rv = false; - if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED) + if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED + || !m_filter.config().require_fully_parsed()) { if (qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_PREPARE_NAMED_STMT)) { @@ -165,7 +166,8 @@ bool MaskingFilterSession::check_binary_query(GWBUF* pPacket) { bool rv = false; - if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED) + if (qc_parse(pPacket, QC_COLLECT_FIELDS | QC_COLLECT_FUNCTIONS) == QC_QUERY_PARSED + || !m_filter.config().require_fully_parsed()) { rv = check_query(pPacket); }