diff --git a/Documentation/About/Limitations.md b/Documentation/About/Limitations.md index 60e253530..53e7918db 100644 --- a/Documentation/About/Limitations.md +++ b/Documentation/About/Limitations.md @@ -262,32 +262,5 @@ variables to the master. ### Schemarouter limitations (schemarouter) -The schemarouter currently has some limitations due to the nature of the -sharding implementation and the way the session variables are detected and -routed. Here is a list of the current limitations: +Please see [SchemaRouter documentation](../Routers/SchemaRouter.md). -* Cross-database queries (e.g. `SELECT column FROM database1.table UNION select -column FROM database2.table`) are not supported and are routed either to the -first explicit database in the query, the current database in use or to the -first available database, depending on which succeeds. - -* Without a default database, queries without explicit databases that do not -modify the session state will be routed to the first available server. This -means that, for example when creating a new database, queries should be done -directly on the node or the router should be equipped with the hint filter and a -routing hint should be used. Queries that modify the session state (e.g. `SET -autocommit=1`) will be routed to all servers regardless of the default database. - -* SELECT queries that modify session variables are not currently supported because -uniform results can not be guaranteed. If such a query is executed, the behavior -of the router is undefined. To work around this limitation, the query must be -executed in separate parts. - -* If a query targets a database the schemarouter hasn't mapped to a server, the -query will be routed to the first available server. This possibly returns an -error about database rights instead of a missing database. - -* The preparation of a prepared statement is routed to all servers. The -execution of a prepared statement is routed to the first available server or to -the server pointed by a routing hint attached to the query. In practice this -means that prepared statements aren't supported by the schemarouter. diff --git a/Documentation/Filters/Masking.md b/Documentation/Filters/Masking.md index a16146a4c..c055b696a 100644 --- a/Documentation/Filters/Masking.md +++ b/Documentation/Filters/Masking.md @@ -5,26 +5,7 @@ This filter was introduced in MariaDB MaxScale 2.1. Table of Contents ================= -* [Overview](#overview) -* [Security](#security) -* [Limitations](#limitations) -* [Configuration](#configuration) - * [Filter Parameters](#filter-parameters) - * [rules](#rules) - * [warn_type_mismatch](#warn_type_mismatch) - * [large_payload](#large_payload) - * [prevent_function_usage](#prevent_function_usage) -* [Rules](#rules-1) - * [replace](#replace) - * [obfuscate](#obfuscate) - * [with](#with) - * [applies_to](#applies_to) - * [exempted](#exempted) -* [Module commands](#module-commands) - * [reload](#reload) -* [Example](#example) - * [Configuration](#configuration-1) - * [masking_rules.json](#masking_rulesjson) +[TOC] ## Overview @@ -79,6 +60,33 @@ Please see the configuration parameter [check_user_variables](#check_user_variables) for how to change the default behaviour. +From MaxScale 2.3.5 onwards, the masking filter will examine unions +and if the second or subsequent SELECT refer to columns that should +be masked, the statement will be rejected. + +Please see the configuration parameter +[check_unions](#check_unions) +for how to change the default behaviour. + +From MaxScale 2.3.5 onwards, the masking filter will examine subqueries +and if a subquery refers to columns that should be masked, the statement +will be rejected. + +Please see the configuration parameter +[check_subqueries](#check_subqueries) +for how to change the default behaviour. + +Note that in order to ensure that it is not possible to get access to +masked data, the privileges of the users should be minimized. For instance, +if a user can create tables and perform inserts, he or she can execute +something like +``` +CREATE TABLE cheat (revealed_ssn TEXT); +INSERT INTO cheat SELECT ssn FROM users; +SELECT revealed_ssn FROM cheat; +``` +to get access to the cleartext version of a masked field `ssn`. + ## Limitations The masking filter can _only_ be used for masking columns of the following @@ -192,6 +200,34 @@ check_user_variables=false The default value is `true`. +#### `check_unions` + +This optional parameter specifies how the masking filter should +behave with respect to UNIONs. If true, then a statement like +``` +SELECT a FROM t1 UNION select b from t2; +``` +will be rejected if `b` is a column that should be masked. +``` +check_unions=false +``` + +The default value is `true`. + +#### `check_subqueries` + +This optional parameter specifies how the masking filter should +behave with respect to subqueries. If true, then a statement like +``` +SELECT * FROM (SELECT a as b FROM t1) as t2; +``` +will be rejected if `a` is a column that should be masked. +``` +check_subqueries=false +``` + +The default value is `true`. + ## Rules The masking rules are expressed as a JSON object. diff --git a/Documentation/Monitors/ColumnStore-Monitor.md b/Documentation/Monitors/ColumnStore-Monitor.md index c3bad5ac3..d93c23837 100644 --- a/Documentation/Monitors/ColumnStore-Monitor.md +++ b/Documentation/Monitors/ColumnStore-Monitor.md @@ -20,10 +20,9 @@ GRANT ALL ON infinidb_vtable.* TO 'maxuser'@'%'; ## Master Selection -The automatic master detection only works with ColumnStore 1.1.7 (planned -version at the time of writing). Older versions of ColumnStore do not implement -the required functionality to automatically detect which of the servers is the -primary UM. +The automatic master detection only works with ColumnStore 1.2. Older versions +of ColumnStore do not implement the required functionality to automatically +detect which of the servers is the primary UM. With older versions the `primary` parameter must be defined to tell the monitor which of the servers is the primary UM node. This guarantees that DDL statements diff --git a/Documentation/Routers/SchemaRouter.md b/Documentation/Routers/SchemaRouter.md index 3e5ac3ca1..e5606cfd8 100644 --- a/Documentation/Routers/SchemaRouter.md +++ b/Documentation/Routers/SchemaRouter.md @@ -8,8 +8,12 @@ database-based sharding, the schemarouter also enables cross-node session variable usage by routing all queries that modify the session to all nodes. -From 2.3.0 onwards, the SchemaRouter is capable of table family sharding, -in addition to being capable of sharding databases. +The main limitation of SchemaRouter is that aside from session variable writes +and some specific queries, a query can only target one server. This means that +queries which depend on results from multiple servers give incorrect results. +See [Limitations](#limitations) for more information. + +From 2.3.0 onwards, SchemaRouter is capable of limited table family sharding. Table of Contents ================= @@ -50,6 +54,19 @@ In almost all the cases these can be avoided by proper server configuration and the databases are always mapped to the same servers. More on configuration in the next chapter. +To check how databases and tables map to servers, execute the special query +`SHOW SHARDS`. The query does not support any modifiers such as `LIKE`. + +``` +show shards; + +Database |Server | +---------|-------------| +db1.t1 |MyServer1 | +db1.t2 |MyServer1 | +db2.t1 |MyServer2 | +``` + ## Configuration Here is an example configuration of the schemarouter: @@ -185,8 +202,38 @@ The minimum interval between database map refreshes in seconds. ## Limitations -For a list of schemarouter limitations, please read the -[Limitations](../About/Limitations.md) document. +1. Cross-database queries (e.g. `SELECT column FROM database1.table UNION select column +FROM database2.table`) are not properly supported. Such queries are routed either to the +first explicit database in the query, the current database in use or to the first +available database, depending on which succeeds. + +* Without a default database, queries without explicit databases that do not modify the +session state will be routed to the first available server. This includes queries such as +`CREATE DATABASE db1`. Such queries should be done directly on the node or the router +should be equipped with the hint filter and a routing hint should be used. Queries that +modify the session state (e.g. `SET autocommit=1`) will be routed to all servers +regardless of the default database. + +* SELECT queries that modify session variables are not supported because uniform results +can not be guaranteed. If such a query is executed, the behavior of the router is +undefined. To work around this limitation, the query must be executed in separate parts. + +* If a query targets a database the SchemaRouter has not mapped to a server, the +query will be routed to the first available server. This possibly returns an +error about database rights instead of a missing database. + +* The preparation of a prepared statement is routed to all servers. The +execution of a prepared statement is routed to the first available server or to +the server pointed by a routing hint attached to the query. In practice this +means that prepared statements aren't supported by the SchemaRouter. + +* `SHOW DATABASES` is handled by the router instead of routed to a server. The router only +answers correctly to the basic version of the query. Any modifiers such as `LIKE` are +ignored. + +* `SHOW TABLES` is routed to the server with the current database. If using table-level +sharding, the results will be incomplete. Use `SHOW SHARDS` to get results from the router +itself. ## Examples diff --git a/include/maxscale/dcb.hh b/include/maxscale/dcb.hh index abde8abe2..948020a61 100644 --- a/include/maxscale/dcb.hh +++ b/include/maxscale/dcb.hh @@ -208,6 +208,7 @@ struct DCB : public MXB_POLL_DATA DCB* tail = nullptr; /**< Last DCB in owning thread's list */ } thread; uint32_t n_close = 0; /** How many times dcb_close has been called. */ + uint64_t m_uid; /**< Unique identifier for this DCB */ }; /** diff --git a/include/maxscale/query_classifier.hh b/include/maxscale/query_classifier.hh index 133ab6c36..6746294a9 100644 --- a/include/maxscale/query_classifier.hh +++ b/include/maxscale/query_classifier.hh @@ -125,13 +125,23 @@ enum qc_parse_result_t }; /** - * QC_FIELD_INFO contains information about a field used in a statement. + * qc_field_context_t defines the context where a field appears. + * + * NOTE: A particular bit does NOT mean that the field appears ONLY in the context, + * but it may appear in other contexts as well. */ +typedef enum qc_field_context +{ + QC_FIELD_UNION = 1, /** The field appears on the right hand side in a UNION. */ + QC_FIELD_SUBQUERY = 2 /** The field appears in a subquery. */ +} qc_field_context_t; + struct QC_FIELD_INFO { char* database; /** Present if the field is of the form "a.b.c", NULL otherwise. */ char* table; /** Present if the field is of the form "a.b", NULL otherwise. */ char* column; /** Always present. */ + uint32_t context; /** The context in which the field appears. */ }; /** diff --git a/maxscale-system-test/masking_auto_firewall.cpp b/maxscale-system-test/masking_auto_firewall.cpp index 7f81f8577..554d1addb 100644 --- a/maxscale-system-test/masking_auto_firewall.cpp +++ b/maxscale-system-test/masking_auto_firewall.cpp @@ -110,6 +110,24 @@ void run(TestConnections& test) // This should NOT succeed as a masked column is used in a statement // defining a variable. test_one(test, "set @a = (SELECT a, b FROM masking_auto_firewall)", Expect::FAILURE); + + // This SHOULD succeed as a masked column is not used in the statment. + test_one(test, "select 1 UNION select b FROM masking_auto_firewall", Expect::SUCCESS); + + // This should NOT succeed as a masked column is used in the statment. + test_one(test, "select 1 UNION select a FROM masking_auto_firewall", Expect::FAILURE); + + // This should NOT succeed as '*' is used in the statment. + test_one(test, "select 1 UNION select * FROM masking_auto_firewall", Expect::FAILURE); + + // This SHOULD succeed as a masked column is not used in the statment. + test_one(test, "select * FROM (select b from masking_auto_firewall)", Expect::SUCCESS); + + // This SHOULD succeed as a masked column is not used in the statment. + test_one(test, "select * FROM (select a as b from masking_auto_firewall)", Expect::FAILURE); + + // This SHOULD succeed as '*' is used in the statment. + test_one(test, "select * FROM (select * from masking_auto_firewall)", Expect::FAILURE); } } diff --git a/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc b/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc index c305ef149..093c7a1a5 100644 --- a/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc +++ b/query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc @@ -1848,14 +1848,14 @@ int32_t qc_mysql_get_database_names(GWBUF* querybuf, char*** databasesp, int* si goto retblock; } - if (lex->describe || is_show_command(lex->sql_command)) + if (lex->describe || (is_show_command(lex->sql_command) && !(lex->sql_command == SQLCOM_SHOW_TABLES))) { goto retblock; } - if (lex->sql_command == SQLCOM_CHANGE_DB) + if (lex->sql_command == SQLCOM_CHANGE_DB || lex->sql_command == SQLCOM_SHOW_TABLES) { - if (lex->select_lex.db) + if (lex->select_lex.db && (strcmp(lex->select_lex.db, "skygw_virtual") != 0)) { if (i >= currsz) { diff --git a/query_classifier/qc_sqlite/qc_sqlite.cc b/query_classifier/qc_sqlite/qc_sqlite.cc index 890cf989b..88b0f5b6a 100644 --- a/query_classifier/qc_sqlite/qc_sqlite.cc +++ b/query_classifier/qc_sqlite/qc_sqlite.cc @@ -635,6 +635,7 @@ public: }; void update_field_info(const QcAliases* pAliases, + uint32_t context, const char* zDatabase, const char* zTable, const char* zColumn, @@ -680,6 +681,7 @@ public: item.table = zTable ? MXS_STRDUP(zTable) : NULL; mxb_assert(zColumn); item.column = MXS_STRDUP(zColumn); + item.context = context; // We are happy if we at least could dup the column. @@ -689,6 +691,10 @@ public: } } } + else + { + i->context |= context; + } } void update_names(const char* zDatabase, const char* zTable, const char* zAlias, QcAliases* pAliases) @@ -798,6 +804,7 @@ public: } void update_field_infos(QcAliases* pAliases, + uint32_t context, int prev_token, const Expr* pExpr, qc_token_position_t pos, @@ -812,15 +819,15 @@ public: switch (pExpr->op) { case TK_ASTERISK: // select * - update_field_infos_from_expr(pAliases, pExpr, pExclude); + update_field_infos_from_expr(pAliases, context, pExpr, pExclude); break; case TK_DOT: // select a.b ... select a.b.c - update_field_infos_from_expr(pAliases, pExpr, pExclude); + update_field_infos_from_expr(pAliases, context, pExpr, pExclude); break; case TK_ID: // select a - update_field_infos_from_expr(pAliases, pExpr, pExclude); + update_field_infos_from_expr(pAliases, context, pExpr, pExclude); break; case TK_VARIABLE: @@ -1007,12 +1014,12 @@ public: if (pLeft) { - update_field_infos(pAliases, pExpr->op, pExpr->pLeft, QC_TOKEN_LEFT, pExclude); + update_field_infos(pAliases, context, pExpr->op, pExpr->pLeft, QC_TOKEN_LEFT, pExclude); } if (pRight) { - update_field_infos(pAliases, pExpr->op, pExpr->pRight, QC_TOKEN_RIGHT, pExclude); + update_field_infos(pAliases, context, pExpr->op, pExpr->pRight, QC_TOKEN_RIGHT, pExclude); } if (pExpr->x.pList) @@ -1022,7 +1029,7 @@ public: case TK_FUNCTION: if (!ignore_exprlist) { - update_field_infos_from_exprlist(pAliases, pExpr->x.pList, pExclude); + update_field_infos_from_exprlist(pAliases, context, pExpr->x.pList, pExclude); } break; @@ -1046,7 +1053,7 @@ public: if (pExpr->flags & EP_xIsSelect) { mxb_assert(pAliases); - update_field_infos_from_subselect(*pAliases, pExpr->x.pSelect, pExclude); + update_field_infos_from_subselect(*pAliases, context, pExpr->x.pSelect, pExclude); if (zName) @@ -1059,7 +1066,7 @@ public: } else { - update_field_infos_from_exprlist(pAliases, pExpr->x.pList, pExclude); + update_field_infos_from_exprlist(pAliases, context, pExpr->x.pList, pExclude); if (zName) { @@ -1151,6 +1158,7 @@ public: } void update_field_infos_from_expr(QcAliases* pAliases, + uint32_t context, const Expr* pExpr, const ExprList* pExclude) { @@ -1162,12 +1170,13 @@ public: { if (get_field_name(pExpr, &zDatabase, &zTable, &zColumn)) { - update_field_info(pAliases, zDatabase, zTable, zColumn, pExclude); + update_field_info(pAliases, context, zDatabase, zTable, zColumn, pExclude); } } } void update_field_infos_from_exprlist(QcAliases* pAliases, + uint32_t context, const ExprList* pEList, const ExprList* pExclude) { @@ -1175,11 +1184,12 @@ public: { ExprList::ExprList_item* pItem = &pEList->a[i]; - update_field_infos(pAliases, 0, pItem->pExpr, QC_TOKEN_MIDDLE, pExclude); + update_field_infos(pAliases, context, 0, pItem->pExpr, QC_TOKEN_MIDDLE, pExclude); } } void update_field_infos_from_idlist(QcAliases* pAliases, + uint32_t context, const IdList* pIds, const ExprList* pExclude) { @@ -1189,7 +1199,7 @@ public: { IdList::IdList_item* pItem = &pIds->a[i]; - update_field_info(pAliases, NULL, NULL, pItem->zName, pExclude); + update_field_info(pAliases, context, NULL, NULL, pItem->zName, pExclude); } } } @@ -1201,6 +1211,7 @@ public: }; void update_field_infos_from_select(QcAliases& aliases, + uint32_t context, const Select* pSelect, const ExprList* pExclude, compound_approach_t compound_approach = ANALYZE_COMPOUND_SELECTS) @@ -1218,7 +1229,7 @@ public: if (pSrc->a[i].pSelect) { - update_field_infos_from_select(aliases, pSrc->a[i].pSelect, pExclude); + update_field_infos_from_select(aliases, context | QC_FIELD_SUBQUERY, pSrc->a[i].pSelect, pExclude); } #ifdef QC_COLLECT_NAMES_FROM_USING @@ -1236,13 +1247,14 @@ public: if (pSelect->pEList) { - update_field_infos_from_exprlist(&aliases, pSelect->pEList, NULL); + update_field_infos_from_exprlist(&aliases, context, pSelect->pEList, NULL); } if (pSelect->pWhere) { m_has_clause = true; update_field_infos(&aliases, + context, 0, pSelect->pWhere, QC_TOKEN_MIDDLE, @@ -1252,6 +1264,7 @@ public: if (pSelect->pGroupBy) { update_field_infos_from_exprlist(&aliases, + context, pSelect->pGroupBy, pSelect->pEList); } @@ -1268,7 +1281,7 @@ public: if (pSelect->pWith) { - update_field_infos_from_with(&aliases, pSelect->pWith); + update_field_infos_from_with(&aliases, context, pSelect->pWith); } if (compound_approach == ANALYZE_COMPOUND_SELECTS) @@ -1279,10 +1292,22 @@ public: while (pPrior) { - update_field_infos_from_subselect(aliases, - pPrior, - pExclude, - IGNORE_COMPOUND_SELECTS); + uint32_t ctx = context; + + if (!pPrior->pPrior) + { + // The fields in the first select in a UNION are not considered to + // be in a union. Those names will be visible in the resultset. + ctx &= ~QC_FIELD_UNION; + } + + QcAliases aliases2(aliases); + + update_field_infos_from_select(aliases2, + ctx, + pPrior, + pExclude, + IGNORE_COMPOUND_SELECTS); pPrior = pPrior->pPrior; } } @@ -1290,16 +1315,19 @@ public: } void update_field_infos_from_subselect(const QcAliases& existing_aliases, + uint32_t context, const Select* pSelect, const ExprList* pExclude, compound_approach_t compound_approach = ANALYZE_COMPOUND_SELECTS) { QcAliases aliases(existing_aliases); - update_field_infos_from_select(aliases, pSelect, pExclude, compound_approach); + context |= QC_FIELD_SUBQUERY; + + update_field_infos_from_select(aliases, context, pSelect, pExclude, compound_approach); } - void update_field_infos_from_with(QcAliases* pAliases, const With* pWith) + void update_field_infos_from_with(QcAliases* pAliases, uint32_t context, const With* pWith) { for (int i = 0; i < pWith->nCte; ++i) { @@ -1308,7 +1336,7 @@ public: if (pCte->pSelect) { mxb_assert(pAliases); - update_field_infos_from_subselect(*pAliases, pCte->pSelect, NULL); + update_field_infos_from_subselect(*pAliases, context, pCte->pSelect, NULL); } } } @@ -1668,7 +1696,8 @@ public: if (pSelect) { - update_field_infos_from_select(aliases, pSelect, NULL); + uint32_t context = 0; + update_field_infos_from_select(aliases, context, pSelect, NULL); } exposed_sqlite3ExprListDelete(pParse->db, pCNames); @@ -1738,7 +1767,8 @@ public: if (pWhere) { - update_field_infos(&aliases, 0, pWhere, QC_TOKEN_MIDDLE, 0); + uint32_t context = 0; + update_field_infos(&aliases, context, 0, pWhere, QC_TOKEN_MIDDLE, 0); } } @@ -1793,7 +1823,8 @@ public: if (pSelect) { QcAliases aliases; - update_field_infos_from_select(aliases, pSelect, NULL); + uint32_t context = 0; + update_field_infos_from_select(aliases, context, pSelect, NULL); } else if (pOldTable) { @@ -1821,12 +1852,13 @@ public: mxb_assert(pTabList->nSrc >= 1); QcAliases aliases; + uint32_t context = 0; update_names_from_srclist(&aliases, pTabList); if (pColumns) { - update_field_infos_from_idlist(&aliases, pColumns, NULL); + update_field_infos_from_idlist(&aliases, context, pColumns, NULL); int i = update_function_info(&aliases, "=", NULL); @@ -1851,12 +1883,12 @@ public: if (pSelect) { - update_field_infos_from_select(aliases, pSelect, NULL); + update_field_infos_from_select(aliases, context, pSelect, NULL); } if (pSet) { - update_field_infos_from_exprlist(&aliases, pSet, NULL); + update_field_infos_from_exprlist(&aliases, context, pSet, NULL); } } @@ -1958,6 +1990,7 @@ public: if (m_operation != QUERY_OP_EXPLAIN) { QcAliases aliases; + uint32_t context = 0; m_type_mask = QUERY_TYPE_WRITE; m_operation = QUERY_OP_UPDATE; @@ -1971,6 +2004,7 @@ public: ExprList::ExprList_item* pItem = &pChanges->a[i]; update_field_infos(&aliases, + context, 0, pItem->pExpr, QC_TOKEN_MIDDLE, @@ -1980,7 +2014,7 @@ public: if (pWhere) { - update_field_infos(&aliases, 0, pWhere, QC_TOKEN_MIDDLE, pChanges); + update_field_infos(&aliases, context, 0, pWhere, QC_TOKEN_MIDDLE, pChanges); } } @@ -2039,7 +2073,8 @@ public: } QcAliases aliases; - update_field_infos_from_select(aliases, pSelect, NULL); + uint32_t context = (pSelect->op == TK_UNION && pSelect->pPrior) ? QC_FIELD_UNION : 0; + update_field_infos_from_select(aliases, context, pSelect, NULL); } void maxscaleAlterTable(Parse* pParse, /* Parser context. */ @@ -2085,7 +2120,8 @@ public: if (pExprList) { QcAliases aliases; - update_field_infos_from_exprlist(&aliases, pExprList, NULL); + uint32_t context = 0; + update_field_infos_from_exprlist(&aliases, context, pExprList, NULL); } exposed_sqlite3SrcListDelete(pParse->db, pName); @@ -2906,7 +2942,8 @@ public: if (pValue->op == TK_SELECT) { QcAliases aliases; - update_field_infos_from_select(aliases, pValue->x.pSelect, NULL); + uint32_t context = 0; + update_field_infos_from_select(aliases, context, pValue->x.pSelect, NULL); } } break; @@ -2993,6 +3030,14 @@ public: case MXS_SHOW_TABLES: m_type_mask = QUERY_TYPE_SHOW_TABLES; + if (pShow->pDatabase->z) + { + char db[pShow->pDatabase->n + 1]; + strncpy(db, pShow->pDatabase->z, pShow->pDatabase->n); + db[pShow->pDatabase->n] = 0; + + update_database_names(db); + } break; case MXS_SHOW_VARIABLES: diff --git a/query_classifier/test/compare.cc b/query_classifier/test/compare.cc index 2e7e180f1..627b03478 100644 --- a/query_classifier/test/compare.cc +++ b/query_classifier/test/compare.cc @@ -876,6 +876,7 @@ public: : m_database(info.database ? info.database : "") , m_table(info.table ? info.table : "") , m_column(info.column ? info.column : "") + , m_context(info.context) { } @@ -939,12 +940,33 @@ public: } out << m_column; + + if (m_context != 0) + { + out << "("; + bool first = true; + + if (m_context & QC_FIELD_UNION) + { + out << (first ? "" : ", ") << "UNION"; + first = false; + } + + if (m_context & QC_FIELD_SUBQUERY) + { + out << (first ? "" : ", ") << "SUBQUERY"; + first = false; + } + + out << ")"; + } } private: std::string m_database; std::string m_table; std::string m_column; + uint32_t m_context; }; ostream& operator<<(ostream& out, const QcFieldInfo& x) @@ -1010,7 +1032,17 @@ bool compare_get_field_info(QUERY_CLASSIFIER* pClassifier1, if (f1 == f2) { ss << "Ok : "; - ss << f1; + + // TODO: Currently qc_sqlite provides context information, while qc_mysqlembedded + // TODO: does not. To ensure that the output always contains the maximum amount + // TODO: of information, we simply generate both output and print the longest. + + stringstream ss1; + ss1 << f1; + stringstream ss2; + ss2 << f2; + + ss << (ss1.str().length() > ss2.str().length() ? ss1.str() : ss2.str()); success = true; } else diff --git a/server/core/dcb.cc b/server/core/dcb.cc index 75759d985..58e3ddd51 100644 --- a/server/core/dcb.cc +++ b/server/core/dcb.cc @@ -38,6 +38,8 @@ #include #include +#include + #include #include #include @@ -78,6 +80,7 @@ static struct { DCB** all_dcbs; /** #workers sized array of pointers to DCBs where dcbs are listed. */ bool check_timeouts; /** Should session timeouts be checked. */ + std::atomic uid_generator {0}; } this_unit; static thread_local struct @@ -161,6 +164,7 @@ DCB::DCB(Role role, MXS_SESSION* session) , low_water(config_writeq_low_water()) , service(session->service) , last_read(mxs_clock()) + , m_uid(this_unit.uid_generator.fetch_add(1, std::memory_order_relaxed)) { // TODO: Remove DCB::Role::INTERNAL to always have a valid listener if (session->listener) @@ -2805,6 +2809,7 @@ public: : m_dcb(dcb) , m_buffer(buf) , m_ev(ev) + , m_uid(dcb->m_uid) { } @@ -2813,7 +2818,7 @@ public: mxb_assert(&worker == RoutingWorker::get_current()); RoutingWorker& rworker = static_cast(worker); - if (dcb_is_still_valid(m_dcb, rworker.id())) + if (dcb_is_still_valid(m_dcb, rworker.id()) && m_dcb->m_uid == m_uid) { m_dcb->fakeq = m_buffer; dcb_handler(m_dcb, m_ev); @@ -2828,6 +2833,7 @@ private: DCB* m_dcb; GWBUF* m_buffer; uint32_t m_ev; + uint64_t m_uid; /**< DCB UID guarantees we deliver the event to the correct DCB */ }; static void poll_add_event_to_dcb(DCB* dcb, GWBUF* buf, uint32_t ev) diff --git a/server/modules/filter/binlogfilter/binlogfiltersession.cc b/server/modules/filter/binlogfilter/binlogfiltersession.cc index 75d369306..10c84bc9c 100644 --- a/server/modules/filter/binlogfilter/binlogfiltersession.cc +++ b/server/modules/filter/binlogfilter/binlogfiltersession.cc @@ -405,6 +405,12 @@ static bool should_skip_query(const BinlogConfig& config, const std::string& sql qc_free_table_names(names, n); } + // Also check for the default database in case the query has no tables in it + if (!rval && should_skip(config, db)) + { + rval = true; + } + gwbuf_free(buf); return rval; } diff --git a/server/modules/filter/masking/maskingfilter.cc b/server/modules/filter/masking/maskingfilter.cc index 6b005e6b8..d1c56cdc2 100644 --- a/server/modules/filter/masking/maskingfilter.cc +++ b/server/modules/filter/masking/maskingfilter.cc @@ -127,6 +127,18 @@ extern "C" MXS_MODULE* MXS_CREATE_MODULE() Config::check_user_variables_default, MXS_MODULE_OPT_NONE, }, + { + Config::check_unions_name, + MXS_MODULE_PARAM_BOOL, + Config::check_unions_default, + MXS_MODULE_OPT_NONE, + }, + { + Config::check_subqueries_name, + MXS_MODULE_PARAM_BOOL, + Config::check_subqueries_default, + MXS_MODULE_OPT_NONE, + }, {MXS_END_MODULE_PARAMS} } }; diff --git a/server/modules/filter/masking/maskingfilterconfig.cc b/server/modules/filter/masking/maskingfilterconfig.cc index 517b580fe..16831914e 100644 --- a/server/modules/filter/masking/maskingfilterconfig.cc +++ b/server/modules/filter/masking/maskingfilterconfig.cc @@ -28,6 +28,8 @@ const char config_value_always[] = "always"; const char config_name_prevent_function_usage[] = "prevent_function_usage"; const char config_check_user_variables[] = "check_user_variables"; +const char config_check_unions[] = "check_unions"; +const char config_check_subqueries[] = "check_subqueries"; const char config_value_true[] = "true"; } @@ -88,11 +90,31 @@ const char* MaskingFilterConfig::prevent_function_usage_default = config_value_t /* * PARAM check_user_variables */ +// static const char* MaskingFilterConfig::check_user_variables_name = config_check_user_variables; // static const char* MaskingFilterConfig::check_user_variables_default = config_value_true; +/* + * PARAM check_unions + */ +// static +const char* MaskingFilterConfig::check_unions_name = config_check_unions; + +// static +const char* MaskingFilterConfig::check_unions_default = config_value_true; + +/* + * PARAM check_subqueries + */ +// static +const char* MaskingFilterConfig::check_subqueries_name = config_check_subqueries; + +// static +const char* MaskingFilterConfig::check_subqueries_default = config_value_true; + + /* * MaskingFilterConfig */ @@ -130,3 +152,15 @@ bool MaskingFilterConfig::get_check_user_variables(const MXS_CONFIG_PARAMETER* p { return pParams->get_bool(check_user_variables_name); } + +// static +bool MaskingFilterConfig::get_check_unions(const MXS_CONFIG_PARAMETER* pParams) +{ + return pParams->get_bool(check_unions_name); +} + +// static +bool MaskingFilterConfig::get_check_subqueries(const MXS_CONFIG_PARAMETER* pParams) +{ + return pParams->get_bool(check_subqueries_name); +} diff --git a/server/modules/filter/masking/maskingfilterconfig.hh b/server/modules/filter/masking/maskingfilterconfig.hh index faad0d470..4388a413d 100644 --- a/server/modules/filter/masking/maskingfilterconfig.hh +++ b/server/modules/filter/masking/maskingfilterconfig.hh @@ -48,6 +48,12 @@ public: static const char* check_user_variables_name; static const char* check_user_variables_default; + static const char* check_unions_name; + static const char* check_unions_default; + + static const char* check_subqueries_name; + static const char* check_subqueries_default; + MaskingFilterConfig(const char* zName, const MXS_CONFIG_PARAMETER* pParams) : m_name(zName) , m_large_payload(get_large_payload(pParams)) @@ -55,8 +61,11 @@ public: , m_warn_type_mismatch(get_warn_type_mismatch(pParams)) , m_prevent_function_usage(get_prevent_function_usage(pParams)) , m_check_user_variables(get_check_user_variables(pParams)) + , m_check_unions(get_check_unions(pParams)) + , m_check_subqueries(get_check_subqueries(pParams)) { } + ~MaskingFilterConfig() { } @@ -91,6 +100,16 @@ public: return m_check_user_variables; } + bool check_unions() const + { + return m_check_unions; + } + + bool check_subqueries() const + { + return m_check_subqueries; + } + void set_large_payload(large_payload_t l) { m_large_payload = l; @@ -115,9 +134,19 @@ public: m_check_user_variables = b; } + void set_check_unions(bool b) + { + m_check_unions = b; + } + + void set_check_subqueries(bool b) + { + m_check_subqueries = b; + } + bool is_parsing_needed() const { - return prevent_function_usage() || check_user_variables(); + return prevent_function_usage() || check_user_variables() || check_unions() || check_subqueries(); } static large_payload_t get_large_payload(const MXS_CONFIG_PARAMETER* pParams); @@ -125,6 +154,8 @@ public: static warn_type_mismatch_t get_warn_type_mismatch(const MXS_CONFIG_PARAMETER* pParams); static bool get_prevent_function_usage(const MXS_CONFIG_PARAMETER* pParams); static bool get_check_user_variables(const MXS_CONFIG_PARAMETER* pParams); + static bool get_check_unions(const MXS_CONFIG_PARAMETER* pParams); + static bool get_check_subqueries(const MXS_CONFIG_PARAMETER* pParams); private: std::string m_name; @@ -133,4 +164,6 @@ private: warn_type_mismatch_t m_warn_type_mismatch; bool m_prevent_function_usage; bool m_check_user_variables; + bool m_check_unions; + bool m_check_subqueries; }; diff --git a/server/modules/filter/masking/maskingfiltersession.cc b/server/modules/filter/masking/maskingfiltersession.cc index ec30f2f2b..e77e612db 100644 --- a/server/modules/filter/masking/maskingfiltersession.cc +++ b/server/modules/filter/masking/maskingfiltersession.cc @@ -82,25 +82,45 @@ bool MaskingFilterSession::check_query(GWBUF* pPacket) zHost = ""; } - bool rv = true; + bool acceptable = true; - if (rv && m_filter.config().prevent_function_usage()) + const MaskingFilter::Config& config = m_filter.config(); + + if (qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_USERVAR_WRITE)) { - if (is_function_used(pPacket, zUser, zHost)) + if (config.check_user_variables()) { - rv = false; + if (is_variable_defined(pPacket, zUser, zHost)) + { + acceptable = false; + } + } + } + else + { + qc_query_op_t op = qc_get_operation(pPacket); + + if (op == QUERY_OP_SELECT) + { + if (config.check_unions() || config.check_subqueries()) + { + if (is_union_or_subquery_used(pPacket, zUser, zHost)) + { + acceptable = false; + } + } + } + + if (acceptable && config.prevent_function_usage()) + { + if (is_function_used(pPacket, zUser, zHost)) + { + acceptable = false; + } } } - if (rv && m_filter.config().check_user_variables()) - { - if (is_variable_defined(pPacket, zUser, zHost)) - { - rv = false; - } - } - - return rv; + return acceptable; } bool MaskingFilterSession::check_textual_query(GWBUF* pPacket) @@ -551,19 +571,26 @@ bool MaskingFilterSession::is_function_used(GWBUF* pPacket, const char* zUser, c bool MaskingFilterSession::is_variable_defined(GWBUF* pPacket, const char* zUser, const char* zHost) { - if (!qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_USERVAR_WRITE)) - { - return false; - } + mxb_assert(qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_USERVAR_WRITE)); bool is_defined = false; SMaskingRules sRules = m_filter.rules(); auto pred = [&sRules, zUser, zHost](const QC_FIELD_INFO& field_info) { - const MaskingRules::Rule* pRule = sRules->get_rule_for(field_info, zUser, zHost); + bool rv = false; - return pRule ? true : false; + if (strcmp(field_info.column, "*") == 0) + { + // If "*" is used, then we must block if there is any rule for the current user. + rv = sRules->has_rule_for(zUser, zHost); + } + else + { + rv = sRules->get_rule_for(field_info, zUser, zHost) ? true : false; + } + + return rv; }; const QC_FIELD_INFO* pInfos; @@ -578,14 +605,121 @@ bool MaskingFilterSession::is_variable_defined(GWBUF* pPacket, const char* zUser if (i != end) { + const char* zColumn = i->column; + std::stringstream ss; - ss << "The field " << i->column << " that should be masked for '" << zUser << "'@'" << zHost - << "' is used when defining a variable, access is denied."; + + if (strcmp(zColumn, "*") == 0) + { + ss << "'*' is used in the definition of a variable and there are masking rules " + << "for '" << zUser << "'@'" << zHost << "', access is denied."; + } + else + { + ss << "The field " << i->column << " that should be masked for '" << zUser << "'@'" << zHost + << "' is used when defining a variable, access is denied."; + } set_response(create_error_response(ss.str().c_str())); - is_defined = true; } return is_defined; } + +bool MaskingFilterSession::is_union_or_subquery_used(GWBUF* pPacket, const char* zUser, const char* zHost) +{ + mxb_assert(qc_get_operation(pPacket) == QUERY_OP_SELECT); + + const MaskingFilter::Config& config = m_filter.config(); + + mxb_assert(config.check_unions() || config.check_subqueries()); + + bool is_used = false; + + SMaskingRules sRules = m_filter.rules(); + + uint32_t mask = 0; + + if (config.check_unions()) + { + mask |= QC_FIELD_UNION; + } + + if (config.check_subqueries()) + { + mask |= QC_FIELD_SUBQUERY; + } + + auto pred = [&sRules, mask, zUser, zHost](const QC_FIELD_INFO& field_info) { + bool rv = false; + + if (field_info.context & mask) + { + if (strcmp(field_info.column, "*") == 0) + { + // If "*" is used, then we must block if there is any rule for the current user. + rv = sRules->has_rule_for(zUser, zHost); + } + else + { + rv = sRules->get_rule_for(field_info, zUser, zHost) ? true : false; + } + } + + return rv; + }; + + const QC_FIELD_INFO* pInfos; + size_t nInfos; + + qc_get_field_info(pPacket, &pInfos, &nInfos); + + const QC_FIELD_INFO* begin = pInfos; + const QC_FIELD_INFO* end = begin + nInfos; + + auto i = std::find_if(begin, end, pred); + + if (i != end) + { + const char* zColumn = i->column; + + std::stringstream ss; + + if (config.check_unions() && (i->context & QC_FIELD_UNION)) + { + if (strcmp(zColumn, "*") == 0) + { + ss << "'*' is used in the second or subsequent SELECT of a UNION and there are " + << "masking rules for '" << zUser << "'@'" << zHost << "', access is denied."; + } + else + { + ss << "The field " << zColumn << " that should be masked for '" << zUser << "'@'" << zHost + << "' is used in the second or subsequent SELECT of a UNION, access is denied."; + } + } + else if (config.check_subqueries() && (i->context & QC_FIELD_SUBQUERY)) + { + if (strcmp(zColumn, "*") == 0) + { + ss << "'*' is used in a subquery and there are masking rules for '" + << zUser << "'@'" << zHost << "', access is denied."; + } + else + { + ss << "The field " << zColumn << " that should be masked for '" + << zUser << "'@'" << zHost << "' is used in a subquery, access is denied."; + } + } + else + { + mxb_assert(!true); + } + + set_response(create_error_response(ss.str().c_str())); + is_used = true; + } + + return is_used; +} diff --git a/server/modules/filter/masking/maskingfiltersession.hh b/server/modules/filter/masking/maskingfiltersession.hh index 39da05596..90e788e6e 100644 --- a/server/modules/filter/masking/maskingfiltersession.hh +++ b/server/modules/filter/masking/maskingfiltersession.hh @@ -67,6 +67,7 @@ private: bool is_function_used(GWBUF* pPacket, const char* zUser, const char* zHost); bool is_variable_defined(GWBUF* pPacket, const char* zUser, const char* zHost); + bool is_union_or_subquery_used(GWBUF* pPacket, const char* zUser, const char* zHost); private: typedef std::shared_ptr SMaskingRules; diff --git a/server/modules/filter/masking/maskingrules.cc b/server/modules/filter/masking/maskingrules.cc index c36dc2f85..5489770cd 100644 --- a/server/modules/filter/masking/maskingrules.cc +++ b/server/modules/filter/masking/maskingrules.cc @@ -1467,3 +1467,12 @@ const MaskingRules::Rule* MaskingRules::get_rule_for(const QC_FIELD_INFO& field_ return pRule; } + +bool MaskingRules::has_rule_for(const char* zUser, const char* zHost) const +{ + auto i = std::find_if(m_rules.begin(), m_rules.end(), [zUser, zHost](SRule sRule) { + return sRule->matches_account(zUser, zHost); + }); + + return i != m_rules.end(); +} diff --git a/server/modules/filter/masking/maskingrules.hh b/server/modules/filter/masking/maskingrules.hh index 6d676c4e5..5bb4ac424 100644 --- a/server/modules/filter/masking/maskingrules.hh +++ b/server/modules/filter/masking/maskingrules.hh @@ -143,13 +143,21 @@ public: */ virtual void rewrite(LEncString& s) const = 0; + /** + * Does this rule apply to a specific account. + * + * @param zUser The current user. + * @param zHost The current host. + * + * @return True, if the rule applies. + */ + bool matches_account(const char* zUser, + const char* zHost) const; + private: Rule(const Rule&); Rule& operator=(const Rule&); - bool matches_account(const char* zUser, - const char* zHost) const; - private: std::string m_column; std::string m_table; @@ -396,6 +404,16 @@ public: typedef std::shared_ptr SRule; + /** + * Is there any rule for the specified user. + * + * @param zUser The current user. + * @param zHost The current host. + * + * @return True, if there is a rule for that user/host combination. + */ + bool has_rule_for(const char* zUser, const char* zHost) const; + private: MaskingRules(json_t* pRoot, const std::vector& rules); diff --git a/server/modules/monitor/csmon/csmon.cc b/server/modules/monitor/csmon/csmon.cc index 56e4b285d..5cc632746 100644 --- a/server/modules/monitor/csmon/csmon.cc +++ b/server/modules/monitor/csmon/csmon.cc @@ -104,10 +104,9 @@ void CsMonitor::update_server_status(MonitorServer* srv) { status |= SERVER_RUNNING; - if (get_cs_version(srv) >= 10107) + if (get_cs_version(srv) >= 10200) { - // 1.1.7 should support the mcsSystemPrimary function - // TODO: Update when the actual release is out + // 1.2 supports the mcsSystemPrimary function status |= do_query(srv, role_query) == "1" ? SERVER_MASTER : SERVER_SLAVE; } else diff --git a/server/modules/routing/schemarouter/schemaroutersession.cc b/server/modules/routing/schemarouter/schemaroutersession.cc index 188cb84ab..9b4809ac6 100644 --- a/server/modules/routing/schemarouter/schemaroutersession.cc +++ b/server/modules/routing/schemarouter/schemaroutersession.cc @@ -342,14 +342,6 @@ int32_t SchemaRouterSession::routeQuery(GWBUF* pPacket) gwbuf_free(pPacket); return 1; } - else if (qc_query_is_type(type, QUERY_TYPE_SHOW_TABLES)) - { - if (send_tables(pPacket)) - { - gwbuf_free(pPacket); - return 1; - } - } else if (detect_show_shards(pPacket)) { if (send_shards()) @@ -1576,69 +1568,6 @@ void SchemaRouterSession::send_databases() set->write(m_client); } -bool SchemaRouterSession::send_tables(GWBUF* pPacket) -{ - char* query = modutil_get_SQL(pPacket); - char* tmp; - std::string database; - - if ((tmp = strcasestr(query, "from"))) - { - const char* delim = "` \n\t;"; - char* saved, * tok = strtok_r(tmp, delim, &saved); - tok = strtok_r(NULL, delim, &saved); - database = tok; - } - - if (database.empty()) - { - // Was not a "show tables from x". If a current database is selected, use that as target. - if (!m_current_db.empty()) - { - database = m_current_db; - } - else - { - // No current db, route the query to a server, likely getting "No database selected" - MXS_FREE(query); - return false; - } - } - - ServerMap tablelist; - std::list table_names; - m_shard.get_content(tablelist); - - for (ServerMap::iterator it = tablelist.begin(); it != tablelist.end(); it++) - { - std::size_t pos = it->first.find("."); - // If the database is empty ignore it - if (pos == std::string::npos) - { - continue; - } - std::string db = it->first.substr(0, pos); - - if (db.compare(database) == 0) - { - std::string table = it->first.substr(pos + 1); - table_names.push_back(table); - } - } - - std::unique_ptr set = ResultSet::create({"Table"}); - - for (const auto& name : table_names) - { - set->add_row({name}); - } - - set->write(m_client); - - MXS_FREE(query); - return true; -} - bool SchemaRouterSession::handle_statement(GWBUF* querybuf, SSRBackend& bref, uint8_t command, uint32_t type) { bool succp = false; diff --git a/server/modules/routing/schemarouter/schemaroutersession.hh b/server/modules/routing/schemarouter/schemaroutersession.hh index f1540d91e..d1d870e22 100644 --- a/server/modules/routing/schemarouter/schemaroutersession.hh +++ b/server/modules/routing/schemarouter/schemaroutersession.hh @@ -144,7 +144,6 @@ private: /** Shard mapping functions */ void send_databases(); bool send_shards(); - bool send_tables(GWBUF* pPacket); void query_databases(); int inspect_mapping_states(SSRBackend& bref, GWBUF** wbuf); enum showdb_response parse_mapping_response(SSRBackend& bref, GWBUF** buffer);