hcq/MaxScale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge commit 'a78f0fbe2537542dc7f3f0dd8b19b93ac8d9d7f8' into develop

Browse Source
This commit is contained in:
Markus Mäkelä
2019-03-28 13:53:40 +02:00
parent 18cd890474 a78f0fbe25
commit 6bc2c54081
22 changed files with 535 additions and 194 deletions
Download Patch File Download Diff File Expand all files Collapse all files

29
Documentation/About/Limitations.md
View File

@ -262,32 +262,5 @@ variables to the master.
### Schemarouter limitations (schemarouter)
The schemarouter currently has some limitations due to the nature of the
sharding implementation and the way the session variables are detected and
routed. Here is a list of the current limitations:
Please see [SchemaRouter documentation](../Routers/SchemaRouter.md).
* Cross-database queries (e.g. `SELECT column FROM database1.table UNION select
column FROM database2.table`) are not supported and are routed either to the
first explicit database in the query, the current database in use or to the
first available database, depending on which succeeds.
* Without a default database, queries without explicit databases that do not
modify the session state will be routed to the first available server. This
means that, for example when creating a new database, queries should be done
directly on the node or the router should be equipped with the hint filter and a
routing hint should be used. Queries that modify the session state (e.g. `SET
autocommit=1`) will be routed to all servers regardless of the default database.
* SELECT queries that modify session variables are not currently supported because
uniform results can not be guaranteed. If such a query is executed, the behavior
of the router is undefined. To work around this limitation, the query must be
executed in separate parts.
* If a query targets a database the schemarouter hasn't mapped to a server, the
query will be routed to the first available server. This possibly returns an
error about database rights instead of a missing database.
* The preparation of a prepared statement is routed to all servers. The
execution of a prepared statement is routed to the first available server or to
the server pointed by a routing hint attached to the query. In practice this
means that prepared statements aren't supported by the schemarouter.

76
Documentation/Filters/Masking.md
View File

@ -5,26 +5,7 @@ This filter was introduced in MariaDB MaxScale 2.1.
Table of Contents
=================
* [Overview](#overview)
* [Security](#security)
* [Limitations](#limitations)
* [Configuration](#configuration)
* [Filter Parameters](#filter-parameters)
* [rules](#rules)
* [warn_type_mismatch](#warn_type_mismatch)
* [large_payload](#large_payload)
* [prevent_function_usage](#prevent_function_usage)
* [Rules](#rules-1)
* [replace](#replace)
* [obfuscate](#obfuscate)
* [with](#with)
* [applies_to](#applies_to)
* [exempted](#exempted)
* [Module commands](#module-commands)
* [reload](#reload)
* [Example](#example)
* [Configuration](#configuration-1)
* [masking_rules.json](#masking_rulesjson)
[TOC]
## Overview
@ -79,6 +60,33 @@ Please see the configuration parameter
[check_user_variables](#check_user_variables)
for how to change the default behaviour.
From MaxScale 2.3.5 onwards, the masking filter will examine unions
and if the second or subsequent SELECT refer to columns that should
be masked, the statement will be rejected.
Please see the configuration parameter
[check_unions](#check_unions)
for how to change the default behaviour.
From MaxScale 2.3.5 onwards, the masking filter will examine subqueries
and if a subquery refers to columns that should be masked, the statement
will be rejected.
Please see the configuration parameter
[check_subqueries](#check_subqueries)
for how to change the default behaviour.
Note that in order to ensure that it is not possible to get access to
masked data, the privileges of the users should be minimized. For instance,
if a user can create tables and perform inserts, he or she can execute
something like
```
CREATE TABLE cheat (revealed_ssn TEXT);
INSERT INTO cheat SELECT ssn FROM users;
SELECT revealed_ssn FROM cheat;
```
to get access to the cleartext version of a masked field `ssn`.
## Limitations
The masking filter can _only_ be used for masking columns of the following
@ -192,6 +200,34 @@ check_user_variables=false
The default value is `true`.
#### `check_unions`
This optional parameter specifies how the masking filter should
behave with respect to UNIONs. If true, then a statement like
```
SELECT a FROM t1 UNION select b from t2;
```
will be rejected if `b` is a column that should be masked.
```
check_unions=false
```
The default value is `true`.
#### `check_subqueries`
This optional parameter specifies how the masking filter should
behave with respect to subqueries. If true, then a statement like
```
SELECT * FROM (SELECT a as b FROM t1) as t2;
```
will be rejected if `a` is a column that should be masked.
```
check_subqueries=false
```
The default value is `true`.
## Rules
The masking rules are expressed as a JSON object.

7
Documentation/Monitors/ColumnStore-Monitor.md
View File

@ -20,10 +20,9 @@ GRANT ALL ON infinidb_vtable.* TO 'maxuser'@'%';
## Master Selection
The automatic master detection only works with ColumnStore 1.1.7 (planned
version at the time of writing). Older versions of ColumnStore do not implement
the required functionality to automatically detect which of the servers is the
primary UM.
The automatic master detection only works with ColumnStore 1.2. Older versions
of ColumnStore do not implement the required functionality to automatically
detect which of the servers is the primary UM.
With older versions the `primary` parameter must be defined to tell the monitor
which of the servers is the primary UM node. This guarantees that DDL statements

55
Documentation/Routers/SchemaRouter.md
View File

@ -8,8 +8,12 @@ database-based sharding, the schemarouter also enables cross-node
session variable usage by routing all queries that modify the session to all
nodes.
From 2.3.0 onwards, the SchemaRouter is capable of table family sharding,
in addition to being capable of sharding databases.
The main limitation of SchemaRouter is that aside from session variable writes
and some specific queries, a query can only target one server. This means that
queries which depend on results from multiple servers give incorrect results.
See [Limitations](#limitations) for more information.
From 2.3.0 onwards, SchemaRouter is capable of limited table family sharding.
Table of Contents
=================
@ -50,6 +54,19 @@ In almost all the cases these can be avoided by proper server configuration
and the databases are always mapped to the same servers. More on
configuration in the next chapter.
To check how databases and tables map to servers, execute the special query
`SHOW SHARDS`. The query does not support any modifiers such as `LIKE`.
```
show shards;
Database |Server |
---------|-------------|
db1.t1 |MyServer1 |
db1.t2 |MyServer1 |
db2.t1 |MyServer2 |
```
## Configuration
Here is an example configuration of the schemarouter:
@ -185,8 +202,38 @@ The minimum interval between database map refreshes in seconds.
## Limitations
For a list of schemarouter limitations, please read the
[Limitations](../About/Limitations.md) document.
1. Cross-database queries (e.g. `SELECT column FROM database1.table UNION select column
FROM database2.table`) are not properly supported. Such queries are routed either to the
first explicit database in the query, the current database in use or to the first
available database, depending on which succeeds.
* Without a default database, queries without explicit databases that do not modify the
session state will be routed to the first available server. This includes queries such as
`CREATE DATABASE db1`. Such queries should be done directly on the node or the router
should be equipped with the hint filter and a routing hint should be used. Queries that
modify the session state (e.g. `SET autocommit=1`) will be routed to all servers
regardless of the default database.
* SELECT queries that modify session variables are not supported because uniform results
can not be guaranteed. If such a query is executed, the behavior of the router is
undefined. To work around this limitation, the query must be executed in separate parts.
* If a query targets a database the SchemaRouter has not mapped to a server, the
query will be routed to the first available server. This possibly returns an
error about database rights instead of a missing database.
* The preparation of a prepared statement is routed to all servers. The
execution of a prepared statement is routed to the first available server or to
the server pointed by a routing hint attached to the query. In practice this
means that prepared statements aren't supported by the SchemaRouter.
* `SHOW DATABASES` is handled by the router instead of routed to a server. The router only
answers correctly to the basic version of the query. Any modifiers such as `LIKE` are
ignored.
* `SHOW TABLES` is routed to the server with the current database. If using table-level
sharding, the results will be incomplete. Use `SHOW SHARDS` to get results from the router
itself.
## Examples

1
include/maxscale/dcb.hh
View File

@ -208,6 +208,7 @@ struct DCB : public MXB_POLL_DATA
DCB* tail = nullptr; /**< Last DCB in owning thread's list */
} thread;
uint32_t n_close = 0; /** How many times dcb_close has been called. */
uint64_t m_uid; /**< Unique identifier for this DCB */
};
/**

12
include/maxscale/query_classifier.hh
View File

@ -125,13 +125,23 @@ enum qc_parse_result_t
};
/**
* QC_FIELD_INFO contains information about a field used in a statement.
* qc_field_context_t defines the context where a field appears.
*
* NOTE: A particular bit does NOT mean that the field appears ONLY in the context,
* but it may appear in other contexts as well.
*/
typedef enum qc_field_context
{
QC_FIELD_UNION = 1, /** The field appears on the right hand side in a UNION. */
QC_FIELD_SUBQUERY = 2 /** The field appears in a subquery. */
} qc_field_context_t;
struct QC_FIELD_INFO
{
char* database; /** Present if the field is of the form "a.b.c", NULL otherwise. */
char* table; /** Present if the field is of the form "a.b", NULL otherwise. */
char* column; /** Always present. */
uint32_t context; /** The context in which the field appears. */
};
/**

18
maxscale-system-test/masking_auto_firewall.cpp
View File

@ -110,6 +110,24 @@ void run(TestConnections& test)
// This should NOT succeed as a masked column is used in a statement
// defining a variable.
test_one(test, "set @a = (SELECT a, b FROM masking_auto_firewall)", Expect::FAILURE);
// This SHOULD succeed as a masked column is not used in the statment.
test_one(test, "select 1 UNION select b FROM masking_auto_firewall", Expect::SUCCESS);
// This should NOT succeed as a masked column is used in the statment.
test_one(test, "select 1 UNION select a FROM masking_auto_firewall", Expect::FAILURE);
// This should NOT succeed as '*' is used in the statment.
test_one(test, "select 1 UNION select * FROM masking_auto_firewall", Expect::FAILURE);
// This SHOULD succeed as a masked column is not used in the statment.
test_one(test, "select * FROM (select b from masking_auto_firewall)", Expect::SUCCESS);
// This SHOULD succeed as a masked column is not used in the statment.
test_one(test, "select * FROM (select a as b from masking_auto_firewall)", Expect::FAILURE);
// This SHOULD succeed as '*' is used in the statment.
test_one(test, "select * FROM (select * from masking_auto_firewall)", Expect::FAILURE);
}
}

6
query_classifier/qc_mysqlembedded/qc_mysqlembedded.cc
View File

@ -1848,14 +1848,14 @@ int32_t qc_mysql_get_database_names(GWBUF* querybuf, char*** databasesp, int* si
goto retblock;
}
if (lex->describe || is_show_command(lex->sql_command))
if (lex->describe || (is_show_command(lex->sql_command) && !(lex->sql_command == SQLCOM_SHOW_TABLES)))
{
goto retblock;
}
if (lex->sql_command == SQLCOM_CHANGE_DB)
if (lex->sql_command == SQLCOM_CHANGE_DB || lex->sql_command == SQLCOM_SHOW_TABLES)
{
if (lex->select_lex.db)
if (lex->select_lex.db && (strcmp(lex->select_lex.db, "skygw_virtual") != 0))
{
if (i >= currsz)
{

107
query_classifier/qc_sqlite/qc_sqlite.cc
View File

@ -635,6 +635,7 @@ public:
};
void update_field_info(const QcAliases* pAliases,
uint32_t context,
const char* zDatabase,
const char* zTable,
const char* zColumn,
@ -680,6 +681,7 @@ public:
item.table = zTable ? MXS_STRDUP(zTable) : NULL;
mxb_assert(zColumn);
item.column = MXS_STRDUP(zColumn);
item.context = context;
// We are happy if we at least could dup the column.
@ -689,6 +691,10 @@ public:
}
}
}
else
{
i->context |= context;
}
}
void update_names(const char* zDatabase, const char* zTable, const char* zAlias, QcAliases* pAliases)
@ -798,6 +804,7 @@ public:
}
void update_field_infos(QcAliases* pAliases,
uint32_t context,
int prev_token,
const Expr* pExpr,
qc_token_position_t pos,
@ -812,15 +819,15 @@ public:
switch (pExpr->op)
{
case TK_ASTERISK: // select *
update_field_infos_from_expr(pAliases, pExpr, pExclude);
update_field_infos_from_expr(pAliases, context, pExpr, pExclude);
break;
case TK_DOT: // select a.b ... select a.b.c
update_field_infos_from_expr(pAliases, pExpr, pExclude);
update_field_infos_from_expr(pAliases, context, pExpr, pExclude);
break;
case TK_ID: // select a
update_field_infos_from_expr(pAliases, pExpr, pExclude);
update_field_infos_from_expr(pAliases, context, pExpr, pExclude);
break;
case TK_VARIABLE:
@ -1007,12 +1014,12 @@ public:
if (pLeft)
{
update_field_infos(pAliases, pExpr->op, pExpr->pLeft, QC_TOKEN_LEFT, pExclude);
update_field_infos(pAliases, context, pExpr->op, pExpr->pLeft, QC_TOKEN_LEFT, pExclude);
}
if (pRight)
{
update_field_infos(pAliases, pExpr->op, pExpr->pRight, QC_TOKEN_RIGHT, pExclude);
update_field_infos(pAliases, context, pExpr->op, pExpr->pRight, QC_TOKEN_RIGHT, pExclude);
}
if (pExpr->x.pList)
@ -1022,7 +1029,7 @@ public:
case TK_FUNCTION:
if (!ignore_exprlist)
{
update_field_infos_from_exprlist(pAliases, pExpr->x.pList, pExclude);
update_field_infos_from_exprlist(pAliases, context, pExpr->x.pList, pExclude);
}
break;
@ -1046,7 +1053,7 @@ public:
if (pExpr->flags & EP_xIsSelect)
{
mxb_assert(pAliases);
update_field_infos_from_subselect(*pAliases, pExpr->x.pSelect, pExclude);
update_field_infos_from_subselect(*pAliases, context, pExpr->x.pSelect, pExclude);
if (zName)
@ -1059,7 +1066,7 @@ public:
}
else
{
update_field_infos_from_exprlist(pAliases, pExpr->x.pList, pExclude);
update_field_infos_from_exprlist(pAliases, context, pExpr->x.pList, pExclude);
if (zName)
{
@ -1151,6 +1158,7 @@ public:
}
void update_field_infos_from_expr(QcAliases* pAliases,
uint32_t context,
const Expr* pExpr,
const ExprList* pExclude)
{
@ -1162,12 +1170,13 @@ public:
{
if (get_field_name(pExpr, &zDatabase, &zTable, &zColumn))
{
update_field_info(pAliases, zDatabase, zTable, zColumn, pExclude);
update_field_info(pAliases, context, zDatabase, zTable, zColumn, pExclude);
}
}
}
void update_field_infos_from_exprlist(QcAliases* pAliases,
uint32_t context,
const ExprList* pEList,
const ExprList* pExclude)
{
@ -1175,11 +1184,12 @@ public:
{
ExprList::ExprList_item* pItem = &pEList->a[i];
update_field_infos(pAliases, 0, pItem->pExpr, QC_TOKEN_MIDDLE, pExclude);
update_field_infos(pAliases, context, 0, pItem->pExpr, QC_TOKEN_MIDDLE, pExclude);
}
}
void update_field_infos_from_idlist(QcAliases* pAliases,
uint32_t context,
const IdList* pIds,
const ExprList* pExclude)
{
@ -1189,7 +1199,7 @@ public:
{
IdList::IdList_item* pItem = &pIds->a[i];
update_field_info(pAliases, NULL, NULL, pItem->zName, pExclude);
update_field_info(pAliases, context, NULL, NULL, pItem->zName, pExclude);
}
}
}
@ -1201,6 +1211,7 @@ public:
};
void update_field_infos_from_select(QcAliases& aliases,
uint32_t context,
const Select* pSelect,
const ExprList* pExclude,
compound_approach_t compound_approach = ANALYZE_COMPOUND_SELECTS)
@ -1218,7 +1229,7 @@ public:
if (pSrc->a[i].pSelect)
{
update_field_infos_from_select(aliases, pSrc->a[i].pSelect, pExclude);
update_field_infos_from_select(aliases, context | QC_FIELD_SUBQUERY, pSrc->a[i].pSelect, pExclude);
}
#ifdef QC_COLLECT_NAMES_FROM_USING
@ -1236,13 +1247,14 @@ public:
if (pSelect->pEList)
{
update_field_infos_from_exprlist(&aliases, pSelect->pEList, NULL);
update_field_infos_from_exprlist(&aliases, context, pSelect->pEList, NULL);
}
if (pSelect->pWhere)
{
m_has_clause = true;
update_field_infos(&aliases,
context,
0,
pSelect->pWhere,
QC_TOKEN_MIDDLE,
@ -1252,6 +1264,7 @@ public:
if (pSelect->pGroupBy)
{
update_field_infos_from_exprlist(&aliases,
context,
pSelect->pGroupBy,
pSelect->pEList);
}
@ -1268,7 +1281,7 @@ public:
if (pSelect->pWith)
{
update_field_infos_from_with(&aliases, pSelect->pWith);
update_field_infos_from_with(&aliases, context, pSelect->pWith);
}
if (compound_approach == ANALYZE_COMPOUND_SELECTS)
@ -1279,10 +1292,22 @@ public:
while (pPrior)
{
update_field_infos_from_subselect(aliases,
pPrior,
pExclude,
IGNORE_COMPOUND_SELECTS);
uint32_t ctx = context;
if (!pPrior->pPrior)
{
// The fields in the first select in a UNION are not considered to
// be in a union. Those names will be visible in the resultset.
ctx &= ~QC_FIELD_UNION;
}
QcAliases aliases2(aliases);
update_field_infos_from_select(aliases2,
ctx,
pPrior,
pExclude,
IGNORE_COMPOUND_SELECTS);
pPrior = pPrior->pPrior;
}
}
@ -1290,16 +1315,19 @@ public:
}
void update_field_infos_from_subselect(const QcAliases& existing_aliases,
uint32_t context,
const Select* pSelect,
const ExprList* pExclude,
compound_approach_t compound_approach = ANALYZE_COMPOUND_SELECTS)
{
QcAliases aliases(existing_aliases);
update_field_infos_from_select(aliases, pSelect, pExclude, compound_approach);
context |= QC_FIELD_SUBQUERY;
update_field_infos_from_select(aliases, context, pSelect, pExclude, compound_approach);
}
void update_field_infos_from_with(QcAliases* pAliases, const With* pWith)
void update_field_infos_from_with(QcAliases* pAliases, uint32_t context, const With* pWith)
{
for (int i = 0; i < pWith->nCte; ++i)
{
@ -1308,7 +1336,7 @@ public:
if (pCte->pSelect)
{
mxb_assert(pAliases);
update_field_infos_from_subselect(*pAliases, pCte->pSelect, NULL);
update_field_infos_from_subselect(*pAliases, context, pCte->pSelect, NULL);
}
}
}
@ -1668,7 +1696,8 @@ public:
if (pSelect)
{
update_field_infos_from_select(aliases, pSelect, NULL);
uint32_t context = 0;
update_field_infos_from_select(aliases, context, pSelect, NULL);
}
exposed_sqlite3ExprListDelete(pParse->db, pCNames);
@ -1738,7 +1767,8 @@ public:
if (pWhere)
{
update_field_infos(&aliases, 0, pWhere, QC_TOKEN_MIDDLE, 0);
uint32_t context = 0;
update_field_infos(&aliases, context, 0, pWhere, QC_TOKEN_MIDDLE, 0);
}
}
@ -1793,7 +1823,8 @@ public:
if (pSelect)
{
QcAliases aliases;
update_field_infos_from_select(aliases, pSelect, NULL);
uint32_t context = 0;
update_field_infos_from_select(aliases, context, pSelect, NULL);
}
else if (pOldTable)
{
@ -1821,12 +1852,13 @@ public:
mxb_assert(pTabList->nSrc >= 1);
QcAliases aliases;
uint32_t context = 0;
update_names_from_srclist(&aliases, pTabList);
if (pColumns)
{
update_field_infos_from_idlist(&aliases, pColumns, NULL);
update_field_infos_from_idlist(&aliases, context, pColumns, NULL);
int i = update_function_info(&aliases, "=", NULL);
@ -1851,12 +1883,12 @@ public:
if (pSelect)
{
update_field_infos_from_select(aliases, pSelect, NULL);
update_field_infos_from_select(aliases, context, pSelect, NULL);
}
if (pSet)
{
update_field_infos_from_exprlist(&aliases, pSet, NULL);
update_field_infos_from_exprlist(&aliases, context, pSet, NULL);
}
}
@ -1958,6 +1990,7 @@ public:
if (m_operation != QUERY_OP_EXPLAIN)
{
QcAliases aliases;
uint32_t context = 0;
m_type_mask = QUERY_TYPE_WRITE;
m_operation = QUERY_OP_UPDATE;
@ -1971,6 +2004,7 @@ public:
ExprList::ExprList_item* pItem = &pChanges->a[i];
update_field_infos(&aliases,
context,
0,
pItem->pExpr,
QC_TOKEN_MIDDLE,
@ -1980,7 +2014,7 @@ public:
if (pWhere)
{
update_field_infos(&aliases, 0, pWhere, QC_TOKEN_MIDDLE, pChanges);
update_field_infos(&aliases, context, 0, pWhere, QC_TOKEN_MIDDLE, pChanges);
}
}
@ -2039,7 +2073,8 @@ public:
}
QcAliases aliases;
update_field_infos_from_select(aliases, pSelect, NULL);
uint32_t context = (pSelect->op == TK_UNION && pSelect->pPrior) ? QC_FIELD_UNION : 0;
update_field_infos_from_select(aliases, context, pSelect, NULL);
}
void maxscaleAlterTable(Parse* pParse, /* Parser context. */
@ -2085,7 +2120,8 @@ public:
if (pExprList)
{
QcAliases aliases;
update_field_infos_from_exprlist(&aliases, pExprList, NULL);
uint32_t context = 0;
update_field_infos_from_exprlist(&aliases, context, pExprList, NULL);
}
exposed_sqlite3SrcListDelete(pParse->db, pName);
@ -2906,7 +2942,8 @@ public:
if (pValue->op == TK_SELECT)
{
QcAliases aliases;
update_field_infos_from_select(aliases, pValue->x.pSelect, NULL);
uint32_t context = 0;
update_field_infos_from_select(aliases, context, pValue->x.pSelect, NULL);
}
}
break;
@ -2993,6 +3030,14 @@ public:
case MXS_SHOW_TABLES:
m_type_mask = QUERY_TYPE_SHOW_TABLES;
if (pShow->pDatabase->z)
{
char db[pShow->pDatabase->n + 1];
strncpy(db, pShow->pDatabase->z, pShow->pDatabase->n);
db[pShow->pDatabase->n] = 0;
update_database_names(db);
}
break;
case MXS_SHOW_VARIABLES:

34
query_classifier/test/compare.cc
View File

@ -876,6 +876,7 @@ public:
: m_database(info.database ? info.database : "")
, m_table(info.table ? info.table : "")
, m_column(info.column ? info.column : "")
, m_context(info.context)
{
}
@ -939,12 +940,33 @@ public:
}
out << m_column;
if (m_context != 0)
{
out << "(";
bool first = true;
if (m_context & QC_FIELD_UNION)
{
out << (first ? "" : ", ") << "UNION";
first = false;
}
if (m_context & QC_FIELD_SUBQUERY)
{
out << (first ? "" : ", ") << "SUBQUERY";
first = false;
}
out << ")";
}
}
private:
std::string m_database;
std::string m_table;
std::string m_column;
uint32_t m_context;
};
ostream& operator<<(ostream& out, const QcFieldInfo& x)
@ -1010,7 +1032,17 @@ bool compare_get_field_info(QUERY_CLASSIFIER* pClassifier1,
if (f1 == f2)
{
ss << "Ok : ";
ss << f1;
// TODO: Currently qc_sqlite provides context information, while qc_mysqlembedded
// TODO: does not. To ensure that the output always contains the maximum amount
// TODO: of information, we simply generate both output and print the longest.
stringstream ss1;
ss1 << f1;
stringstream ss2;
ss2 << f2;
ss << (ss1.str().length() > ss2.str().length() ? ss1.str() : ss2.str());
success = true;
}
else

8
server/core/dcb.cc
View File

@ -38,6 +38,8 @@
#include <sys/un.h>
#include <time.h>
#include <atomic>
#include <maxscale/alloc.h>
#include <maxbase/atomic.h>
#include <maxbase/atomic.hh>
@ -78,6 +80,7 @@ static struct
{
DCB** all_dcbs; /** #workers sized array of pointers to DCBs where dcbs are listed. */
bool check_timeouts; /** Should session timeouts be checked. */
std::atomic<uint64_t> uid_generator {0};
} this_unit;
static thread_local struct
@ -161,6 +164,7 @@ DCB::DCB(Role role, MXS_SESSION* session)
, low_water(config_writeq_low_water())
, service(session->service)
, last_read(mxs_clock())
, m_uid(this_unit.uid_generator.fetch_add(1, std::memory_order_relaxed))
{
// TODO: Remove DCB::Role::INTERNAL to always have a valid listener
if (session->listener)
@ -2805,6 +2809,7 @@ public:
: m_dcb(dcb)
, m_buffer(buf)
, m_ev(ev)
, m_uid(dcb->m_uid)
{
}
@ -2813,7 +2818,7 @@ public:
mxb_assert(&worker == RoutingWorker::get_current());
RoutingWorker& rworker = static_cast<RoutingWorker&>(worker);
if (dcb_is_still_valid(m_dcb, rworker.id()))
if (dcb_is_still_valid(m_dcb, rworker.id()) && m_dcb->m_uid == m_uid)
{
m_dcb->fakeq = m_buffer;
dcb_handler(m_dcb, m_ev);
@ -2828,6 +2833,7 @@ private:
DCB* m_dcb;
GWBUF* m_buffer;
uint32_t m_ev;
uint64_t m_uid; /**< DCB UID guarantees we deliver the event to the correct DCB */
};
static void poll_add_event_to_dcb(DCB* dcb, GWBUF* buf, uint32_t ev)

6
server/modules/filter/binlogfilter/binlogfiltersession.cc
View File

@ -405,6 +405,12 @@ static bool should_skip_query(const BinlogConfig& config, const std::string& sql
qc_free_table_names(names, n);
}
// Also check for the default database in case the query has no tables in it
if (!rval && should_skip(config, db))
{
rval = true;
}
gwbuf_free(buf);
return rval;
}

12
server/modules/filter/masking/maskingfilter.cc
View File

@ -127,6 +127,18 @@ extern "C" MXS_MODULE* MXS_CREATE_MODULE()
Config::check_user_variables_default,
MXS_MODULE_OPT_NONE,
},
{
Config::check_unions_name,
MXS_MODULE_PARAM_BOOL,
Config::check_unions_default,
MXS_MODULE_OPT_NONE,
},
{
Config::check_subqueries_name,
MXS_MODULE_PARAM_BOOL,
Config::check_subqueries_default,
MXS_MODULE_OPT_NONE,
},
{MXS_END_MODULE_PARAMS}
}
};

34
server/modules/filter/masking/maskingfilterconfig.cc
View File

@ -28,6 +28,8 @@ const char config_value_always[] = "always";
const char config_name_prevent_function_usage[] = "prevent_function_usage";
const char config_check_user_variables[] = "check_user_variables";
const char config_check_unions[] = "check_unions";
const char config_check_subqueries[] = "check_subqueries";
const char config_value_true[] = "true";
}
@ -88,11 +90,31 @@ const char* MaskingFilterConfig::prevent_function_usage_default = config_value_t
/*
* PARAM check_user_variables
*/
// static
const char* MaskingFilterConfig::check_user_variables_name = config_check_user_variables;
// static
const char* MaskingFilterConfig::check_user_variables_default = config_value_true;
/*
* PARAM check_unions
*/
// static
const char* MaskingFilterConfig::check_unions_name = config_check_unions;
// static
const char* MaskingFilterConfig::check_unions_default = config_value_true;
/*
* PARAM check_subqueries
*/
// static
const char* MaskingFilterConfig::check_subqueries_name = config_check_subqueries;
// static
const char* MaskingFilterConfig::check_subqueries_default = config_value_true;
/*
* MaskingFilterConfig
*/
@ -130,3 +152,15 @@ bool MaskingFilterConfig::get_check_user_variables(const MXS_CONFIG_PARAMETER* p
{
return pParams->get_bool(check_user_variables_name);
}
// static
bool MaskingFilterConfig::get_check_unions(const MXS_CONFIG_PARAMETER* pParams)
{
return pParams->get_bool(check_unions_name);
}
// static
bool MaskingFilterConfig::get_check_subqueries(const MXS_CONFIG_PARAMETER* pParams)
{
return pParams->get_bool(check_subqueries_name);
}

35
server/modules/filter/masking/maskingfilterconfig.hh
View File

@ -48,6 +48,12 @@ public:
static const char* check_user_variables_name;
static const char* check_user_variables_default;
static const char* check_unions_name;
static const char* check_unions_default;
static const char* check_subqueries_name;
static const char* check_subqueries_default;
MaskingFilterConfig(const char* zName, const MXS_CONFIG_PARAMETER* pParams)
: m_name(zName)
, m_large_payload(get_large_payload(pParams))
@ -55,8 +61,11 @@ public:
, m_warn_type_mismatch(get_warn_type_mismatch(pParams))
, m_prevent_function_usage(get_prevent_function_usage(pParams))
, m_check_user_variables(get_check_user_variables(pParams))
, m_check_unions(get_check_unions(pParams))
, m_check_subqueries(get_check_subqueries(pParams))
{
}
~MaskingFilterConfig()
{
}
@ -91,6 +100,16 @@ public:
return m_check_user_variables;
}
bool check_unions() const
{
return m_check_unions;
}
bool check_subqueries() const
{
return m_check_subqueries;
}
void set_large_payload(large_payload_t l)
{
m_large_payload = l;
@ -115,9 +134,19 @@ public:
m_check_user_variables = b;
}
void set_check_unions(bool b)
{
m_check_unions = b;
}
void set_check_subqueries(bool b)
{
m_check_subqueries = b;
}
bool is_parsing_needed() const
{
return prevent_function_usage() || check_user_variables();
return prevent_function_usage() || check_user_variables() || check_unions() || check_subqueries();
}
static large_payload_t get_large_payload(const MXS_CONFIG_PARAMETER* pParams);
@ -125,6 +154,8 @@ public:
static warn_type_mismatch_t get_warn_type_mismatch(const MXS_CONFIG_PARAMETER* pParams);
static bool get_prevent_function_usage(const MXS_CONFIG_PARAMETER* pParams);
static bool get_check_user_variables(const MXS_CONFIG_PARAMETER* pParams);
static bool get_check_unions(const MXS_CONFIG_PARAMETER* pParams);
static bool get_check_subqueries(const MXS_CONFIG_PARAMETER* pParams);
private:
std::string m_name;
@ -133,4 +164,6 @@ private:
warn_type_mismatch_t m_warn_type_mismatch;
bool m_prevent_function_usage;
bool m_check_user_variables;
bool m_check_unions;
bool m_check_subqueries;
};

178
server/modules/filter/masking/maskingfiltersession.cc
View File

@ -82,25 +82,45 @@ bool MaskingFilterSession::check_query(GWBUF* pPacket)
zHost = "";
}
bool rv = true;
bool acceptable = true;
if (rv && m_filter.config().prevent_function_usage())
const MaskingFilter::Config& config = m_filter.config();
if (qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_USERVAR_WRITE))
{
if (is_function_used(pPacket, zUser, zHost))
if (config.check_user_variables())
{
rv = false;
if (is_variable_defined(pPacket, zUser, zHost))
{
acceptable = false;
}
}
}
else
{
qc_query_op_t op = qc_get_operation(pPacket);
if (op == QUERY_OP_SELECT)
{
if (config.check_unions() || config.check_subqueries())
{
if (is_union_or_subquery_used(pPacket, zUser, zHost))
{
acceptable = false;
}
}
}
if (acceptable && config.prevent_function_usage())
{
if (is_function_used(pPacket, zUser, zHost))
{
acceptable = false;
}
}
}
if (rv && m_filter.config().check_user_variables())
{
if (is_variable_defined(pPacket, zUser, zHost))
{
rv = false;
}
}
return rv;
return acceptable;
}
bool MaskingFilterSession::check_textual_query(GWBUF* pPacket)
@ -551,19 +571,26 @@ bool MaskingFilterSession::is_function_used(GWBUF* pPacket, const char* zUser, c
bool MaskingFilterSession::is_variable_defined(GWBUF* pPacket, const char* zUser, const char* zHost)
{
if (!qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_USERVAR_WRITE))
{
return false;
}
mxb_assert(qc_query_is_type(qc_get_type_mask(pPacket), QUERY_TYPE_USERVAR_WRITE));
bool is_defined = false;
SMaskingRules sRules = m_filter.rules();
auto pred = [&sRules, zUser, zHost](const QC_FIELD_INFO& field_info) {
const MaskingRules::Rule* pRule = sRules->get_rule_for(field_info, zUser, zHost);
bool rv = false;
return pRule ? true : false;
if (strcmp(field_info.column, "*") == 0)
{
// If "*" is used, then we must block if there is any rule for the current user.
rv = sRules->has_rule_for(zUser, zHost);
}
else
{
rv = sRules->get_rule_for(field_info, zUser, zHost) ? true : false;
}
return rv;
};
const QC_FIELD_INFO* pInfos;
@ -578,14 +605,121 @@ bool MaskingFilterSession::is_variable_defined(GWBUF* pPacket, const char* zUser
if (i != end)
{
const char* zColumn = i->column;
std::stringstream ss;
ss << "The field " << i->column << " that should be masked for '" << zUser << "'@'" << zHost
<< "' is used when defining a variable, access is denied.";
if (strcmp(zColumn, "*") == 0)
{
ss << "'*' is used in the definition of a variable and there are masking rules "
<< "for '" << zUser << "'@'" << zHost << "', access is denied.";
}
else
{
ss << "The field " << i->column << " that should be masked for '" << zUser << "'@'" << zHost
<< "' is used when defining a variable, access is denied.";
}
set_response(create_error_response(ss.str().c_str()));
is_defined = true;
}
return is_defined;
}
bool MaskingFilterSession::is_union_or_subquery_used(GWBUF* pPacket, const char* zUser, const char* zHost)
{
mxb_assert(qc_get_operation(pPacket) == QUERY_OP_SELECT);
const MaskingFilter::Config& config = m_filter.config();
mxb_assert(config.check_unions() || config.check_subqueries());
bool is_used = false;
SMaskingRules sRules = m_filter.rules();
uint32_t mask = 0;
if (config.check_unions())
{
mask |= QC_FIELD_UNION;
}
if (config.check_subqueries())
{
mask |= QC_FIELD_SUBQUERY;
}
auto pred = [&sRules, mask, zUser, zHost](const QC_FIELD_INFO& field_info) {
bool rv = false;
if (field_info.context & mask)
{
if (strcmp(field_info.column, "*") == 0)
{
// If "*" is used, then we must block if there is any rule for the current user.
rv = sRules->has_rule_for(zUser, zHost);
}
else
{
rv = sRules->get_rule_for(field_info, zUser, zHost) ? true : false;
}
}
return rv;
};
const QC_FIELD_INFO* pInfos;
size_t nInfos;
qc_get_field_info(pPacket, &pInfos, &nInfos);
const QC_FIELD_INFO* begin = pInfos;
const QC_FIELD_INFO* end = begin + nInfos;
auto i = std::find_if(begin, end, pred);
if (i != end)
{
const char* zColumn = i->column;
std::stringstream ss;
if (config.check_unions() && (i->context & QC_FIELD_UNION))
{
if (strcmp(zColumn, "*") == 0)
{
ss << "'*' is used in the second or subsequent SELECT of a UNION and there are "
<< "masking rules for '" << zUser << "'@'" << zHost << "', access is denied.";
}
else
{
ss << "The field " << zColumn << " that should be masked for '" << zUser << "'@'" << zHost
<< "' is used in the second or subsequent SELECT of a UNION, access is denied.";
}
}
else if (config.check_subqueries() && (i->context & QC_FIELD_SUBQUERY))
{
if (strcmp(zColumn, "*") == 0)
{
ss << "'*' is used in a subquery and there are masking rules for '"
<< zUser << "'@'" << zHost << "', access is denied.";
}
else
{
ss << "The field " << zColumn << " that should be masked for '"
<< zUser << "'@'" << zHost << "' is used in a subquery, access is denied.";
}
}
else
{
mxb_assert(!true);
}
set_response(create_error_response(ss.str().c_str()));
is_used = true;
}
return is_used;
}

1
server/modules/filter/masking/maskingfiltersession.hh
View File

@ -67,6 +67,7 @@ private:
bool is_function_used(GWBUF* pPacket, const char* zUser, const char* zHost);
bool is_variable_defined(GWBUF* pPacket, const char* zUser, const char* zHost);
bool is_union_or_subquery_used(GWBUF* pPacket, const char* zUser, const char* zHost);
private:
typedef std::shared_ptr<MaskingRules> SMaskingRules;

9
server/modules/filter/masking/maskingrules.cc
View File

@ -1467,3 +1467,12 @@ const MaskingRules::Rule* MaskingRules::get_rule_for(const QC_FIELD_INFO& field_
return pRule;
}
bool MaskingRules::has_rule_for(const char* zUser, const char* zHost) const
{
auto i = std::find_if(m_rules.begin(), m_rules.end(), [zUser, zHost](SRule sRule) {
return sRule->matches_account(zUser, zHost);
});
return i != m_rules.end();
}

24
server/modules/filter/masking/maskingrules.hh
View File

@ -143,13 +143,21 @@ public:
*/
virtual void rewrite(LEncString& s) const = 0;
/**
* Does this rule apply to a specific account.
*
* @param zUser The current user.
* @param zHost The current host.
*
* @return True, if the rule applies.
*/
bool matches_account(const char* zUser,
const char* zHost) const;
private:
Rule(const Rule&);
Rule& operator=(const Rule&);
bool matches_account(const char* zUser,
const char* zHost) const;
private:
std::string m_column;
std::string m_table;
@ -396,6 +404,16 @@ public:
typedef std::shared_ptr<Rule> SRule;
/**
* Is there any rule for the specified user.
*
* @param zUser The current user.
* @param zHost The current host.
*
* @return True, if there is a rule for that user/host combination.
*/
bool has_rule_for(const char* zUser, const char* zHost) const;
private:
MaskingRules(json_t* pRoot, const std::vector<SRule>& rules);

5
server/modules/monitor/csmon/csmon.cc
View File

@ -104,10 +104,9 @@ void CsMonitor::update_server_status(MonitorServer* srv)
{
status |= SERVER_RUNNING;
if (get_cs_version(srv) >= 10107)
if (get_cs_version(srv) >= 10200)
{
// 1.1.7 should support the mcsSystemPrimary function
// TODO: Update when the actual release is out
// 1.2 supports the mcsSystemPrimary function
status |= do_query(srv, role_query) == "1" ? SERVER_MASTER : SERVER_SLAVE;
}
else

71
server/modules/routing/schemarouter/schemaroutersession.cc
View File

@ -342,14 +342,6 @@ int32_t SchemaRouterSession::routeQuery(GWBUF* pPacket)
gwbuf_free(pPacket);
return 1;
}
else if (qc_query_is_type(type, QUERY_TYPE_SHOW_TABLES))
{
if (send_tables(pPacket))
{
gwbuf_free(pPacket);
return 1;
}
}
else if (detect_show_shards(pPacket))
{
if (send_shards())
@ -1576,69 +1568,6 @@ void SchemaRouterSession::send_databases()
set->write(m_client);
}
bool SchemaRouterSession::send_tables(GWBUF* pPacket)
{
char* query = modutil_get_SQL(pPacket);
char* tmp;
std::string database;
if ((tmp = strcasestr(query, "from")))
{
const char* delim = "` \n\t;";
char* saved, * tok = strtok_r(tmp, delim, &saved);
tok = strtok_r(NULL, delim, &saved);
database = tok;
}
if (database.empty())
{
// Was not a "show tables from x". If a current database is selected, use that as target.
if (!m_current_db.empty())
{
database = m_current_db;
}
else
{
// No current db, route the query to a server, likely getting "No database selected"
MXS_FREE(query);
return false;
}
}
ServerMap tablelist;
std::list<std::string> table_names;
m_shard.get_content(tablelist);
for (ServerMap::iterator it = tablelist.begin(); it != tablelist.end(); it++)
{
std::size_t pos = it->first.find(".");
// If the database is empty ignore it
if (pos == std::string::npos)
{
continue;
}
std::string db = it->first.substr(0, pos);
if (db.compare(database) == 0)
{
std::string table = it->first.substr(pos + 1);
table_names.push_back(table);
}
}
std::unique_ptr<ResultSet> set = ResultSet::create({"Table"});
for (const auto& name : table_names)
{
set->add_row({name});
}
set->write(m_client);
MXS_FREE(query);
return true;
}
bool SchemaRouterSession::handle_statement(GWBUF* querybuf, SSRBackend& bref, uint8_t command, uint32_t type)
{
bool succp = false;

1
server/modules/routing/schemarouter/schemaroutersession.hh
View File

@ -144,7 +144,6 @@ private:
/** Shard mapping functions */
void send_databases();
bool send_shards();
bool send_tables(GWBUF* pPacket);
void query_databases();
int inspect_mapping_states(SSRBackend& bref, GWBUF** wbuf);
enum showdb_response parse_mapping_response(SSRBackend& bref, GWBUF** buffer);