hcq/MaxScale
hcq/MaxScale
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

MXS-1475 Add @maxscale.cache.[populate|use]

Browse Source 
The earlier @maxscale.cache.enabled has now been replaced with
@maxscale.cache.populate and @maxscale.cache.use that provide
for more flexibility.

With the former it is possible to control in what circumstances
the cache is populated and with the latter one when it is used.
Together they can be used for having a completely client driven
caching.
This commit is contained in:
Johan Wikman 2018-03-09 13:47:34 +02:00
parent 34dd8a52bb
commit 82b55ff362
6 changed files with 356 additions and 190 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

115
Documentation/Filters/Cache.md
View File

@ -311,39 +311,118 @@ enabled=false
```
Default is `true`.
Note that this affects only whether data from the cache is returned; the
cache will be populated normally. Please see
The value affects the initial state of the MaxScale user
variables using which the behaviour of the cache can be modified
at runtime. Please see
[Runtime Configuration](#runtime-configuation)
for how to enable/disable the cache at runtime.
for details.
## Runtime Configuration
Using the variable `@maxscale.cache.enabled` it is possible to specify at
runtime whether the cache should be used or not. Its initial value is the
value of the configuration parameter `enabled`. That is, by default the
### `@maxscale.cache.populate`
Using the variable `@maxscale.cache.populate` it is possible to specify at
runtime whether the cache should be populated or not. Its initial value is
the value of the configuration parameter `enabled`. That is, by default the
value is `true`.
The purpose of this variable is make it possible for an application to decide
statement by statement whether data from the cache can be returned.
statement by statement whether the cache should be populated.
```
set @maxscale.cache.enabled=false;
SET @maxscale.cache.populate=true;
SELECT a, b FROM tbl;
SET @maxscale.cache.populate=false;
SELECT a, b FROM tbl;
set @maxscale.cache.enabled=true;
select c, d FROM tbl;
```
In the example above, the first `SELECT` will always be sent to the
server, while the latter will be served from the cache, unless the data is
stale.
server and the result will be cached, provided the actual cache rules
specifies that it should be. The second `SELECT` may be served from the
cache, depending on the value of `@maxscale.cache.use` (and the cache
rules).
Note that the value of `@maxscale.cache.enabled` has no impact on the
population of the cache; if a `SELECT` matches the rules then the data
returned from the cache will be populated.
The value of `@maxscale.cache.populate` can be queried
```
SELECT @maxscale.cache.populate;
```
but only _after_ it has been explicitly set once.
The value of `@maxscale.cache.enabled` can be queried
### `@maxscale.cache.use`
Using the variable `@maxscale.cache.use` it is possible to specify at
runtime whether the cache should be used or not. Its initial value is
the value of the configuration parameter `enabled`. That is, by default the
value is `true`.
The purpose of this variable is make it possible for an application to decide
statement by statement whether the cache should be used.
```
select @maxscale.cache.enabled;
SET @maxscale.cache.use=true;
SELECT a, b FROM tbl;
SET @maxscale.cache.use=false;
SELECT a, b FROM tbl;
```
The first `SELECT` will be served from the cache, providing the rules
specify that the statement should be cached, the cache indeed contains
the result and the date is not stale (as specified by the _TTL_).
If the data is stale, the `SELECT` will be sent to the server **and**
the cache entry will be updated, irrespective of the value of
`@maxscale.cache.populate`.
If `@maxscale.cache.use` is `true` but the result is not found in the
cache, and the result is subsequently fetched from the server, the
result will **not** be added to the cache, unless
`@maxscale.cache.populate` is also `true`.
The value of `@maxscale.cache.use` can be queried
```
SELECT @maxscale.cache.use;
```
but only after it has explicitly been set once.
### Client Driven Caching
With `@maxscale.cache.populate` and `@maxscale.cache.use` is it possible
to make the caching completely client driven.
Provide no `rules` file, which means that _all_ `SELECT` statements are
subject to caching and that all users receive data from the cache. Set
the startup mode of the cache to _disabled_.
```
[TheCache]
type=filter
module=cache
enabled=false
```
Now, in order to _mark_ statements that should be cached, set
`@maxscale.cache.populate` to `true`, and perform those `SELECT`s.
```
SET @maxscale.cache.populate=true;
SELECT a, b FROM tbl1;
SELECT c, d FROM tbl2;
SELECT e, f FROM tbl3;
SET @maxscale.cache.populate=false;
```
Note that those `SELECT`s must return something in order for the
statement to be _marked_ for caching.
After this, the value of `@maxscale.cache.use` will decide whether
or not the cache is considered.
```
SET @maxscale.cache.use=true;
SELECT a, b FROM tbl1;
SET @maxscale.cache.use=false;
```
With `@maxscale.cache.use` being `true`, the cache is considered
and the result returned from there, if not stale. If it is stale,
the result is fetched from the server and the cached entry is updated.
By setting a very long _TTL_ it is possible to prevent the cache
from ever considering an entry to be stale and instead manually
cause the cache to be updated when needed.
```
UPDATE tbl1 SET a = ...;
SET @maxscale.cache.populate=true;
SELECT a, b FROM tbl1;
SET @maxscale.cache.populate=false;
```
but only after it has been set once.
# Rules

4
include/maxscale/session.h
View File

@ -133,7 +133,9 @@ typedef struct mxs_upstream
* has been removed. The handler must itself parse the value string.
*
* @param context Context provided when handler was registered.
* @param name The variable that is being set.
* @param name The variable that is being set. Note that it
* will always be in all lower-case irrespective
* of the case used when registering.
* @param value_begin The beginning of the value as specified in the
* "set @maxscale.x.y = VALUE" statement.
* @param value_end One past the end of the VALUE.

4
server/core/session.cc
View File

@ -1119,7 +1119,7 @@ bool session_add_variable(MXS_SESSION* session,
{
string key(name);
std::transform(key.begin(), key.end(), key.begin(), toupper);
std::transform(key.begin(), key.end(), key.begin(), tolower);
if (session->variables->find(key) == session->variables->end())
{
@ -1153,7 +1153,7 @@ char* session_set_variable_value(MXS_SESSION* session,
string key(name_begin, name_end - name_begin);
transform(key.begin(), key.end(), key.begin(), toupper);
transform(key.begin(), key.end(), key.begin(), tolower);
SessionVarsByName::iterator i = session->variables->find(key);

416
server/modules/filter/cache/cachefiltersession.cc vendored
View File

@ -38,7 +38,8 @@ inline bool cache_max_resultset_size_exceeded(const CACHE_CONFIG& config, uint64
namespace
{
const char SV_MAXSCALE_CACHE_ENABLED[] = "@maxscale.cache.enabled";
const char SV_MAXSCALE_CACHE_POPULATE[] = "@maxscale.cache.populate";
const char SV_MAXSCALE_CACHE_USE[] = "@maxscale.cache.use";
const char* NON_CACHEABLE_FUNCTIONS[] =
{
@ -187,32 +188,32 @@ CacheFilterSession::CacheFilterSession(MXS_SESSION* pSession, Cache* pCache, cha
, m_zUseDb(NULL)
, m_refreshing(false)
, m_is_read_only(true)
, m_enabled(pCache->config().enabled)
, m_variable_added(false)
, m_use(pCache->config().enabled)
, m_populate(pCache->config().enabled)
{
m_key.data = 0;
reset_response_state();
if (session_add_variable(pSession, SV_MAXSCALE_CACHE_ENABLED,
&CacheFilterSession::session_variable_handler, this))
{
m_variable_added = true;
}
else
if (!session_add_variable(pSession, SV_MAXSCALE_CACHE_POPULATE,
&CacheFilterSession::session_variable_handler, this))
{
MXS_ERROR("Could not add MaxScale user variable '%s', dynamically "
"enabling/disabling caching not possible.", SV_MAXSCALE_CACHE_ENABLED);
"enabling/disabling the populating of the cache is not possible.",
SV_MAXSCALE_CACHE_POPULATE);
}
if (!session_add_variable(pSession, SV_MAXSCALE_CACHE_USE,
&CacheFilterSession::session_variable_handler, this))
{
MXS_ERROR("Could not add MaxScale user variable '%s', dynamically "
"enabling/disabling the using of the cache not possible.",
SV_MAXSCALE_CACHE_USE);
}
}
CacheFilterSession::~CacheFilterSession()
{
if (m_variable_added)
{
session_remove_variable(m_pSession, SV_MAXSCALE_CACHE_ENABLED, NULL);
}
MXS_FREE(m_zUseDb);
MXS_FREE(m_zDefaultDb);
}
@ -766,163 +767,202 @@ CacheFilterSession::cache_action_t CacheFilterSession::get_cache_action(GWBUF* p
{
cache_action_t action = CACHE_IGNORE;
uint32_t type_mask = qc_get_trx_type_mask(pPacket); // Note, only trx-related type mask
const char* zReason = NULL;
const CACHE_CONFIG& config = m_pCache->config();
if (qc_query_is_type(type_mask, QUERY_TYPE_BEGIN_TRX))
if (m_use || m_populate)
{
if (log_decisions())
{
zReason = "transaction start";
}
uint32_t type_mask = qc_get_trx_type_mask(pPacket); // Note, only trx-related type mask
// When a transaction is started, we initially assume it is read-only.
m_is_read_only = true;
}
else if (!session_trx_is_active(m_pSession))
{
if (log_decisions())
{
zReason = "no transaction";
}
action = CACHE_USE_AND_POPULATE;
}
else if (session_trx_is_read_only(m_pSession))
{
if (config.cache_in_trxs >= CACHE_IN_TRXS_READ_ONLY)
const char* zReason = NULL;
const CACHE_CONFIG& config = m_pCache->config();
if (qc_query_is_type(type_mask, QUERY_TYPE_BEGIN_TRX))
{
if (log_decisions())
{
zReason = "explicitly read-only transaction";
zReason = "transaction start";
}
// When a transaction is started, we initially assume it is read-only.
m_is_read_only = true;
}
else if (!session_trx_is_active(m_pSession))
{
if (log_decisions())
{
zReason = "no transaction";
}
action = CACHE_USE_AND_POPULATE;
}
else
else if (session_trx_is_read_only(m_pSession))
{
ss_dassert(config.cache_in_trxs == CACHE_IN_TRXS_NEVER);
if (log_decisions())
if (config.cache_in_trxs >= CACHE_IN_TRXS_READ_ONLY)
{
zReason = "populating but not using cache inside read-only transactions";
if (log_decisions())
{
zReason = "explicitly read-only transaction";
}
action = CACHE_USE_AND_POPULATE;
}
else
{
ss_dassert(config.cache_in_trxs == CACHE_IN_TRXS_NEVER);
if (log_decisions())
{
zReason = "populating but not using cache inside read-only transactions";
}
action = CACHE_POPULATE;
}
action = CACHE_POPULATE;
}
}
else if (m_is_read_only)
{
// There is a transaction and it is *not* explicitly read-only,
// although so far there has only been SELECTs.
if (config.cache_in_trxs >= CACHE_IN_TRXS_ALL)
else if (m_is_read_only)
{
if (log_decisions())
// There is a transaction and it is *not* explicitly read-only,
// although so far there has only been SELECTs.
if (config.cache_in_trxs >= CACHE_IN_TRXS_ALL)
{
zReason = "ordinary transaction that has so far been read-only";
if (log_decisions())
{
zReason = "ordinary transaction that has so far been read-only";
}
action = CACHE_USE_AND_POPULATE;
}
else
{
ss_dassert((config.cache_in_trxs == CACHE_IN_TRXS_NEVER) ||
(config.cache_in_trxs == CACHE_IN_TRXS_READ_ONLY));
if (log_decisions())
{
zReason =
"populating but not using cache inside transaction that is not "
"explicitly read-only, but that has used only SELECTs sofar";
}
action = CACHE_POPULATE;
}
action = CACHE_USE_AND_POPULATE;
}
else
{
ss_dassert((config.cache_in_trxs == CACHE_IN_TRXS_NEVER) ||
(config.cache_in_trxs == CACHE_IN_TRXS_READ_ONLY));
if (log_decisions())
{
zReason =
"populating but not using cache inside transaction that is not "
"explicitly read-only, but that has used only SELECTs sofar";
zReason = "ordinary transaction with non-read statements";
}
action = CACHE_POPULATE;
}
if (action != CACHE_IGNORE)
{
if (is_select_statement(pPacket))
{
if (config.selects == CACHE_SELECTS_VERIFY_CACHEABLE)
{
// Note that the type mask must be obtained a new. A few lines
// above we only got the transaction state related type mask.
type_mask = qc_get_type_mask(pPacket);
if (qc_query_is_type(type_mask, QUERY_TYPE_USERVAR_READ))
{
action = CACHE_IGNORE;
zReason = "user variables are read";
}
else if (qc_query_is_type(type_mask, QUERY_TYPE_SYSVAR_READ))
{
action = CACHE_IGNORE;
zReason = "system variables are read";
}
else if (uses_non_cacheable_function(pPacket))
{
action = CACHE_IGNORE;
zReason = "uses non-cacheable function";
}
else if (uses_non_cacheable_variable(pPacket))
{
action = CACHE_IGNORE;
zReason = "uses non-cacheable variable";
}
}
}
else
{
// A bit broad, as e.g. SHOW will cause the read only state to be turned
// off. However, during normal use this will always be an UPDATE, INSERT
// or DELETE. Note that 'm_is_read_only' only affects transactions that
// are not explicitly read-only.
m_is_read_only = false;
action = CACHE_IGNORE;
zReason = "statement is not SELECT";
}
}
if (action == CACHE_USE_AND_POPULATE)
{
if (!m_use && !m_populate)
{
action = CACHE_IGNORE;
zReason = "usage and populating disabled";
}
else if (!m_use)
{
action = CACHE_POPULATE;
zReason = "usage disabled";
}
else if (!m_populate)
{
action = CACHE_USE;
zReason = "populating disabled";
}
}
else if (action == CACHE_USE)
{
if (!m_use)
{
action = CACHE_IGNORE;
zReason = "usage disabled";
}
}
else if (action == CACHE_POPULATE)
{
if (!m_populate)
{
action = CACHE_IGNORE;
zReason = "populating disabled";
}
}
if (log_decisions())
{
char* pSql;
int length;
const int max_length = 40;
// At this point we know it's a COM_QUERY and that the buffer is contiguous
modutil_extract_SQL(pPacket, &pSql, &length);
const char* zFormat;
if (length <= max_length)
{
zFormat = "%s, \"%.*s\", %s.";
}
else
{
zFormat = "%s, \"%.*s...\", %s.";
length = max_length - 3; // strlen("...");
}
const char* zDecision = (action == CACHE_IGNORE) ? "IGNORE" : "CONSULT";
ss_dassert(zReason);
MXS_NOTICE(zFormat, zDecision, length, pSql, zReason);
}
}
else
{
if (log_decisions())
{
zReason = "ordinary transaction with non-read statements";
MXS_NOTICE("IGNORE: Both 'use' and 'populate' are disabled.");
}
}
if (action != CACHE_IGNORE)
{
if (is_select_statement(pPacket))
{
if (config.selects == CACHE_SELECTS_VERIFY_CACHEABLE)
{
// Note that the type mask must be obtained a new. A few lines
// above we only got the transaction state related type mask.
type_mask = qc_get_type_mask(pPacket);
if (qc_query_is_type(type_mask, QUERY_TYPE_USERVAR_READ))
{
action = CACHE_IGNORE;
zReason = "user variables are read";
}
else if (qc_query_is_type(type_mask, QUERY_TYPE_SYSVAR_READ))
{
action = CACHE_IGNORE;
zReason = "system variables are read";
}
else if (uses_non_cacheable_function(pPacket))
{
action = CACHE_IGNORE;
zReason = "uses non-cacheable function";
}
else if (uses_non_cacheable_variable(pPacket))
{
action = CACHE_IGNORE;
zReason = "uses non-cacheable variable";
}
}
}
else
{
// A bit broad, as e.g. SHOW will cause the read only state to be turned
// off. However, during normal use this will always be an UPDATE, INSERT
// or DELETE. Note that 'm_is_read_only' only affects transactions that
// are not explicitly read-only.
m_is_read_only = false;
action = CACHE_IGNORE;
zReason = "statement is not SELECT";
}
}
if (!m_enabled && (action == CACHE_USE_AND_POPULATE))
{
action = CACHE_POPULATE;
zReason = "usage disabled";
}
if (log_decisions())
{
char* pSql;
int length;
const int max_length = 40;
// At this point we know it's a COM_QUERY and that the buffer is contiguous
modutil_extract_SQL(pPacket, &pSql, &length);
const char* zFormat;
if (length <= max_length)
{
zFormat = "%s, \"%.*s\", %s.";
}
else
{
zFormat = "%s, \"%.*s...\", %s.";
length = max_length - 3; // strlen("...");
}
const char* zDecision = (action == CACHE_IGNORE) ? "IGNORE" : "CONSULT";
ss_dassert(zReason);
MXS_NOTICE(zFormat, zDecision, length, pSql, zReason);
}
return action;
}
@ -1040,7 +1080,14 @@ CacheFilterSession::routing_action_t CacheFilterSession::route_SELECT(cache_acti
if (routing_action == ROUTING_CONTINUE)
{
m_state = CACHE_EXPECTING_RESPONSE;
if (m_populate || m_refreshing)
{
m_state = CACHE_EXPECTING_RESPONSE;
}
else
{
m_state = CACHE_IGNORING_RESPONSE;
}
}
else
{
@ -1053,10 +1100,8 @@ CacheFilterSession::routing_action_t CacheFilterSession::route_SELECT(cache_acti
dcb->func.write(dcb, pResponse);
}
}
else
else if (should_populate(cache_action))
{
ss_dassert(should_populate(cache_action));
// We will not use any value in the cache, but we will update
// the existing value.
if (log_decisions())
@ -1066,54 +1111,91 @@ CacheFilterSession::routing_action_t CacheFilterSession::route_SELECT(cache_acti
}
m_state = CACHE_EXPECTING_RESPONSE;
}
else
{
// We will not use any value in the cache and we will not
// update the existing value either.
if (log_decisions())
{
MXS_NOTICE("Fetching data from server, without storing to the cache.");
}
m_state = CACHE_IGNORING_RESPONSE;
}
return routing_action;
}
char* CacheFilterSession::handle_session_variable(const char* zName,
const char* pValue_begin,
const char* pValue_end)
namespace
{
bool get_truth_value(const char* begin, const char* end, bool* pValue)
{
bool rv = false;
static const char TRUE[] = "true";
static const char FALSE[] = "false";
static const size_t nTrue = sizeof(TRUE) - 1;
static const size_t nFalse = sizeof(FALSE) - 1;
char* message = NULL;
int len = (end - begin);
int len = (pValue_end - pValue_begin);
if (((len == nTrue) && (strncasecmp(pValue_begin, TRUE, nTrue) == 0)) ||
((len == 1) && (*pValue_begin == '1')))
if (((len == nTrue) && (strncasecmp(begin, TRUE, nTrue) == 0)) ||
((len == 1) && (*begin == '1')))
{
MXS_INFO("Caching enabled.");
m_enabled = true;
*pValue = true;
rv = true;
}
else if (((len == nFalse) && (strncasecmp(pValue_begin, FALSE, nFalse) == 0)) ||
((len == 1) && (*pValue_begin == '0')))
else if (((len == nFalse) && (strncasecmp(begin, FALSE, nFalse) == 0)) ||
((len == 1) && (*begin == '0')))
{
MXS_INFO("Caching disabled.");
m_enabled = false;
*pValue = false;
rv = true;
}
return rv;
}
}
char* CacheFilterSession::handle_session_variable(const char* zName,
const char* pValue_begin,
const char* pValue_end)
{
char* zMessage = NULL;
bool enabled;
if (get_truth_value(pValue_begin, pValue_end, &enabled))
{
if (strcmp(zName, SV_MAXSCALE_CACHE_POPULATE) == 0)
{
m_populate = enabled;
}
else
{
ss_dassert(strcmp(zName, SV_MAXSCALE_CACHE_USE) == 0);
m_use = enabled;
}
}
else
{
static const char FORMAT[] = "The variable %s can only have the values true/false/1/0";
int n = snprintf(NULL, 0, FORMAT, SV_MAXSCALE_CACHE_ENABLED) + 1;
int n = snprintf(NULL, 0, FORMAT, zName) + 1;
message = static_cast<char*>(MXS_MALLOC(n));
zMessage = static_cast<char*>(MXS_MALLOC(n));
if (message)
if (zMessage)
{
sprintf(message, FORMAT, SV_MAXSCALE_CACHE_ENABLED);
sprintf(zMessage, FORMAT, zName);
}
MXS_WARNING("Attempt to set the variable %s to the invalid value \"%.*s\".",
SV_MAXSCALE_CACHE_ENABLED, len, pValue_begin);
zName, n, pValue_begin);
}
return message;
return zMessage;
}
//static

4
server/modules/filter/cache/cachefiltersession.hh vendored
View File

@ -162,7 +162,7 @@ private:
char* m_zUseDb; /**< Pending default database. Needs server response. */
bool m_refreshing; /**< Whether the session is updating a stale cache entry. */
bool m_is_read_only;/**< Whether the current trx has been read-only in pratice. */
bool m_enabled; /**< Whether caching is enabled for this session. */
bool m_variable_added; /*<< Whether the variable was added or not. */
bool m_use; /**< Whether the cache should be used in this session. */
bool m_populate; /**< Whether the cache should be populated in this session. */
};

3
server/modules/filter/test/mock_session.cc
View File

@ -37,11 +37,14 @@ Session::Session(Client* pClient)
strcpy(m_mysql_session.db, "dummy");
pSession->variables = new SessionVarsByName;
m_client_dcb.data = &m_mysql_session;
}
Session::~Session()
{
delete variables;
}
Client& Session::client() const