Commit Graph

591 Commits

Author SHA1 Message Date
c7dfd1b0bd Cache: Introduce soft and hard TTL
- Hard TTL; the maximum time a value will be used from the cache.
- Soft TLL; the time after which the cache value should be updated
  from the server.

So as not to unnecessarily fetch the same value multiple times, when
the soft TTL has been reached, the value will be updated for the first
client, while all other clients will use the stale value until it has
become updated.

With different soft and hard TTLs there is a definite upper bound for
how old a value can be used.
2016-12-22 12:09:46 +02:00
97fcb94daa Cache: Always return staleness
In order for the LRU storage to correctly track the state in the
real storage, we always need to know stale-state of a returned
(or not returned) value.

The return type is now not an a pure enumeration, but a 16-bit
enumeration plus a 16-bit bitmask that can be used for conveying
more information.
2016-12-22 12:09:46 +02:00
2ea436a5c7 Cache: Test TTL behaviour 2016-12-22 12:09:46 +02:00
69acb8b9aa Cache: Fix review issues 2016-12-22 12:09:46 +02:00
e90f0d31a6 Cache: Pass argument as reference when it must be non-NULL 2016-12-22 12:09:46 +02:00
3f2c6d844b Cache: Use consistent naming in storage imlementations
- CamelCase for all C++ class and template names
- snake_case for all variables, including member variables and
  member functions
  - Static member functions start with a capital letter
- Member variables prefixed with m_
- Static member variables prefixed with s_
- Null terminated character string variables prefixed with z
- Pointers prefixed with p
- Smart-pointers prefixed with s
- Where a z, p, or s prefix is present, the following letter is
  capitalized, i.e., pEntry and not pentry.
2016-12-22 12:09:46 +02:00
3967a0b8c2 Cache: Provide Storage boiler-plate using template
All storage modules must implement the same interface and use the
same exception guards. Consequently that is conveniently provided
using a template.
2016-12-22 12:09:46 +02:00
ee422fd3ec Cache: Provide storage config as an object
Now all storage configuration values are provided in a single object.
That way it'll be easier to provide more if necessary and also makes
it straightforward to fetch the configuration, which makes it possible,
for instance, to adapt tests according to the way the storage has been
configured.
2016-12-22 12:09:46 +02:00
34b3f5c06e Cache: Invert test for capabilities 2016-12-22 12:09:46 +02:00
6da835ee44 MXS-1068: Maxrows filter doesn't count properly the result rows with size greater than16MBytes
16MBytes packets are now handled in MULTI result sets as well.
MAXROWS_DISCARDING_RESPONSE state has been removed.
Two new flags control large packets and result discarding
2016-12-21 10:06:59 +01:00
65b6ba6b1e added a comment explaining log format. 2016-12-20 13:04:01 +02:00
3e6cbeff19 1. Applied changes suggested from Markus.
2. tpmfilter now prints username of the client instead of the hostname
of the target server.
2016-12-20 13:04:01 +02:00
2866ef8dae added default values for options. 2016-12-20 13:04:01 +02:00
7776d55963 tpmfilter now prints which server the query has been executed on. 2016-12-20 13:04:01 +02:00
cb243f47a0 Remove usage of LUA_OK
LUA_OK is not defined in Lua 5.1.
2016-12-20 12:40:26 +02:00
036a6de0f1 Cache: Add more storage tests
- Check LRU behaviour
- Limit both by count and size
2016-12-20 10:06:54 +02:00
ec12786f49 Cache: Do not update head when accessing tail
Accessing the tail is only for debugging purposes, so it should not
cause the head item to change.
2016-12-20 10:01:29 +02:00
28f828b8a7 Fix luafilter build failure
Lua 5.1 doesn't return the type of the global variable so it has to be
ignored.
2016-12-19 18:40:14 +02:00
3a3632e75e Cache: Test LRU size limit 2016-12-19 11:46:56 +02:00
83c19c0b1a Cache: Factor out commonality of test[raw|lru]storage.cc 2016-12-19 11:46:56 +02:00
9998f60493 Cache: Specify item count explcitly in tests 2016-12-19 11:46:56 +02:00
43c694020c Cache: Use Tester facilities in testkeygeneration 2016-12-19 11:46:56 +02:00
23556cdfa2 Cache: Allow number of items + sizes to be specified in tests 2016-12-19 11:46:56 +02:00
6e6a3e4626 Replace gwbuf_clone_all with gwbuf_clone
gwbuf_clone cloned only the first buffer of a chain of buffers,
which never can be the desired outcome, while gwbuf_clone_all
cloned all buffers.

Now, gwbuf_clone behaves the way gwbuf_clone_all used to behave
and gwbuf_clone_all has been removed.
2016-12-19 11:00:47 +02:00
875766cf62 Make modutil_create_query parameters const
The parameters to modutil_create_query are not modified so they can be
declared const.
2016-12-19 10:33:55 +02:00
cfaf03de3c Avoid calling nil Lua functions
If a global Lua variable is not a function, it should not be called.

Also cleaned up parts of the code and fixed a minor error reporting bug.
2016-12-19 10:33:55 +02:00
be1b868938 Expose the query classifier through the luafilter
The luafilter exposes two of the main functions provided by the query
classifier API; the type and operation classification.

The functions can be used by the Lua script with minimal overhead as the
current query being executed is stored only as a pointer. The functions
should only be called inside the `routeQuery` entry point of a Lua script.
2016-12-19 10:33:55 +02:00
d457f9cd03 Cache: Add LRU storage test
Just initial tests; to be expanded.
2016-12-16 12:50:30 +02:00
7a7cfbab36 Cache: Refactor Tester hierarchy
Now that the cache key can be generated using the StorageFactory
there is no need for calling back into the derived class from Tester
to get hold of one. Instead the preparatory work is performed by
the abstract base classes, then the control is moved back to the
derived concrete class that decides what to actually do.
2016-12-16 12:42:45 +02:00
883b82a26a Cache: Use StorageFactory for creating the cache key
Now the preparatory work is done using the storage factory and the
storage instance is created only when the actual testing is performed.
2016-12-16 12:42:45 +02:00
9e89fe9246 Cache: Move key generation to module level
The key generation is dependent upon the storage implemenation,
but not on a particular storage instance.
2016-12-16 12:42:45 +02:00
297b8e1a44 Cache: Implement testrawstorage using TesterRawStorage 2016-12-16 12:42:45 +02:00
80ac69d667 Cache: Add TesterRawStorage class
A class dedicated for performing tests for raw storages.
2016-12-16 12:42:45 +02:00
eda84a1f96 Cache: Add TesterStorage class
A class dedicated for performing basic testing of Storages.
2016-12-16 12:42:45 +02:00
179762ff0a Cache: Add Tester class
A Tester class that simplifies the creation of various cache tests.
It provides the basic mechanisms for reading statements from MySQL/
MariaDB test files, for converting statements into equivalent
cache items (key + statement) and for running various tasks in
separate threads for a specific amount of time.
2016-12-16 12:42:45 +02:00
2751640a02 Adjust for RocksDB behaviour
RocksDB returns success when deleting a non-existing key. To deal
with that the book-keeping of LRUStorage is used and the real
underlying storage is used only if LRUStorage thinks a particular
key exists.
2016-12-16 12:42:45 +02:00
f59c6d67c3 Expand testrawstorage
Now also rudimentary tests the LRU mechanism, which at the same
time makes the name a misnomer. These will be split into separate
programs to allow tests to be run individually.
2016-12-15 10:17:39 +02:00
8019bf71e0 Honour max_size also with existing values
If an exiting value is updated and the new size would cause the
maximum size to be exceeded, we must evict items so that we stay
below the boundary.
2016-12-15 10:17:39 +02:00
8864753edf Rename things to make it clearer 2016-12-15 10:17:39 +02:00
c81394445b Cache: Plug some leaks 2016-12-15 10:17:39 +02:00
a8a5bafdb6 Fix pointer manipulation in LRUStorage
The head and tail pointers were not properly updated in
all circumstances.
2016-12-15 10:17:39 +02:00
9f08e1300f Make Cache/Storage const correct
- From the outside, all actions getting data from the cache/storage
  are const.
- Update documentation.
2016-12-15 10:17:39 +02:00
e5cd9ba47e Refactor testrawstorage slightly
Different storage tests will be added to the same file.
2016-12-15 10:17:39 +02:00
f9a7edc7d2 Fix tee filter
With the addition of filter capabilities, the tee filter should work with
all sorts of routers that require at most the RCAP_TYPE_CONTIGUOUS_INPUT
capability.

Due to a recent discovery of the server's capability to process multiple
requests, the filter can safely send data from one service to another
without waiting for the earlier replies.

This also fixes a minor problem with the cloning of DCBs where the backend
DCBs could end up in the wrong thread's pool.
2016-12-14 23:55:33 +02:00
5e2cbf3809 Cache: Provide access to head, tail, size, items
In order to be able to test the LRU mechanism properly you need
to be able to access the head and tail from the outside. The same
is regarding the size and items in the cache. In order to be able
to test that the guarantees are upheld, you need to be able to
access those values from the outside.
2016-12-13 12:37:28 +02:00
80c77413ba Cache: Do not use random indexes when testing
It would seem that the likelyhood of different threads accessing
the same items at the same time is greater if each thread continuosly
loops across all items from beginning to finish. That will also ensure
that head and tail surely are accessed. In addition, some function names
have been disambiguated.
2016-12-13 12:37:28 +02:00
75b2895ec2 Cache: If LRUStorage is used, create ST real storage
If the underlying storage does not support max_count and/or max_size
and it accordinly is decorated with LRUStorage, then create the real
storage as single-threaded. Since LRUStorage will do locking it is of
no use to do locking in the real storage as well.
2016-12-13 12:37:28 +02:00
651e1122c2 Fix minor issues in testrawstorage
- Release memory
- Set libdir
- Adjust output
2016-12-13 12:37:28 +02:00
f71be685b6 Cache: Test raw storage
Here we create a number of threads and then randomly start getting
putting and deleting values. The intent is to test that the locking
behaviour of the storage modules is correctly implemented.
2016-12-13 12:37:28 +02:00
3cbfd3f1d0 Cache: Create MT storage if that is asked for 2016-12-13 12:37:28 +02:00