Commit Graph

511 Commits

Author SHA1 Message Date
9175295542 Cache: Now also CacheRules is shared
Unnecessary methods were also removed from CachePT and CacheMT
as it does not make sense to create more than one single instance
of those per filter instance. Consequently there is no need for
them to be able to use an existing StorageFactory (and CacheRules).
2016-11-30 10:22:53 +02:00
93103fd64a Cache: shared_ptr used with CacheRules and StorageFactory
shared_ptr is now used for managing the lifetime of CacheRules
and StorageFactory instances.
2016-11-30 10:22:53 +02:00
d52145054f Cache: Now use CacheRules instead of CACHE_RULES 2016-11-30 10:22:53 +02:00
2e4ac55aa4 cache: Add CacheRules class
Easier to use in a C++ context than the C API
2016-11-30 10:22:53 +02:00
2a2c530857 Cache: Trivial cleanup of inmemory storage 2016-11-30 10:22:53 +02:00
2423031df8 Cache: Use lock guard for locking spinlocks
Now the locks will always be release, also in the presence of an
unantecipated exception.
2016-11-30 10:22:53 +02:00
efa8713b17 Cache: Include inmemory storage in build
Add logging note to storage_rocksdb at the same time.
2016-11-30 10:22:53 +02:00
23bb7c4de7 Cache: Add in memory storage
Storage implementation that simply uses std::unordered_map as
storage structure.
2016-11-30 10:22:53 +02:00
e5a9eceff4 Cache: Fix prev/next twiddling 2016-11-30 10:22:53 +02:00
9c1b9c188e Cache: Enable LRU eviction
The maximum count and maximum size of the cache can now be
specified and a storage can declare what capabilities it has.
If a storage modile cannot enforce the maximum count or maximum
size limits, the storage is decorated with an LRU storage that
can.
2016-11-30 10:22:53 +02:00
e597523c47 Cache: Allow storage modules to specify capabilites
Will be used to decide whether a LRUStorage facade is needed in front
of the actual storage or not.
2016-11-30 10:22:53 +02:00
1701979696 Cache: Add LRU storage
The LRUStorage hierarchy implements the decorator pattern and
is capable of providing LRU behaviour to any underlying storage.
There are two concrete classes - LRUStorageST and LRUStorageMT -
that can be used in single thread and multiple thread contentx,
respectively.

Also tests the convention of placing _ as suffix on member
variables.
2016-11-30 10:22:53 +02:00
ecdeb009b3 Fix dbfwfilter regression
The query throttling functionality was broken by the move to the per
thread rule system. One way to fix this is to move the session specific
throttling information into the client session itself. This allows for a
simpler system with no direct dependencies on the rules or users
themselves.
2016-11-29 14:19:08 +02:00
fa58fdcde8 Cache: Make Storage abstract
Storage is now abstract with StorageReal (that uses an actual
storage module) derived from it. This to prepare for a stackable
case where LRU behaviour is implemented in front of the actual
physcal storage.
2016-11-29 11:23:23 +02:00
b4ddf23ba3 Cache: Now uses unordered_map instead of HASHTABLE 2016-11-29 11:23:23 +02:00
02265c5dc2 Cache: All functions now named using snake case
Exception is where the call goes across a plugin boundary, as
is the current MaxScale convention.
2016-11-29 11:23:23 +02:00
a2daacd465 Cache: Add options for choose sharing mode
Now possible to use same cache for all threads or separate cache
for each.
2016-11-29 11:23:23 +02:00
933e09d245 Cache: Add per thread Cache
The per thread cache is implemented so that there is one single
thread cache per thread.

This uses shared_ptr, which is available in different places
depending on which GCC version is default on the target platform.
So, before this is merged to develop, that needs to be sorted out.
2016-11-29 11:23:23 +02:00
910a349f86 Cache: Store name as string and not as char* 2016-11-29 11:23:23 +02:00
4f1f71a05a Cache: Make config and storage factory sharable
In antecipation of a CachePT (cache per thread), rearrange things
so that the storage factory and config can be shared between cache
instances.
2016-11-29 11:23:23 +02:00
c3319b24a6 Cache: Wrongly protected methods made public 2016-11-29 11:23:23 +02:00
124453b756 Cache: Add simple cache
A simple cache is one that does not transparently use multiple
storages in the background. That will be the case when a separate
cache per thread is used. Both CacheST and CacheMT are now derived
from CacheSimple.

A fair amount of what used to be in Cache has now been moved to
CacheSimple. What remains is what surely is common for all cache
types.
2016-11-29 11:23:23 +02:00
5bae5b8b8d Cache: Add cache for single thread 2016-11-29 11:23:23 +02:00
279bf7e2fa Cache: Change key from char ptr to struct 2016-11-29 11:23:23 +02:00
c1c447985d Cache: Make it possible to specify thread model for storage 2016-11-29 11:23:23 +02:00
7a0d6307fe Cache: Make Cache abstract
The methods that need to be different depending on the thread model
are now pure virtual.
2016-11-29 11:23:23 +02:00
ff3173d588 Cache: Remove obsolete method 2016-11-25 13:44:41 +02:00
68f70ee5b2 Cache: Prepare for ST, MT, and PT caches
With a small cost it is possible to prepare for a single-thread,
multi-thread and cache-per-thread specific cases.
2016-11-25 13:03:48 +02:00
a71d1f0877 Cache: Param processing moved to cachefilter.cc
This in antecipation of different Cache instances being created
depending on passed arguments.
2016-11-25 11:08:36 +02:00
f8f0ef8d34 Cache: Ensure no C++ exceptions can escape 2016-11-25 09:32:35 +02:00
14c8a5bb5f Cache: Take Cache class into use
Now the basic structure is in place:

- cachefilter.cc is the MaxScale filter interface.
- Cache is the actual cache class that will also handle LRU issues.
- SessionCache (sessioncache.cc) is the session specific cache class
  that using Cache acts as the cache for a particular session.
  If an item is stale, then one SessionCache will update it.
- StorageFactory is the component that is capable of loading a module
  providing storage facilities.
- Storage is the actual key/value store.
2016-11-25 09:32:35 +02:00
719cffd596 Cache: Add Cache class.
This class provides the cache function used by the session caches.

Not taken into use yet.
2016-11-25 09:32:35 +02:00
5a6c4b5970 Cache: Take SessionCache into use 2016-11-25 09:32:35 +02:00
b5022e1deb Add SessionCache
SessionCache abstracts the cache for a specific session. It then
uses the actual Cache instance for storing data.

It is not used yet.
2016-11-25 09:32:35 +02:00
10deb9f07b Cache: Introduce StorageFactory and Storage
The storage module is abstracted with StorageFactory that is capable
of creating Storage instances. The latter contains the data and
provides the behaviour for using the actual storage implementation,
which sits behing a C API, conveniently.
2016-11-25 09:32:35 +02:00
762da5f2df Cache: Make everything C++ and rename some files 2016-11-25 09:32:35 +02:00
ff405a4f5f Cache compiled as C++
Cache compiled as C++ in order to allow the use of standard
C++ data structures.
2016-11-25 09:32:35 +02:00
dd63253261 Fix internal test suite failures
The server test used the wrong name.

MySQL users test loaded multiple modules in one function call and wasn't
appropriate for an internal test suite test as it requires a working
installation.

The cache filter didn't set the library paths before trying to load
modules.

The binlogrouter was missing a NULL check which caused a crash.
2016-11-24 15:22:55 +02:00
265aacaf15 GWBUF_DATA(...) explicitly returns uint8_t* 2016-11-24 13:31:29 +02:00
9d3e5a715b Cache: Remove item if putting fails
If an item cannot be put to the database, it is explicitly removed
to ensure that we cannot have the situation that a stale item is
continuously returned because the updating of the value fails for
whatever reason.
2016-11-23 12:15:58 +02:00
e9029b183e Cache: Allow entries to be explicitly deleted 2016-11-23 12:15:58 +02:00
2ecd5f3340 Prevent stale item from being fetched more than once
If a cached entry becomes stale, then if no extra measures are
taken then every thread hitting that item while it is being fetched
from the server will also refresh it, even though it obviously is
sufficient that one does it.

Now the knowledge that the item is being refreshed is recorded,
so that all other clients are simply returned the stale item. That
way, we'll hit the server once per stale item.
2016-11-23 12:15:58 +02:00
53413a3260 MXS-929: Add reloading of rules to dbfwfilter
The rules and users are stored in thread-local pointers which removes the
need to hold global locks. This allows greater scalability but causes a
slightly larger memory footprint. Usually the increase in memory
consumption is trivial compared to the benefits in scaling.

Using a per-thread rule and user list allows changes to be applied
immediately without locking on the instance level.

The updating of the rules uses the new functinality described in the
debugcmd.h header. The module registers two functions at startup; one for
reloading rules and one for printing them. These custom functions can be
seen in the `maxadmin list functions` output.
2016-11-23 11:14:31 +02:00
79a03049a1 Enable fetching of stale items
It is now possible to specify that a cache item should be returned
if it exists, even if it is stale.
2016-11-22 11:15:26 +02:00
97b039689f Disable write-ahead log
Since it's a cache, we do not need to retain the data from one
MaxScale invocation to the next. Consequently, we can turn off
write ahead logging completely if we simply wipe the RocksDB
database at each startup. In addition, the location of the
cache directory can now be specified explicitly, so it can be
placed, for instance, on a RAM disk.
2016-11-22 08:44:00 +02:00
a018e732eb Make the dbfwfilter rule check a separate utility
The code for the utility is now stored in a separate file. This also
removes the need to include testing headers from other directories.

Also added a function to reload rules that uses the newly modified rule
parsing mechanism. This can be used later on to update the rules at
runtime.
2016-11-18 08:59:52 +02:00
8ae76e3ced Clean up dbfwfilter
Removed unused functions, made function names more consistent.
2016-11-18 08:53:43 +02:00
042491ac07 Rename dbfwfilter objects to more descriptive ones
The object names in the dbfwfilter weren't very descriptive and were not
that easy to comprehend. The same goes for function names which had
somewhat cryptic names.
2016-11-17 12:36:44 +02:00
c4999232ce MXS-935: Implement column matching
With the advent of qc_get_field_info, columns can now be matched.
However, there is still some undeterminism caused by the table
information not containing contextual information (exactly where
is the table used).

Further, suppose table X contains the column A and table Y contains
the column B, then given a statement like

    SELECT a, b from X, Z;

we cannot know whether a is in X or Z, or b in X or Z, without being
aware of the schema, which we currently are not.

Consequently, as long as MaxScale is not aware of the schema, some
heuristics must be applied. For instance, if exactly one table is
referred to, then we can assume that columns that are not explicitly
qualified are from that table.

The rule tests are currently rather rudimentary and need to be
expanded.
2016-11-16 10:44:56 +02:00
2334c663e0 Merge branch 'develop' into MXS-936 2016-11-10 15:42:09 +01:00