There's now double bookkeeping:
- All delayed calls are in a map whose key is the next
invocation time. Since it's a map (and not an unordered_map)
it's sorted just the way we want to have it.
- In addition, there's an unordered set for each tag.
With this arrangement we can easily invoke the delayed calls
in the right order and be able to efficiently remove all
delayed calls related to a particular tag.
When canceling, a DelayedCall instance must be removed from the
collection holding all delayed calls. Consequently priority_queue
cannot be used as it 1) does not provide access to the underlying
collection and 2) the underlying collection (vector or deque)
is a bad choise if items in the middle needs to be removed.
The interface for canceling calls is now geared towards the needs
of sessions. Basically the idea is as follows:
class MyFilterSession : public maxscale::FilterSession
{
...
int MyFilterSession::routeQuery(GWBUF* pPacket)
{
...
if (needs_to_be_delayed())
{
Worker* pWorker = Worker::current();
void* pTag = this;
pWorker->delayed_call(5000, pTag, this,
&MyFilterSession::delayed_routeQuery,
pPacket);
return 1;
}
...
}
bool MyFilterSession::delayed_routeQuery(Worker::Call:action_t action,
GWBUF* pPacket)
{
if (action == Worker::Call::EXECUTE)
{
routeQuery(pPacket);
}
else
{
ss_dassert(action == Worker::Call::CANCEL);
gwbuf_free(pPacket);
}
return false;
}
~MyFilterSession()
{
void* pTag = this;
Worker::current()->cancel_delayed_calls(pTag);
}
}
The alternative, returning some key that the caller must keep
around seems more cumbersome for the general case.
It's now possible to provide Worker with a function to call
at a later time. It's possible to provide a function or a
member function (with the object), taking zero or one argument
of any kind. The argument must be copyable.
There's currently no way to cancel a call, which must be added
as typically the delayed calling is associated with a session
and if the session is closed before the delayed call is made,
bad things are likely to happen.
Worker::Timer class and Worker::DelegatingTimer templates are
timers built on top of timerfd_create(2). As such they consume
descriptor and hence cannot be created independently for each
timer need.
Each Worker has now a private timer member variable on top of
which a general timer mechanism will be provided.
The maximum number of workers and routing workers are now
hardwired to 128 and 100, respectively. It is still so that
all workers must be created at startup and destroyed at
shutdown, creating/destorying workers at runtime is not
possible.
The documentation stated that all CPUs would be used when threads=auto was
used. In reality the behavior was the same as was with 2.0 (number of CPUs
minus one).
With the changes to the DCB handling, the service pointer of a client DCB
must always be assigned.
Also removed the unnecessary parentheses around the comparison.
The state of each individual listener is now displayed in the REST
API. Created common functions for printing the listener state and took
them into use. Added the new state into MaxCtrl output.
Worker is now the base class of all workers. It has a message
queue and can be run in a thread of its own, or in the calling
thread. Worker can not be used as such, but a concrete worker
class must be derived from it. Currently there is only one
concrete class RoutingWorker.
There is some overlapping in functionality between Worker and
RoutingWorker, as there is e.g. a need for broadcasting a
message to all routing workers, but not to other workers.
Currently other workers can not be created as the array for
holding the pointers to the workers is exactly as large as
there will be RoutingWorkers. That will be changed so that
the maximum number of threads is hardwired to some ridiculous
value such as 128. That's the first step in the path towards
a situation where the number of worker threads can be changed
at runtime.
A new class mxs::Worker will be introduced and mxs::RoutingWorker
will be inherited from that. mxs::Worker will basically only be a
thread with a message-loop.
Once available, all current non-worker threads (but the one
implicitly created by microhttpd) can be creating by inheriting
from that; in practice that means the housekeeping thread, all
monitor threads and possibly the logging thread.
The benefit of this arrangement is that there then will be a general
mechanism for cross thread communication without having to use any
shared data structures.
If maxadmin connections are handled by different workers, then
there may be a deadlock if some maxadmin command requires
communication with all workers.
Namely, in that case a message will be sent to all other workers
but the current one, but that message will not be handled if that
other worker at that point sits in the debugcmd_lock spinlock
in debugcmd.c:execute_cmd().
We can prevent that deadlock from happening simply by ensuring
that all maxadmin connections are handled by one thread.
The worker task should never be immediately executed to allow the task to
be executed on the next "tick" of the worker. This prevents recursive
calls to e.g. routeQuery in readwritesplit when errors are handled.
Readwritesplit would not handle multiple overlapping COM_STMT_EXECUTE
commands properly if they opened cursors. This was due to the fact that
the result would not be marked as complete and COM_STMT_FETCH commands
were executed as if they did not return results.
The correct implementation is to consider a COM_STMT_EXECUTE that opens a
cursor complete only when the first EOF packet is read (that is, when the
resultset header is read). This allows subsequent COM_STMT_FETCH commands
to be handled separately.
The separate COM_STMT_FETCH handling must count the number of packets that
are being fetched. This allows correct tracking of the state of a
COM_STMT_FETCH by checking that the number of packets is correct or the
second EOF/ERR packet is read.
Now that the readwritesplit uses the same mechanism for both
retry_failed_reads and delayed query retries, the re-routing function
should accept a delay of 0 seconds. This makes the mechanism more suitable
for other uses e.g. delaying of queries in filters.
The `error` variable was never used. Also added a more convenient typedef
for both the downstream and upstream functions and updated filter API
version.
The tasks themselves now control whether they are executed again. To
compare it to the old system, oneshot tasks now return `false` and
repeating tasks return `true`.
Letting the housekeeper remove the tasks makes the code simpler and
removes the possibility of the task being removed while it is being
executed. It does introduce a deadlock possibility if a housekeeper
function is called inside a housekeeper task.
The class now does all of the work and the API wraps the calls to the
member methods. Using an STL container makes the list management a lot
more convenient.
The old hkheartbeat variable was changed to the mxs_clock() function that
simply wraps an atomic load of the variable. This allows it to be
correctly read by MaxScale as well as opening up the possibility of
converting the value load to a relaxed memory order read.
Renamed the header and associated macros. Removed inclusion of the
heartbeat header from the housekeeper header and added it to the files
that were missing it.
This is a proof-of-concept that validates the query retrying method. The
actual implementation of the query retrying mechanism needs more thought
as using the housekeeper is not very efficient.
