Commit Graph

203 Commits

Author SHA1 Message Date
c56b2063aa MXS-1881: Rename event processing related objects
The main objects involved in replication event processing now use camel
case.
2018-06-08 12:18:11 +03:00
69855f9214 MXS-1881: Refactor AVRO_TABLE
Moved initialization into the constructor and removed unused member
variables.
2018-06-08 12:18:11 +03:00
f61c56228c MXS-1881: Refactor TABLE_MAP initialization
Changed TABLE_MAP to use STL containers and types. The initialization is
now done in the constructor. Removed unnecessary linkage between TABLE_MAP
and TABLE_CREATE.
2018-06-08 12:18:11 +03:00
e35d9dfc10 Move client functionality into AvroSession
Most of the code is unchanged. Some of the code that relied on modifiable
char pointers was modified to use std::string.
2018-06-08 12:18:11 +03:00
4a3216d483 Merge branch '2.2' into develop 2018-06-04 16:00:19 +03:00
c88aa11e11 Copy FDE events in avrorouter
Commit 67386980e327ad063b24cb55971cf44f4930e241 caused the actual events
to be ignored. This meant that the larger event size was assumed for all
events. In most cases this works but it is not the correct way to do it.
2018-06-04 15:17:48 +03:00
be098538e5 MXS-1881: Read event header in a separate function
Reading the binlog event header in a separate function makes it easier to
comprehend. Cleaned up some unused variables and code that would is not
used.
2018-05-28 10:32:18 +03:00
14a3b0052b MXS-1881: Perform task management in main worker
The creation and cancellation of delayed tasks must be done on the worker
where the task is executed.
2018-05-28 10:32:18 +03:00
ec919c367b Move Avro creation into Avro::create
The Avro instance is now created inside a static class method. This brings
it in line with how other modules create instances.

Converted all strings to std::string and updated their usage.
2018-05-28 10:32:17 +03:00
232d86e67a Make active table map array dynamic
By storing the table maps in a std::unordered_map, the storage of mapped
tables is made dynamic. This also makes the management of mapped tables a
lot more robust.
2018-05-28 10:32:17 +03:00
f8ceb875a0 Refactor TABLE_CREATE allocation
Using std::string for names removes the need to handle memory
allocation. Moving the column attributes into a class of its own greatly
simplifies the creation of the TABLE_CREATE as well as modifications that
are done to it.
2018-05-28 10:32:17 +03:00
d5760f4301 Move AvroSession initialization inside the class
The AvroSession now has the AvroSession::create method for creation new
sessions. The destructor handles the freeing of all allocated resources.
2018-05-28 10:32:16 +03:00
fb53bbf4da Use STL containers in avrorouter
The HASHTABLE can be replaced with std::unordered_map. This simplifies the
management by making the deletion of old objects automatic.

More cleanup and refactoring is needed to make the contained classes
cleaner.
2018-05-28 10:32:16 +03:00
a6eef98597 Initialize all AvroSession variables
This is done to get tests to pass before further refactoring is done.
2018-05-28 10:32:16 +03:00
6159f863ce Remove unnecessary locking
The lock didn't really protect anything but since it was not explicitly
initialized, it caused a hang.
2018-05-28 10:32:16 +03:00
3fce61a615 Do avro conversion in main thread
The avrorouter no longer uses the housekeeper for the conversion
task. This prevents the deadlock which could occur when clients were
notified at the same time that the binlogrouter was adding a master
reconnection task.
2018-05-28 10:32:16 +03:00
16d2ff9564 Set read and execute permissions for all paths
If a path requires read or execute permission, it is granted to all. This
keeps path parameters in line with other directories that MaxScale
creates.
2018-05-28 10:32:16 +03:00
6898397d49 Clean up avrorouter.hh
Cleaned up the header and removed most of unused member variables.
2018-05-28 10:32:15 +03:00
8dd89548f4 Fix avrorouter client notification
The avrorouter client notification now uses dcb_foreach.
2018-05-28 10:32:15 +03:00
3095b18b14 Convert avrorouter objects to C++
The instance and session objects are now C++ structs. The next pointers
for the sessions was removed as it is not the appropriate place to store
this information. This means that the client notification functionality is
broken in this commit.
2018-05-28 10:32:15 +03:00
354e9aeefc Clean up avrorouter files
Removed commented out code, manual revision notes and extraneous comments.
2018-05-28 10:32:15 +03:00
1f6cc6db8a MXS-1840 Compile all routers as C++
Minimal changes, only what is needed in order to make it compile.
2018-05-07 14:06:22 +03:00
e01cfe0fba Remove avrorouter deadlock
The avrorouter would attempt to enable the conversion task from within the
conversion task itself. This caused a deadlock which led to test failures.
2018-04-27 12:00:38 +03:00
c70216390f MXS-1506: Combine housekeeper task types
The tasks themselves now control whether they are executed again. To
compare it to the old system, oneshot tasks now return `false` and
repeating tasks return `true`.

Letting the housekeeper remove the tasks makes the code simpler and
removes the possibility of the task being removed while it is being
executed. It does introduce a deadlock possibility if a housekeeper
function is called inside a housekeeper task.
2018-04-10 15:29:30 +03:00
96a0aae7fe MXS-1506: Move all functionality into Housekeeper
The class now does all of the work and the API wraps the calls to the
member methods. Using an STL container makes the list management a lot
more convenient.
2018-04-10 15:29:30 +03:00
8bfb4f231a Merge branch '2.1' into 2.2 2018-03-06 10:38:01 +02:00
93f3529bb4 MXS-1697: Treat missing libraries as an error
If the avrorouter is being build and the required libraries are not found,
the configuration process should fail. Adding the command to bypass this
into the error message should make it easier to disable this part if it is
not needed.
2018-03-05 13:58:09 +02:00
e1f1d8e58a Merge branch '2.1' into 2.2 2018-02-02 16:05:14 +02:00
7093a5bdf8 Fix CREATE TABLE tokenization
The token skipping function did not check for a period or an opening
parenthesis when parsing the test. Also fixed a debug assertion when only
NULL values were inserted.
2018-02-02 12:24:26 +02:00
3dfb972d87 Merge branch '2.1' into 2.2 2018-01-30 16:28:11 +02:00
6dcc71d862 MXS-1621: Fix minor bugs caused by previous changes
Used the correct value in table_create_alloc and remove unused
parameter. Use the pre-calculated end pointer when looking for events.
Always use the column count of the TABLE_MAP event as all mismatches are
detected earlier.
2018-01-30 15:59:06 +02:00
93923acafb MXS-1621: Skip unneeded ALTER TABLE operations
Some ALTER TABLE operations (e.g. ADD INDEX) are not useful to the
avrorouter. These need to be detected and skipped.
2018-01-30 15:59:05 +02:00
e14710ab2b Fix ALTER TABLE detection regex
The regular expression expected that the COLUMN keyword was always
present.
2018-01-30 15:59:05 +02:00
df86ee3579 Fix buffer overflow assertions
The buffer overflow assertions were off by one: The data pointer can be
equal to the end pointer when the last column of the row is processed.
2018-01-30 15:59:05 +02:00
9146a215f7 Fix DDL table identifier parsing
The parsing was inadequate and did not support all forms of quoted
identifiers.
2018-01-30 15:59:05 +02:00
b7e475f316 MXS-1621: Detect TABLE_MAP ↔ TABLE_CREATE column count mismatch
If the TABLE_MAP and TABLE_CREATE have different column counts, an error
is logged and the row events are skipped.
2018-01-30 15:59:05 +02:00
c000b3186c MXS-1575: Fix optional COLUMN keywork handling
The COLUMN keyword is optional and cannot be assumed to exist.
2018-01-30 15:59:05 +02:00
8dfb1d0113 MXS-1621: Add ALTER TABLE ... [FIRST | AFTER col ] parsing
The parser checks whether the FIRST or AFTER keywords are used and, if
AFTER is used, extracts the relevant column name.

Added a test case that checks that the parsing works and detects the
correct column names.
2018-01-30 15:59:05 +02:00
6410b4f19a MXS-1633 Turn off collecting of sqlite3 memstats
According to customer reports collecting the statistics has a significant
impact on the performance. As we don't need that information we can just
as well turn off that.

Further, since maxscale-common now links to the sqlite3-library, no
module needs to do that explicitly.
2018-01-30 13:58:37 +02:00
0afe10ffb9 Merge branch '2.1' into 2.2 2018-01-16 09:47:49 +02:00
9ca6d586d1 MXS-1575: Fix Avro schema versioning
The versions were not updated if the table was dropped and created again.
2018-01-15 16:15:20 +02:00
dbad6b737d Detect redundant table map events
When two identical tables are mapped into the same slot, the newer one is
ignored.
2018-01-15 16:15:20 +02:00
5e4b7ae7c7 Fix memory leak in avrorouter
The TABLE_MAP freeing function leaked memory.
2018-01-15 16:15:20 +02:00
f2771d5ad8 Remove obsolete message
The warning that a schema already exists is obsolete as mapped tables are
now always opened instead of being reused. This causes the schema checks
to be done for each mapped table.
2018-01-15 16:15:19 +02:00
3f78dbc923 Improve avrorouter assertion output
When an assertion fails due to an overflow of the event buffer, all
processed values for that event are dumped.

This commit also enables the assertions even for non-debug builds which
should speed up the elimination process for bugs in the avrorouter. The
overhead of doing this is minimal as the output is already gathered for
the INFO level logging.
2018-01-05 13:39:57 +02:00
ce1c45828a MXS-1575: Fix CREATE TABLE processing
The CREATE TABLE processing failed to identify the explicit database names
that were generated by mysqldump.
2018-01-05 12:50:39 +02:00
95ad3aa7df MXS-1543: Log a warning if STATEMENT or MIXED is used
If avrorouter suspects that statement based or mixed replication formats
are being used, it logs a warning.
2018-01-05 12:50:39 +02:00
7d8905d91f Merge branch '2.1' into 2.2 2018-01-03 14:05:31 +02:00
f5484a1011 MXS-1576: Add purge command to avrorouter
The `purge` command can be used to reset the conversion
process. Currently, executing the `purge` module command and restarting
MaxScale is the only correct way to reset the conversion process.
2018-01-03 09:44:41 +02:00
a86b81824f MXS-1575: Add extra debug assertions
Added more debug assertions to JSON serialization of the Avro schema. Also
checked that the column count for the TABLE_CREATE object matches that of
the TABLE_MAP object.
2018-01-03 09:44:41 +02:00