Only flush tables to disk at the end of the binlog

Currently, when the avrorouter finishes reading a binlog file or when a
certain number of rows or transactions is reached, it will flush all
tables to disk. This is quite slow as events can easily be written faster
into the binlog than they can be processed by avrorouter.

A solution to this would be to only sync the tables (close the Avro block)
instead of flushing them to disk. This would allow more efficient
processing of the files while still retaining the safe shutdown that
flushing offers.
This commit is contained in:
Markus Mäkelä
2016-12-30 20:32:06 +02:00
parent 30a3475898
commit eaf3633728
3 changed files with 26 additions and 6 deletions

View File

@ -1076,7 +1076,7 @@ void converter_func(void* data)
/** We reached end of file, flush unwritten records to disk */
if (router->task_delay == 1)
{
avro_flush_all_tables(router);
avro_flush_all_tables(router, true);
avro_save_conversion_state(router);
}