Only flush tables to disk at the end of the binlog
Currently, when the avrorouter finishes reading a binlog file or when a certain number of rows or transactions is reached, it will flush all tables to disk. This is quite slow as events can easily be written faster into the binlog than they can be processed by avrorouter. A solution to this would be to only sync the tables (close the Avro block) instead of flushing them to disk. This would allow more efficient processing of the files while still retaining the safe shutdown that flushing offers.
This commit is contained in:
@ -1076,7 +1076,7 @@ void converter_func(void* data)
|
||||
/** We reached end of file, flush unwritten records to disk */
|
||||
if (router->task_delay == 1)
|
||||
{
|
||||
avro_flush_all_tables(router);
|
||||
avro_flush_all_tables(router, true);
|
||||
avro_save_conversion_state(router);
|
||||
}
|
||||
|
||||
|
Reference in New Issue
Block a user