This change adds a load property named strict_mode which is used to prohibit the incorrect data.
When it is set to false, the incorrect data will be loaded by NULL just like before.
When it is set to true, the incorrect data which belongs to a column without expr will be filtered.
The strict_mode is supported in broker load v2 now. It will be supported in stream load later.
Before RowBlock will reserve memory for all columns in schema, even if
it is not queried. Which will cause bad performance when quering wide
table.
In this patch, RowBlock will reserve memory for needed columns. In a
case, this reduce ConvertBatchTime from 10s to 60ms when quering a wide
table who has 178 columns.
#1236
When I declared that the compilation mode was BUILD_TYPE=LSAN, there was a memory leak after running doris.
be.out:
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
#0 0x1089666 in __interceptor_malloc ../../../../libsanitizer/lsan/lsan_interceptors.cc:53
#1 0x7ff459547280 in __alloc_dir (/lib64/libc.so.6+0xc0280)
SUMMARY: LeakSanitizer: 32816 byte(s) leaked in 1 allocation(s).
* Fix the bug of += decimal in olap engine
[ISSUE-1225] This change fix the olap engine bug of decimal agg. Using ^ instead of * to judge result is less then zero.
The result of * will be less then zero when the result is overflow. So the answer of += is incorrect.
The non-streaming hint of insert into will use the streamin plan which is same as the plan of stream insert.
It will also record the load info and return the label of insert stmt.
The partition is supportted in insert into stmt. The result which meet the target partitions will be loaded.
The introduction of example has been changed especially non-streaming insert.
Also, the param of partition_names is added in sql syntax which is used to declare the target partition_names in target table.
Change META_VERSION to 50
When creating new tablet by alter tablet task,
next_unique_id will increase on the base old tablet.
If next_unique_id is eqaul to zero, it will cause that
ColumnDataMessage not match with tablet meta.
1. get_json_xxx() now support using quoto to escape dot
2. Implement json_path_prepare() function to preprocess json_path
Performance of get_json_string() on 1000000 rows reduces from 2.27s to 0.27s
A bad tablet is always be chosen to do compaction, and failed again
and again, which may block compaction of other tablets.
Add a BE config 'min_compaction_failure_interval_ms' to avoid choosing
bad tablet again at a certain interval, so that other tablets have
chance to do the compaction.
Also fix a bug that using avg() function on varchar column return
unexpected exception.
* Change label of broker load txn
1. put broker load label into txn label
2. fix the bug of `label is already used`
3. fix partition error of new broker load
* Fix count error in mini load and broker load
There are three params (num_rows_load_total, num_rows_load_filtered, num_rows_load_unselected) which are used to count dpp.norm.ALL and dpp.abnorm.ALL.
num_rows_load_total is the number rows of source file.
num_rows_load_unselected is the not satisfied (where conjuncts) rows of num_rows_load_total
num_rows_load_filtered is the rows (quality not good enough) of (num_rows_load_total-num_rows_load_unselected)
This change include the show load of loadv2 and some bug fix of loadv2.
Firstly, the show load will perform both load and loadv2 info. According to loadv2, the ETL progress of loadv2 is N/A during the period of loading.
Secondly, the loadv2 will be created when version of property is v2.
This is a temporary property which will not influence the old broker load.
After the loadv2 is finished, the default load will be changed to loadv2.
Finally, there are some bug in LoadingTaskPlanner fixed by this change.
* Enhence the usabilities
1. Add metrics to monitor transactions and steaming load process in BE.
2. Modify BE config 'result_buffer_cancelled_interval_time' to 300s.
3. Modify FE config 'enable_metric_calculator' to true.
4. Add more log for tracing broker load process.
5. Modify the query report process, to cancel query immediately if some instance failed.
* Fix bugs
1. Avoid NullPointer when enabling colocation join with broker load
2. Return immediately when pull load task coordinator execution failed
1. max io util of disks
2. max network send/receive bytes rate of all network devices
3. base/cumulative compaction request counter and failure counter
