This change adds a load property named strict_mode which is used to prohibit the incorrect data.
When it is set to false, the incorrect data will be loaded by NULL just like before.
When it is set to true, the incorrect data which belongs to a column without expr will be filtered.
The strict_mode is supported in broker load v2 now. It will be supported in stream load later.
For example, we start the process for the first time. The pid is 12345. Due to the accident, the process is killed and the fe.pid exists. Then we start the process for the second time. The pid is 6789. The fe.pid shows 67895 , Because file.write only cover the first four digits. This case can happen easily when we use supervise. Then I add the file.setLength(0) and delete the old data.
Before RowBlock will reserve memory for all columns in schema, even if
it is not queried. Which will cause bad performance when quering wide
table.
In this patch, RowBlock will reserve memory for needed columns. In a
case, this reduce ConvertBatchTime from 10s to 60ms when quering a wide
table who has 178 columns.
#1236
When I declared that the compilation mode was BUILD_TYPE=LSAN, there was a memory leak after running doris.
be.out:
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
#0 0x1089666 in __interceptor_malloc ../../../../libsanitizer/lsan/lsan_interceptors.cc:53
#1 0x7ff459547280 in __alloc_dir (/lib64/libc.so.6+0xc0280)
SUMMARY: LeakSanitizer: 32816 byte(s) leaked in 1 allocation(s).
* Fix the bug of += decimal in olap engine
[ISSUE-1225] This change fix the olap engine bug of decimal agg. Using ^ instead of * to judge result is less then zero.
The result of * will be less then zero when the result is overflow. So the answer of += is incorrect.
All FE logs are now with suffix yyyyMMdd or yyyyMMddHH, specified by 2 new
configurations: 'sys_log_roll_interval' and 'audit_log_roll_interval'
All FE logs roll at max size of 1024MB(default), specified by new
configuration: 'log_roll_size_mb'
By default, the new FE logs will look like this:
log/
fe.audit.log
fe.audit.log.20190524-1
fe.audit.log.20190523-2
fe.log
fe.log.20190524-1
fe.log.20190524-2
fe.log.20190523-3
fe.warn.log
fe.warn.log.20190524-1
fe.warn.log.20190523-2
fe.gc.log.20190524
Configurations 'sys_log_roll_mode' and 'audit_log_roll_mode' are deprecated.
The non-streaming hint of insert into will use the streamin plan which is same as the plan of stream insert.
It will also record the load info and return the label of insert stmt.
The partition is supportted in insert into stmt. The result which meet the target partitions will be loaded.
The introduction of example has been changed especially non-streaming insert.
Also, the param of partition_names is added in sql syntax which is used to declare the target partition_names in target table.
Change META_VERSION to 50
When creating new tablet by alter tablet task,
next_unique_id will increase on the base old tablet.
If next_unique_id is eqaul to zero, it will cause that
ColumnDataMessage not match with tablet meta.
1. get_json_xxx() now support using quoto to escape dot
2. Implement json_path_prepare() function to preprocess json_path
Performance of get_json_string() on 1000000 rows reduces from 2.27s to 0.27s
