Commit Graph

648 Commits

Author SHA1 Message Date
bedd94dca2 Upgrade brpc to 0.9.5 (#1243)
Change some ut
2019-06-04 11:13:23 +08:00
ae75e44e05 fixup leak memory (#1244)
When I declared that the compilation mode was BUILD_TYPE=LSAN, there was a memory leak after running doris.

be.out:
Direct leak of 32816 byte(s) in 1 object(s) allocated from:
    #0 0x1089666 in __interceptor_malloc ../../../../libsanitizer/lsan/lsan_interceptors.cc:53
    #1 0x7ff459547280 in __alloc_dir (/lib64/libc.so.6+0xc0280)

SUMMARY: LeakSanitizer: 32816 byte(s) leaked in 1 allocation(s).
2019-06-04 11:07:37 +08:00
6231fe0abc Fix FragmentMgrTest crash sometimes (#1232) 2019-06-01 18:10:24 +08:00
741539de91 Release udf headers & lib (#1231)
remove internal headers from udf.h
release udf headers & lib
2019-05-31 17:47:41 +08:00
7cdaba66dc Add spatial func (#1213)
Support some spatial functions, such as ST_Contains.
2019-05-31 14:23:09 +08:00
c20d62679e Add negative load from StreamLoad (#1227) 2019-05-31 07:14:06 +08:00
dc0cd5fd67 Fix the bug of += decimal in olap engine (#1226)
* Fix the bug of += decimal in olap engine
[ISSUE-1225] This change fix the olap engine bug of decimal agg. Using ^ instead of * to judge result is less then zero.
The result of * will be less then zero when the result is overflow. So the answer of += is incorrect.
2019-05-31 07:12:22 +08:00
309b779a7d Check colocate table name should be case-sensitive (#1224) 2019-05-30 22:47:22 +08:00
180d8e5cbd Modify some thirdparties (#1228)
1. Change Kafka java client from 2.0.0 to 0.10.1.1. Because high version client may not support low server server.
2. Enable SSL in librdkafka
2019-05-30 21:23:37 +08:00
fa4ac9f751 Replay GlobalVariable by Annotation (#1219) 2019-05-29 19:21:42 +08:00
319c71f69f Remove unnecessary import sun.security.ssl.Debug (#1215) 2019-05-29 12:35:20 +08:00
f648bdd968 Fix datediff function (#1208) 2019-05-28 15:55:31 +08:00
9d19c6c315 Support arbitrary kafka properties (#1204) 2019-05-28 10:03:50 +08:00
5ca2805701 Add some date time function doc (#1206) 2019-05-27 17:36:09 +08:00
08c8caeacf Add max cache size to ClientCache in BE (#1202)
Currently, unlimited client cache pool may cause too many connections in FE
2019-05-24 22:02:09 +08:00
41152f5c84 Modify FE sys log and audit log configuration (#1198)
All FE logs are now with suffix yyyyMMdd or yyyyMMddHH, specified by 2 new
configurations: 'sys_log_roll_interval' and 'audit_log_roll_interval'

All FE logs roll at max size of 1024MB(default), specified by new
configuration: 'log_roll_size_mb'

By default, the new FE logs will look like this:

log/
    fe.audit.log
    fe.audit.log.20190524-1
    fe.audit.log.20190523-2
    fe.log
    fe.log.20190524-1
    fe.log.20190524-2
    fe.log.20190523-3
    fe.warn.log
    fe.warn.log.20190524-1
    fe.warn.log.20190523-2
    fe.gc.log.20190524

Configurations 'sys_log_roll_mode' and 'audit_log_roll_mode' are deprecated.
2019-05-24 11:24:38 +08:00
85b4619d54 Change insert into to streaming (#1191)
The non-streaming hint of insert into will use the streamin plan which is same as the plan of stream insert.
It will also record the load info and return the label of insert stmt.
The partition is supportted in insert into stmt. The result which meet the target partitions will be loaded.
The introduction of example has been changed especially non-streaming insert.
Also, the param of partition_names is added in sql syntax which is used to declare the target partition_names in target table.

Change META_VERSION to 50
2019-05-23 20:53:30 +08:00
42601f2d88 Print unfinished tablet when cancel load job (#1186) 2019-05-23 19:22:54 +08:00
488e3825f7 Fix bug that restore process in BE causes BE crash (#1193)
When calling SnapshotLoader.move(), all files should be revoked if they
are in GC queue, or the file may be deleted after move() success.
2019-05-23 19:22:29 +08:00
cde315c9e9 Add date-function doc (#1190) 2019-05-23 15:29:08 +08:00
5d1457c0b6 Add check to create tablet upon alter tablet task (#1187)
When creating new tablet by alter tablet task,
next_unique_id will increase on the base old tablet.
If next_unique_id is eqaul to zero, it will cause that
ColumnDataMessage not match with tablet meta.
2019-05-23 14:17:22 +08:00
2fe1950d84 Fix minor string format bug (#1189) 2019-05-23 14:14:51 +08:00
d42409cc35 Fix short key not fill up all space (#1183) 2019-05-22 11:32:46 +08:00
5e245e031b Change source type to BATCH_LOAD_JOB (#1182) 2019-05-22 11:07:38 +08:00
592c2c24d9 Fix revoke files bug (#1181) 2019-05-22 11:06:31 +08:00
c5bf1a8da1 Fix prefix index comparison (#1180)
1. Upon prefix index comparison, it should only compare the fixed length of prefix index
2019-05-21 20:17:24 +08:00
b98b07efe2 Add persistence of load job v2 (#1176) 2019-05-21 19:30:26 +08:00
b132f4ac0c Add a configuration to force seek for block (#1179) 2019-05-21 14:53:37 +08:00
722a9e71c7 Optimize json functions (#1177)
1. get_json_xxx() now support using quoto to escape dot
2. Implement json_path_prepare() function to preprocess json_path

Performance of get_json_string() on 1000000 rows reduces from 2.27s to 0.27s
2019-05-21 09:13:12 +08:00
ff2746157e Remove log info from decimalv2_value to avoid performance degradation (#1175) 2019-05-20 14:26:14 +08:00
6528689205 Add samples about mini_load & insert (#997) 2019-05-20 12:26:49 +08:00
32c4a67f05 Fix thirdpary build error on Ubuntu 18.04 OS (#1174) 2019-05-20 12:24:35 +08:00
7f8a1bcdb6 Threadpool should be shutdown before join() (#1171) 2019-05-17 19:10:22 +08:00
9bf19f13bb Fix borker load submit job fail with null (#1172) 2019-05-17 19:09:44 +08:00
18e06d2e67 Fix position seek bug for varchar short key (#1167) 2019-05-16 17:29:43 +08:00
b2e63910a6 Fix bug that routine load task may be blocked due to premature deconstruction (#1166)
Data consumer group should wait all data consumers finished before return.
2019-05-16 16:15:00 +08:00
f985ea99fc Add support column reference in LOAD statement (#1162) 2019-05-15 20:26:10 +08:00
b24fab48cd Add some logs for compaction process (#1163) 2019-05-15 18:47:08 +08:00
afff3b3692 Modify dockerfile (#1164)
New docker image include new thirdparties
2019-05-15 18:35:47 +08:00
758adce761 Change the compaction thread number based on disks number (#1161)
Add 2 BE configs: base_compaction_num_threads_per_disk and cumulative_compaction_num_threads_per_disk to control the number of threads per disks.
2019-05-15 14:17:11 +08:00
47fb206cdf Skip tablet under compaction when choose candidate (#1160) 2019-05-15 13:05:32 +08:00
ad88741b69 Fix bug that bad tablet blocking compaction of other tablets (#1158)
A bad tablet is always be chosen to do compaction, and failed again
and again, which may block compaction of other tablets.
Add a BE config 'min_compaction_failure_interval_ms' to avoid choosing
bad tablet again at a certain interval, so that other tablets have
chance to do the compaction.

Also fix a bug that using avg() function on varchar column return
unexpected exception.
2019-05-15 12:44:38 +08:00
398055ef3e Add logic of cancel job (#1154) 2019-05-14 17:26:45 +08:00
910f16af81 Fix bug that using wrong capacity in trash sweep policy (#1156)
We should use total disk usage capacity instread of data used capacity,
otherwise, the config 'disk_capacity_insufficient_percentage' will not work.
2019-05-14 11:30:17 +08:00
76a8093c70 Add documentation for doris on es (#1151) 2019-05-13 21:58:05 +08:00
31f55532f7 Fix bug that cast decimal or bool to other type (#1152) 2019-05-13 17:41:10 +08:00
559db490e0 Fix bug that rename table to an existing rollup index name should not be allowed (#1150)
Also fix another bug that backup/restore job should add to task map after writing edit log.
ISSUE: #522
2019-05-13 17:10:51 +08:00
ffe3eaa1a7 Implement adddate, days_add and from_unixtime function in FE (#1149) 2019-05-13 16:59:52 +08:00
e7e92f0a89 Add build script for arrow (#1144) 2019-05-13 13:40:24 +08:00
02f36c23ed Set tablet as bad when loading index failed (#1146)
Bad tablet will be reported to FE and be handled

And add a config auto_recover_index_loading_failure to control the index loading failure processing
2019-05-13 10:22:04 +08:00