Commit Graph

2365 Commits

Author SHA1 Message Date
dafd41824b [fix](unique-key-merge-on-write) Fix missing delete bitmap introduced by DSIP-018 (#10548) (#10863)
Missing delete bitmap for copy constructor of `TabletMeta(TabletMeta&)`.
2022-07-15 11:58:49 +08:00
39325bd666 [Bug](tablet) Fix bug that segment files are removed as trash but tablet meta is normal (#10721) 2022-07-15 11:07:16 +08:00
18348a83ad [chore][compile] fix java udf compile error (#10841) 2022-07-14 20:51:44 +08:00
364c8733fa fix light schema change coredump (#10828) 2022-07-14 15:43:15 +08:00
ca5dbb1bcc Fix olap scan node normalize_in_and_eq_predicate infinite loop bug. (#10817) 2022-07-14 14:54:57 +08:00
d245ab76cc [improvement]Use uint32 instead of size_t to reduce agg key's length (#10832) 2022-07-14 14:11:55 +08:00
13e9cb146f [feature-wip](unique-key-merge-on-write) Add option to enable unique-key-merge-on-write, DSIP-018[5/1] (#10814)
* Add option in FE

* add opt in be

* some fix

* update

* fix code style

* fix typo

* fix typo

* update

* code format
2022-07-14 12:10:58 +08:00
d1573e1a4a [improvement]Use phmap for aggregation with serialized key (#10821) 2022-07-14 11:26:09 +08:00
e361eb385e [vectorized][udf] improvement java-udaf with group by clause (#10296)
save for file about udaf
add bool _destory_deserialize
update some code according reviewer
change destroy all data at once
2022-07-14 11:23:42 +08:00
3d52bff8d1 [improvement]output query_id when be core dumped. (#10822) 2022-07-14 10:55:28 +08:00
3b46242483 [feature-wip] Optimize Decimal type (#10794)
* [feature-wip](decimalv3) support decimalv3

* [feature-wip] Optimize Decimal type

Co-authored-by: liaoxin <liaoxinbit@126.com>
2022-07-14 10:50:50 +08:00
4f4ce4674a [fix](array) regression test fix for array (#10815) 2022-07-13 21:21:34 +08:00
56b55563c6 [feature-wip](unique-key-merge-on-write) add bloom filter index for primary key, DSIP-018[1.2] (#10706) 2022-07-13 18:58:45 +08:00
def59a686e [improvement]output fetal log to stderr (#10789) 2022-07-13 16:34:37 +08:00
d03b5c29a8 [bugfix] fix bug of ComparisonPredicate for nullable column (#10798) 2022-07-13 12:26:05 +08:00
Pxl
4190f7354c [Bug][Memtable] fix core dump on int128 because not aligned by 16 byte (#10775)
* fix core dump on int128 because not aligned by 16 byte

* update
2022-07-13 08:30:58 +08:00
6063c0c9c8 [enhancement](singal) output git commit id when the program coredump (#10788)
* [enhancement](singal) output git commit id when the program coredump

* modift output info
2022-07-13 08:24:58 +08:00
89e2678f4e [improvement]Increase min_ht_mem of StreamingHtMinReductionEntry (#10787) 2022-07-12 22:20:02 +08:00
89bec9b56a [enhancement](be) be asan add asan_suppr.conf to ignore known leak. (#10768) 2022-07-12 19:51:34 +08:00
486cf0ebd4 [Feature] Lightweight schema change of add/drop column (#10136)
* [Schema Change] support fast add/drop column  (#49)

* [feature](schema-change) support fast schema change. coauthor: yixiutt

* [schema change] Using columns desc from fe to read data. coauthor: Lchangliang

* [feature](schema change) schema change optimize for add/drop columns.

1.add uniqueId field for class column.
2.schema change for add/drop columns directly update schema meta

Co-authored-by: yixiutt <yixiu@selectdb.com>
Co-authored-by: SWJTU-ZhangLei <1091517373@qq.com>

[Feature](schema change) fix write and add regression test (#69)

Co-authored-by: yixiutt <yixiu@selectdb.com>

[schema change] be ssupport that delete use newest schema

add delete regression test

fix regression case (#107)

tmp

[feature](schema change) light schema change exclude rollup and agg/uniq/dup key type.

[feature](schema change) fe olapTable maxUniqueId write in disk.

[feature](schema change) add rpc iface for sc add column.

[feature](schema change) add columnsDesc to TPushReq for ligtht sc.

resolve the deadlock when schema change (#124)

fix columns from fe don't has bitmap_index flag (#134)

add update/delete case

construct MATERIALIZED schema from origin schema when insert

fix not vectorized compaction coredump

use segment cache

choose newest schema by schema version when compaction (#182)

[bugfix](schema change) fix ligth schema change problem.

[feature](schema change) light schema change add alter job. (#1)

fix be ut

[bug] (schema change) unique drop key column should not light schema
change

[feature](schema change) add schema change regression-test.

fix regression test

[bugfix](schema change) fix multi alter clauses for light schema change. (#2)

[bugfix](schema change) fix multi clauses calculate column unique id (#3)

modify PushTask process (#217)

[Bugfix](schema change) fix jobId replay cause bdbje exception.

[bug](schema change) fix max col unique id repeatitive. (#232)

[optimize](schema change) modify pendingMaxColUniqueId generate rule.

fix compaction error
* fix be ut

* fix snapshot load core

fix unique_id error (#278)

[refact](fe) remove redundant code for light schema change. (#4)

[refact](fe) remove redundant code for light schema change. (#4)

format fe core

format be core

fix be ut

modify fe meta version

fix rebase error

flush schema into rowset_meta in old table

[refactor](schema change) refact fe light schema change. (#5)

delete the change of schemahash and support get max version schema

* modify for review

* fix be ut

* fix schema change test
2022-07-12 19:41:06 +08:00
41f9ee2f9e mem_tracker_factor_v2 (#10743) 2022-07-12 18:09:41 +08:00
eb079950cb [feature-wip] (array-type) add the array_distinct function (#10388)
* add the array_distinct function

* add the support for decimal and update variable names

* add docs and regression test for array_distinct function

Co-authored-by: hucheng01 <hucheng01@baidu.com>
2022-07-12 17:02:42 +08:00
5f99a95816 fix use released memory bugs (#10777)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-07-12 16:54:32 +08:00
2084d8bdf3 [feature-wip](unique-key-merge-on-write) Add delete bitmap for DSIP-018 (#10548)
Add delete bitmap for
DSIP-018: Support Merge-On-Write implementation for UNIQUE KEY data model
2022-07-12 16:34:42 +08:00
4e9d5a7f7a optimize substr performance and fix ASAN global buffer overflow (#10442)
* add volnitsky substr algorithm

* replace std::search with volnitsky search algorithm in StringSearch

* optimize substring for constant_substring_fn case
use long run length search for performance
2022-07-12 08:36:21 +08:00
62fba5aa5d [Fix](distrubution) fix random tablet (#10756)
_compute_tablet_index catched up a local variable
2022-07-11 23:12:56 +08:00
3b9cb524bc [feature-wip](unique-key-merge-on-write) Add rowset tree, based on interval-tree, DSIP-018[3/3] (#10714)
* port from rowset-tree from kudu

* use shared_ptr

* some update

* add mock rowset

* some compatibility update

* fix ut fail

* reformat code
2022-07-11 23:10:38 +08:00
88f466ab86 [bugfix] temporarily disable pushing RF to scanner to avoid coredump (#10776) 2022-07-11 22:48:08 +08:00
a266d7b040 [bug](be) fix be _quick_compaction_thread_pool without shutdown. (#10758) 2022-07-11 22:33:56 +08:00
195d3b4a5a fix Level1Iterator memory leak (#10772) 2022-07-11 22:00:50 +08:00
9b554be698 [improvement]Division of integer is too slow (#10769) 2022-07-11 19:36:12 +08:00
5eb38467ef [bug](be) be asan core doris::DiskIoMgr::~DiskIoMgr(#10759) (#10760) 2022-07-11 19:04:16 +08:00
f21ce35059 [refactor]remove unused private field _profile (#10732) 2022-07-11 14:04:09 +08:00
277a7dd97e [bugfix]ColumnDecimal missed some interfaces about pre-serialization (#10751) 2022-07-11 14:00:58 +08:00
cc279d09a1 [BUG] Wrong result when build size is beyond IN runtime filter threshold (#10735) 2022-07-11 12:19:38 +08:00
639f1cd26c [improvement](parquet-reader) Add some profile for parquet reader (#10740) 2022-07-11 12:19:06 +08:00
8472ea8324 Revert "[Enhancement] Add column prune support for VOlapScanNode (#10615)" (#10734) 2022-07-11 12:16:08 +08:00
b04a791895 [Enhancement] support compile with jemalloc (#10542)
A test feature to use jemalloc as default malloc.
2022-07-11 12:15:35 +08:00
a044b5dcc5 [refactor](predicate) refactor predicates in scan node (#10701)
* [reafactor](predicate) refactor predicates in scan node

* update
2022-07-11 09:21:01 +08:00
4cb80c5733 [memtracker]fix fix_memtracker_performance_ (#10629) 2022-07-11 08:35:05 +08:00
9e9d6a4dea [Load][Vectorized] load opt code by change replace and replace_if_not_null do not copy value (#10447)
load opt code by change `replace` and `replace_if_not_null` do not copy value
2022-07-10 22:04:32 +08:00
502ac4e76b [Load][Vectorized] opt the mem use of aggregate function in load to speed up (#10448)
opt the mem use of aggregate function in load to speed up
2022-07-10 13:34:25 +08:00
7f9eeb8fc3 [BUG] runtime filter core dump (#10716) 2022-07-09 21:36:22 +08:00
1f08f2d144 [Bug][Vectorized] Support array function in where pre in volap_scan_node (#10467)
Co-authored-by: lihaopeng <lihaopeng@baidu.com>
Support array function in where pre in volap_scan_node
2022-07-09 16:22:01 +08:00
24d824a783 [improvement](multi-catalog) Impl parallel for file scanner to improve the scanner performance (#10620)
Add multi-thread support in FileScanNode on be and impl the file spilt logic in fe.
2022-07-09 15:52:53 +08:00
d5ea677282 [feature](tracing) Support query tracing to improve doris observability by introducing OpenTelemetry. (#10533)
The collection of query traces is implemented in fe and be, and the spans are exported to zipkin.
DSIP: https://cwiki.apache.org/confluence/display/DORIS/DSIP-012%3A+Introduce+opentelemetry
2022-07-09 15:50:40 +08:00
1112dba525 [be ut]add some case for array type in block_test (#10656)
Co-authored-by: hucheng01 <hucheng01@baidu.com>
2022-07-09 12:00:42 +08:00
ed4b2140d7 [BUG](datev2) fix bloom filter for datev2 and remove redundant code (#10695) 2022-07-09 06:26:28 +08:00
08384fea1c [BUG] fix DCHECK failed for vectorized InPredicate (#10709) 2022-07-09 06:25:32 +08:00
e293fbd277 [improvement]pre-serialize aggregation keys (#10700) 2022-07-09 06:21:56 +08:00