Commit Graph

6100 Commits

Author SHA1 Message Date
33e46ee13d [enhancement](config) enable single_replica_load by default in BE (#26619)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-11-09 12:14:37 +08:00
d1438a8563 [Fix](orc-reader) Fix orc complex types when late materialization was turned on by disabling late materialization in this case. (#26548)
Fix orc complex types when late materialization was turned on in orc reader by disabling late materialization in this case.
2023-11-09 12:05:43 +08:00
01094fd25e [Coverage](BE) Delete vinfo_func in BE (#26562)
Delete vinfo_func in BE
2023-11-09 11:00:15 +08:00
95f74f1544 [FIX](complextype)fix shrink in topN for complex type #26609 2023-11-09 10:56:14 +08:00
74e452f19c [bug](bitmap) fix bitmap value copy operator not call reset (#26451)
when a empty bitmap assign to other bitmap
the other bitmap should reset self firstly, and then set empty type.
2023-11-09 10:05:09 +08:00
66e591f7f2 [enhancement](brpc) add a auto release closure to ensure the closue safety (#26567) 2023-11-09 08:50:42 +08:00
55b2988bfd [Opt](date_add/sub) Throw exception when result of date_add/sub out of range (#26475) 2023-11-09 08:46:51 +08:00
a6f9df7096 [LOG] Add fatal log in exchange sink buffer (#26594) 2023-11-08 21:52:21 +08:00
d0960bac56 [Fix](partial update) Fix partial update info loss when the delete bitmaps of the committed transactions are calculated by the compaction (#26556)
a fix for #25147
2023-11-08 19:56:31 +08:00
3bce6d3828 [Opt](orc-reader) Optimize orc string dict filter in not_single_conjunct case. (#26386)
Optimize orc/parquet string dict filter in not_single_conjunct case. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string.

For example:
```
select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate  and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
```
 `l_receiptdate` and `l_shipmode` will using string dict filtering, and `l_commitdate < l_receiptdate` is the an not_single_conjunct which contains dict filter field. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string.

### Test Result:
Before:
 mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate  and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
+----------------------+
| count(l_receiptdate) |
+----------------------+
|             49314694 |
+----------------------+
1 row in set (6.87 sec)

After:
mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate  and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
+----------------------+
| count(l_receiptdate) |
+----------------------+
|             49314694 |
+----------------------+
1 row in set (4.85 sec)
2023-11-08 18:03:18 +08:00
58bf79f79e [fix](move-memtable) pass load stream num to backends (#26198) 2023-11-08 16:16:33 +08:00
6637f9c15f Add enable_cgroup_cpu_soft_limit (#26510) 2023-11-08 15:52:13 +08:00
f018b00646 [ci](perf) add new pipeline of tpch-sf100 (#26334)
* [ci](perf) add new pipeline of tpch-sf100
Co-authored-by: stephen <hello-stephen@qq.com>
2023-11-08 15:32:02 +08:00
a3666aa87e [feature](decimal) support decimal256 when creating table (#26308) 2023-11-08 15:21:01 +08:00
44b51bf0b9 [Feature](Variant) support variant load (#26572) 2023-11-08 00:37:57 -06:00
9502cc758d [fix](regression) fix group commit regression test (#26557) 2023-11-08 11:57:07 +08:00
a2419a8eb4 [enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 (#26432)
For better performance and elasticity, we move memtable from loadchannel to
sink, VTabletSinkV2 is introduced, then there are VTabletWriter and
VTabletSinkV2 distributing rows to tablets. where clauses on mvs are
executed in VTabletWriter, while VTabletSinkV2 needs it too. So common code
is moved to row distribution.

Actually, we can layer code by rows' data flow, then the code is much more
understood and maintainable.

ScanNode -> Sink/Writer (RowDistribution -> IndexChannel / DeltaWriter)
2023-11-08 11:51:40 +08:00
47ba4aaf30 [Enhancement](load) add timer and partitions number limit (#26549)
add timer and partitions number limit
2023-11-08 11:22:40 +08:00
1544110c1b [feature-wip](arrow-flight)(step4) Support other DML and DDL statements, besides Select (#25919)
Design Documentation Linked to #25514
2023-11-08 10:50:42 +08:00
Pxl
3cdbb6e637 [Bug](materialized-view) fix some bugs on create mv with percentile_approx (#26528)
1. percentile_approx have wrong symbol
2. fnCall.getParams() get obsolete childrens
2023-11-08 10:09:37 +08:00
519b48648e [fix](move-memtable) handle status when possible (#26526) 2023-11-08 10:09:06 +08:00
607a5d25f1 [feature](streamload) support HTTP request with chunked transfer (#26520) 2023-11-08 10:07:05 +08:00
a354f87d2e [refactor](pipeline) simplify runtime state ctor (#26461) 2023-11-08 09:57:09 +08:00
70bc8600a9 [fix](regression) fix regression framework bug: if real test result is negative, it will miss check test result (#25734) 2023-11-08 09:05:58 +08:00
a6756b4660 [pipelineX](bug) Fix broadcast buffer reference count (#26545) 2023-11-08 00:14:48 +08:00
4995ca8fba [fix](move-memtable) ensure segment is flushed before add segment (#26522) 2023-11-07 22:42:16 +08:00
32b36d3c9c [refactor](move-memtable) rename proto OpenStreamSink to OpenLoadStream (#26527) 2023-11-07 22:41:20 +08:00
5d80e7dc2f [Improvement](pipelineX) Improve local exchange on pipelineX engine (#26464) 2023-11-07 22:11:44 +08:00
2bb3ef1981 [refactor](scan) delete bloom_filter_predicate (#26499) 2023-11-07 19:37:31 +08:00
d6eb3324a1 [cleanup](load) remove unused code in sink v2 header (#26521) 2023-11-07 19:35:12 +08:00
a404ff5ab9 [fix](regression) fix group commit regression test (#26519) 2023-11-07 18:17:45 +08:00
ef95e962c7 [fix](timev2) fix Type not implemented in fold by be (#26478) 2023-11-07 17:25:20 +08:00
3ad8e27b09 [Fix](autoinc) Init auto increment info in VOlapTableSinkV2 (#26502) 2023-11-07 16:51:38 +08:00
8da1a9a370 [pipeline](fix) remove unreasonable CHECK (#26504) 2023-11-07 15:48:07 +08:00
277329c035 [fix](auditlog) fix without lock in QueryStatisticsRecvr find #26440 2023-11-07 13:53:22 +08:00
9687932d57 [refactor](function) improve compoundPred optimization work with children is nullable (#26160)
before this optimization work has limit, it's children must not nullable.
2023-11-07 13:52:10 +08:00
16644eff7f [opt](load) optimize the performance of row distribution (#25546)
For non-pipeline non-sinkv2:
before: 14s
now: 6s-
For pipeline + sinkv2:
before: 230ms *48 instances
now: 38ms *48 instances
2023-11-07 10:04:59 +08:00
fa7a38b587 [fix](runtime filter) append late arrival runtime filters in vfilecanner (#25996)
`VFileScanner` will try to append late arrival runtime filters in each loop of `ScannerScheduler::_scanner_scan`.  However, `VFileScanner::_get_next_reader` only generates the `_push_down_conjuncts` in the first loop, so the late arrival runtime filters are ignored.
2023-11-07 09:50:35 +08:00
bf8793de70 [fix](load) fix merged row number miscounting because of race condition (#26470)
row numbers miscounting because of race condition, will cause load to
fail sometimes with warning 'the rows number written doesn't match'.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-11-06 22:50:32 +08:00
c9d90b636e [feature](pipelineX) add time unit when slow_dependency need to log (#26466) 2023-11-06 19:58:07 +08:00
bd89028306 [bug](pipelineX) Fix potential bug using broadcast shuffle (#26458) 2023-11-06 17:33:20 +08:00
1a83a39aec Revert "[fix](auto-partition) Fix auto partition concurrent conflict (#26166)" (#26448)
This reverts commit f22611769944e78c28f1b0a1eeb7b7414a16e8db.
2023-11-06 16:39:19 +08:00
c0d9a8d53c [minor](pipelineX) refine error message for broadcast shuffle buffer (#26442) 2023-11-06 15:10:13 +08:00
2cc68381ec [feature](binlog) Add ingest_binlog/http_get_snapshot limit download speed && Add async ingest_binlog (#26323) 2023-11-06 11:14:44 +08:00
1e2a614a46 [fix](workflow) Fix failure test cases in BE UT (macOS) (#26425)
1. Fix memory issues in LoadStreamMgrTest.
2. Skip S3FileWriterTest by default because it depends on the environment in teamcity.
3. Fix VTimestampFunctionsTest.convert_tz_test.
2023-11-06 10:44:44 +08:00
f226117699 [fix](auto-partition) Fix auto partition concurrent conflict (#26166) 2023-11-06 10:34:26 +08:00
a5b9d61f43 [bug](udf) fix java-udf function const column without arguments (#26409) 2023-11-06 10:34:09 +08:00
99de6c7afe [fix](invert index) Fix the timing error when opening the searcher (#26401) 2023-11-04 01:52:35 -05:00
6ef2e62600 [FIX](struct) fix struct be nested when read will make core (#26270) 2023-11-03 09:55:39 -05:00
f92b572a97 [fix](invert index) optimize SeekTime profile (#26353) 2023-11-03 03:03:29 -05:00