4ebb517af0
[fix](be-ut) Fix compilation errors caused by missing opentelemetry headers ( #26739 )
2023-11-10 14:58:46 +08:00
899630d0eb
[chore](key_util) remove useless null_first parameter ( #26635 )
...
Doris always put null in the first when sorting key, the parameter null_first of encode_keys is useless.
2023-11-10 14:27:47 +08:00
7878c08e15
[Revert](merge-on-write) Don't use delete bitmap to mark delete for rows with delete sign when sequence column doesn't exist ( #26721 )
2023-11-10 13:55:40 +08:00
7754791146
[improvement](disk balance) Prevent duplicate disk balance tasks afte… ( #25990 )
2023-11-10 10:14:42 +08:00
2bf48d7829
Revert "[Coverage](BE) Delete vinfo_func in BE ( #26562 )" ( #26723 )
...
This reverts commit 01094fd25ed539a8025066d8823c1e907109048a.
2023-11-10 10:14:11 +08:00
d767804815
[feature](merge-cloud) Decouple rowset id generator and local rowsets gc implementation ( #25921 )
2023-11-10 10:07:02 +08:00
d988193d39
[pipelineX](shuffle) block exchange sink by memory usage ( #26595 )
2023-11-09 21:28:22 +08:00
c07a70e22a
[Fix](orc-reader) Add missing break introduced by #26548 . ( #26633 )
...
Add missing break introduced by #26548 . Sorry for this mistake.
2023-11-09 18:29:44 +08:00
a5565f68b2
[Refactor](opentelemetry) Remove opentelemetry ( #26605 )
2023-11-09 18:05:34 +08:00
eca747413d
[Fix](partial update) Fix core when doing partial update on tables with row column after schema change ( #26632 )
2023-11-09 18:00:05 +08:00
baae7bf339
[fix](information_schema)fix bug that metadata_name_ids error tableid and append information_schema case. ( #26238 )
...
fix bug that #24059 .
Added some information_schema scanner tests.
files
schema_privileges
table_privileges
partitions
rowsets
statistics
table_constraints
Based on infodb_support_ext_catalog=false, it currently includes tests for all tables under the information_schema database.
2023-11-09 14:07:12 +08:00
22bf2889e5
[feature](tvf)(jni-avro)jni-avro scanner add complex data types ( #26236 )
...
Support avro's enum, record, union data types
2023-11-09 13:58:49 +08:00
5f62a4462d
[Enhancement](wal) Add wal space back pressure ( #26483 )
2023-11-09 12:29:05 +08:00
33e46ee13d
[enhancement](config) enable single_replica_load by default in BE ( #26619 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-11-09 12:14:37 +08:00
d1438a8563
[Fix](orc-reader) Fix orc complex types when late materialization was turned on by disabling late materialization in this case. ( #26548 )
...
Fix orc complex types when late materialization was turned on in orc reader by disabling late materialization in this case.
2023-11-09 12:05:43 +08:00
01094fd25e
[Coverage](BE) Delete vinfo_func in BE ( #26562 )
...
Delete vinfo_func in BE
2023-11-09 11:00:15 +08:00
95f74f1544
[FIX](complextype)fix shrink in topN for complex type #26609
2023-11-09 10:56:14 +08:00
74e452f19c
[bug](bitmap) fix bitmap value copy operator not call reset ( #26451 )
...
when a empty bitmap assign to other bitmap
the other bitmap should reset self firstly, and then set empty type.
2023-11-09 10:05:09 +08:00
66e591f7f2
[enhancement](brpc) add a auto release closure to ensure the closue safety ( #26567 )
2023-11-09 08:50:42 +08:00
55b2988bfd
[Opt](date_add/sub) Throw exception when result of date_add/sub out of range ( #26475 )
2023-11-09 08:46:51 +08:00
a6f9df7096
[LOG] Add fatal log in exchange sink buffer ( #26594 )
2023-11-08 21:52:21 +08:00
d0960bac56
[Fix](partial update) Fix partial update info loss when the delete bitmaps of the committed transactions are calculated by the compaction ( #26556 )
...
a fix for #25147
2023-11-08 19:56:31 +08:00
3bce6d3828
[Opt](orc-reader) Optimize orc string dict filter in not_single_conjunct case. ( #26386 )
...
Optimize orc/parquet string dict filter in not_single_conjunct case. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string.
For example:
```
select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
```
`l_receiptdate` and `l_shipmode` will using string dict filtering, and `l_commitdate < l_receiptdate` is the an not_single_conjunct which contains dict filter field. We can optimize this processing to filter block firstly by dict code, then filter by not_single_conjunct. Because dict code is int, it will filter faster than string.
### Test Result:
Before:
mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
+----------------------+
| count(l_receiptdate) |
+----------------------+
| 49314694 |
+----------------------+
1 row in set (6.87 sec)
After:
mysql> select count(l_receiptdate) from lineitem_date_as_string where l_shipmode in ('MAIL', 'SHIP') and l_commitdate < l_receiptdate and l_receiptdate >= '1994-01-01' and l_receiptdate < '1995-01-01';
+----------------------+
| count(l_receiptdate) |
+----------------------+
| 49314694 |
+----------------------+
1 row in set (4.85 sec)
2023-11-08 18:03:18 +08:00
58bf79f79e
[fix](move-memtable) pass load stream num to backends ( #26198 )
2023-11-08 16:16:33 +08:00
6637f9c15f
Add enable_cgroup_cpu_soft_limit ( #26510 )
2023-11-08 15:52:13 +08:00
f018b00646
[ci](perf) add new pipeline of tpch-sf100 ( #26334 )
...
* [ci](perf) add new pipeline of tpch-sf100
Co-authored-by: stephen <hello-stephen@qq.com >
2023-11-08 15:32:02 +08:00
a3666aa87e
[feature](decimal) support decimal256 when creating table ( #26308 )
2023-11-08 15:21:01 +08:00
44b51bf0b9
[Feature](Variant) support variant load ( #26572 )
2023-11-08 00:37:57 -06:00
9502cc758d
[fix](regression) fix group commit regression test ( #26557 )
2023-11-08 11:57:07 +08:00
a2419a8eb4
[enhancement](sink) refactor code of auto partition and where clause and enable them on sinkv2 ( #26432 )
...
For better performance and elasticity, we move memtable from loadchannel to
sink, VTabletSinkV2 is introduced, then there are VTabletWriter and
VTabletSinkV2 distributing rows to tablets. where clauses on mvs are
executed in VTabletWriter, while VTabletSinkV2 needs it too. So common code
is moved to row distribution.
Actually, we can layer code by rows' data flow, then the code is much more
understood and maintainable.
ScanNode -> Sink/Writer (RowDistribution -> IndexChannel / DeltaWriter)
2023-11-08 11:51:40 +08:00
47ba4aaf30
[Enhancement](load) add timer and partitions number limit ( #26549 )
...
add timer and partitions number limit
2023-11-08 11:22:40 +08:00
1544110c1b
[feature-wip](arrow-flight)(step4) Support other DML and DDL statements, besides Select ( #25919 )
...
Design Documentation Linked to #25514
2023-11-08 10:50:42 +08:00
3cdbb6e637
[Bug](materialized-view) fix some bugs on create mv with percentile_approx ( #26528 )
...
1. percentile_approx have wrong symbol
2. fnCall.getParams() get obsolete childrens
2023-11-08 10:09:37 +08:00
519b48648e
[fix](move-memtable) handle status when possible ( #26526 )
2023-11-08 10:09:06 +08:00
607a5d25f1
[feature](streamload) support HTTP request with chunked transfer ( #26520 )
2023-11-08 10:07:05 +08:00
a354f87d2e
[refactor](pipeline) simplify runtime state ctor ( #26461 )
2023-11-08 09:57:09 +08:00
70bc8600a9
[fix](regression) fix regression framework bug: if real test result is negative, it will miss check test result ( #25734 )
2023-11-08 09:05:58 +08:00
a6756b4660
[pipelineX](bug) Fix broadcast buffer reference count ( #26545 )
2023-11-08 00:14:48 +08:00
4995ca8fba
[fix](move-memtable) ensure segment is flushed before add segment ( #26522 )
2023-11-07 22:42:16 +08:00
32b36d3c9c
[refactor](move-memtable) rename proto OpenStreamSink to OpenLoadStream ( #26527 )
2023-11-07 22:41:20 +08:00
5d80e7dc2f
[Improvement](pipelineX) Improve local exchange on pipelineX engine ( #26464 )
2023-11-07 22:11:44 +08:00
2bb3ef1981
[refactor](scan) delete bloom_filter_predicate ( #26499 )
2023-11-07 19:37:31 +08:00
d6eb3324a1
[cleanup](load) remove unused code in sink v2 header ( #26521 )
2023-11-07 19:35:12 +08:00
a404ff5ab9
[fix](regression) fix group commit regression test ( #26519 )
2023-11-07 18:17:45 +08:00
ef95e962c7
[fix](timev2) fix Type not implemented in fold by be ( #26478 )
2023-11-07 17:25:20 +08:00
3ad8e27b09
[Fix](autoinc) Init auto increment info in VOlapTableSinkV2 ( #26502 )
2023-11-07 16:51:38 +08:00
8da1a9a370
[pipeline](fix) remove unreasonable CHECK ( #26504 )
2023-11-07 15:48:07 +08:00
277329c035
[fix](auditlog) fix without lock in QueryStatisticsRecvr find #26440
2023-11-07 13:53:22 +08:00
9687932d57
[refactor](function) improve compoundPred optimization work with children is nullable ( #26160 )
...
before this optimization work has limit, it's children must not nullable.
2023-11-07 13:52:10 +08:00
16644eff7f
[opt](load) optimize the performance of row distribution ( #25546 )
...
For non-pipeline non-sinkv2:
before: 14s
now: 6s-
For pipeline + sinkv2:
before: 230ms *48 instances
now: 38ms *48 instances
2023-11-07 10:04:59 +08:00
