bbf58c5aa4
[improve](move-memtable) cancel load rapidly when stream close wait ( #29322 )
2023-12-31 16:26:41 +08:00
4634c723d7
[fix](move-memtable) only check missing tablets when commit info is not empty ( #29326 )
...
Some sink may not get any success tablets, because load stream will only report them at the last CLOSE_LOAD.
For these sinks (no commit info), we should not check missing tablets since it's expected to be empty.
2023-12-31 13:50:13 +08:00
fcc4cfb900
[Fix](Variant) add more info before crash in serialization ( #29344 )
2023-12-31 11:17:36 +08:00
7623b5cc31
[cleanup](move-memtable) remove namespace stream_load ( #27441 )
2023-12-30 20:08:23 +08:00
03901b9a7a
[enhancement](group_commit): refector relay wal code ( #29183 )
2023-12-30 12:59:46 +08:00
e7d67e9411
[fix](be) resolves some unused-raii and used-after-moved issues ( #29285 )
2023-12-30 12:14:49 +08:00
51cb15d032
[improve](move-memtable) cancel load immediately when back pressure in delta writer v2 ( #29280 )
2023-12-30 10:45:06 +08:00
5bc72bda78
[improve](move-memtable) add fault injection in writer v2 ( #29177 )
2023-12-30 01:32:26 +08:00
41d907912a
[improve](move-memtable) add fault injection in load stream stub ( #29105 )
2023-12-30 01:29:28 +08:00
411f819306
[Opt](join) opt the join copy column from build side ( #29007 )
2023-12-29 14:39:33 +08:00
b9572f9de0
[pipelineX](fix) Fix pip scanner context bug ( #29229 )
2023-12-29 13:24:39 +08:00
a525d5c5a3
[refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion ( #29265 )
...
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
2023-12-29 10:11:44 +08:00
269c1b189d
[improve](vtablet_writer) check runtime state is cancel when back pressure ( #29260 )
2023-12-29 09:11:24 +08:00
fb0ed8c253
[fix](move-memtable) check missing tablets before commit ( #29223 )
2023-12-29 00:33:58 +08:00
9be0f04506
(improv)[group commit] refactor some group commit code ( #29180 )
2023-12-29 00:26:10 +08:00
a90304c208
[fix](parquet) complex type in parquet is case sensitive ( #29245 )
...
Change name of complex type in parquet to case-insensitive. Otherwise, uppercase column names of complex types will return null.
2023-12-28 22:43:11 +08:00
ffd178f5ff
[feat](pipelinex) support parallel scan on pipeline x engine ( #29070 )
...
* [feat](pipelinex) support parallel scan on pipeline x engine
* make parallel scan be independent of shared scan
2023-12-28 21:29:07 +08:00
5129ab5738
[fix](decimalv2) fix decimalv2 agg errors ( #29246 )
2023-12-28 21:17:16 +08:00
fe93a8f1d0
[cleanup](move-memtable) remove unused log in load stream stub ( #29084 )
2023-12-28 14:39:10 +08:00
3dc3e81734
[Improvement](datatype) Update Parser for IPv4/v6 data types ( #29044 )
...
Transforming from parsing std:: string to parsing char * to accelerate the parsing of ipv4/v6 data types.
2023-12-28 11:00:38 +08:00
e5b6826de6
[fix](partial update) update error code when failed to fill the missing fields ( #29103 )
...
1. InternalError is not clear for such error, use InvalidSchema Error instead
2. avoid some useless stacktrace on InternalError when load failed
2023-12-28 10:33:03 +08:00
8b225c6c3c
[pipelineX](fix) Fix core dump if cancelled ( #29138 )
2023-12-28 10:04:51 +08:00
0cc4ee52bf
[fix](move-memtable) fix streams for node memory leak in sink v2 ( #29146 )
2023-12-27 21:48:32 +08:00
0436013baf
[fix](decimal) fix cast decimal overflow and add test cases for casting decimalv2 to decimalv3 ( #29165 )
2023-12-27 20:58:37 +08:00
5f71691401
[fix](read) fix unexpected overflow of uninitialized column data in VStatisticsIterator::next_batch ( #29141 )
2023-12-27 20:58:02 +08:00
c75e63a2a5
[Improvement](scan) Use scanner to do projection of scan node ( #29124 )
2023-12-27 16:00:52 +08:00
2d2f14bc75
[fix](paimon) use SlotDescriptor to parse the required fields ( #28990 )
...
Before this PR, Paimon has created the schema of `VectorTable` by accessing meta information. However, once the schema of `VectorTable` in java is not same as `Block` in c++, BE will crashed, and there is no good way to troubleshoot errors.
2023-12-27 15:45:53 +08:00
cfed36afbf
[Fix](topn opt) prevent from merge __TEMP__ column in segment iterator ( #29121 )
2023-12-27 15:42:48 +08:00
9ff8bd2e9c
[Enhancement](Wal)Support dynamic wal space limit ( #27726 )
2023-12-27 11:51:32 +08:00
6d26aca4ca
[fix](pipeline) sort_merge should throw exception in has_next_block if got failed status ( #29076 )
...
Test in regression-test/suites/datatype_p0/decimalv3/test_decimalv3_overflow.groovy::249 sometimes failed when there are multiple BEs and FE process report status slowly for some reason.
explain select k1, k2, k1 * k2 from test_decimal128_overflow2 order by 1,2,3
--------------
+----------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner) |
+----------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0 |
| OUTPUT EXPRS: |
| k1[#5 ] |
| k2[#6 ] |
| (k1 * k2)[#7 ] |
| PARTITION: UNPARTITIONED |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| VRESULT SINK |
| MYSQL_PROTOCAL |
| |
| 111:VMERGING-EXCHANGE |
| offset: 0 |
| |
| PLAN FRAGMENT 1 |
| |
| PARTITION: HASH_PARTITIONED: k1[#0 ], k2[#1 ] |
| |
| HAS_COLO_PLAN_NODE: false |
| |
| STREAM DATA SINK |
| EXCHANGE ID: 111 |
| UNPARTITIONED |
| |
| 108:VSORT |
| | order by: k1[#5 ] ASC, k2[#6 ] ASC, (k1 * k2)[#7 ] ASC |
| | offset: 0 |
| | |
| 102:VOlapScanNode |
| TABLE: regression_test_datatype_p0_decimalv3.test_decimal128_overflow2(test_decimal128_overflow2), PREAGGREGATION: ON |
| partitions=1/1 (test_decimal128_overflow2), tablets=8/8, tabletList=22841,22843,22845 ... |
| cardinality=6, avgRowSize=0.0, numNodes=1 |
| pushAggOp=NONE |
| projections: k1[#0 ], k2[#1 ], (k1[#0 ] * k2[#1 ]) |
| project output tuple id: 1 |
+----------------------------------------------------------------------------------------------------------------------------+
36 rows in set (0.03 sec)
Why failed:
Multiple BEs
Fragments 0 and 1 are MUST on different BEs
Pipeline task of VOlapScanNode which executes k1*k2 failed sets query status to cancelled
Pipeline task of VSort call try close, send Cancelled status to VMergeExchange
sort_curso did not throw exception when it meets error
2023-12-27 10:06:01 +08:00
6440fbfab6
[feature](scan) Implement parallel scanning by dividing the tablets based on the row range ( #28967 )
...
* [feature](scan) parallel scann on dup/mow mode
* fix bugs
2023-12-26 17:18:41 +08:00
f30e50676e
[opt](scanner) optimize the number of threads of scanners ( #28640 )
...
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
2023-12-26 10:24:12 +08:00
137f785698
[fix](parquet_reader) misused bool pointer ( #28986 )
...
Signed-off-by: pengyu <pengyu@selectdb.com >
2023-12-25 22:58:08 +08:00
c2c5df9341
[opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain ( #28935 )
...
* NEED
* Update pipeline x
* fix pipelinex compile
2023-12-25 22:47:23 +08:00
0af9371a96
[fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse ( #28991 )
...
Introduced by #28851 , after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.
e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.
2023-12-25 22:19:01 +08:00
7081139bdc
[fix](block) fix be core while mutable block merge may cause different row size between columns in origin block ( #27943 )
2023-12-25 20:35:22 +08:00
e9e1e2894b
[performance](variant) support topn 2phase read for variant column ( #28318 )
...
[performance](variant) support topn 2phase read for variant column
2023-12-25 11:50:41 +08:00
f374beaa4e
[fix](log) regularise some BE error type and fix a load task check #28729
2023-12-25 10:45:19 +08:00
e326ebb63e
[feature](pipelineX) control exchange sink by memory usage ( #28814 )
2023-12-25 10:31:50 +08:00
b7ae7a07c7
[fix](join) incorrect result of left semi/anti join with empty build side ( #28898 )
2023-12-25 09:07:38 +08:00
1545c36d16
Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile ( #28727 )" ( #28931 )
...
This reverts commit 4066de375efe6ff8e156a61df4f9316b3d9eaa4e.
2023-12-24 20:37:33 +08:00
db1da161f5
[optimize](zonemap) skip zonemap if predicate does not support_zonemap ( #28595 )
...
* [optimize](zonemap) skip zonemap if predicate does not support_zonemap #27608 (#28506 )
2023-12-24 19:34:13 +08:00
96d4778f2e
[fix](parquet) the end offset of column chunk may be wrong in parquet metadata ( #28891 )
2023-12-23 22:21:04 +08:00
2014396707
[fix](block) add block columns size dcheck ( #28539 )
2023-12-23 15:21:53 +08:00
e51f75e424
[FIX](map)fix map with rowstore table ( #28877 )
2023-12-23 12:11:06 +08:00
4066de375e
[bugfix](scannercore) scanner will core in deconstructor during collect profile ( #28727 )
2023-12-23 11:09:46 +08:00
fa0ad56817
[exec](compress) use FragmentTransmissionCompressionCodec control the exchange compress behavior ( #28818 )
2023-12-22 19:50:57 +08:00
aca8406e31
[refactor](executor)remove scan group #28847
2023-12-22 17:05:50 +08:00
d75300f166
[fix](hash join) fix stack overflow caused by evaluate case expr on huge build block ( #28851 )
2023-12-22 15:45:12 +08:00
012e66729a
[improvement](executor) Add tvf and regression test for Workload Scheduler ( #28733 )
...
1 Add select workload schedule policy tvf
2 Add reg test
2023-12-22 12:09:51 +08:00
