bb709ad917
[branch-2.1] Picks "[Fix](merge-on-write) Fix duplicate key problem after adding sequence column for merge-on-write table #39958 " ( #40010 )
...
## Proposed changes
picks https://github.com/apache/doris/pull/39958
2024-08-28 23:00:31 +08:00
af6c39d770
[fix](compile) compile fail on MacOS because 'bits/types/struct_timespec.h' not found ( #39567 ) ( #40044 )
...
pick #39567 to branch-2.1
Co-authored-by: 神技圈子 <songguangfan@gmail.com >
Co-authored-by: 宋光璠 <songguangfan@sf.com >
2024-08-28 22:01:49 +08:00
1d439d2ea9
[opt](parquet) add predicate filter time for parquet reader ( #40005 ) ( #40053 )
...
bp #40005
2024-08-28 21:57:00 +08:00
4909c34555
[fix](compile) Fix Type Mismatch in min Function for VMysqlResultWriter on macOS ( #38202 ) ( #40042 )
...
pick #38202 to branch-2.1
Co-authored-by: zy-kkk <zhongyk10@gmail.com >
2024-08-28 21:36:04 +08:00
b052f46944
[fix](json-quote) fix json quote func for not find the func ( #39931 ) ( #40061 )
...
1. fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;
[17:47]> select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
```
after pr , we can carry on it
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
1# 0x00007F9429595B50 in /lib64/libc.so.6
2# gsignal in /lib64/libc.so.6
3# __GI_abort in /lib64/libc.so.6
4# _nl_load_domain.cold.0 in /lib64/libc.so.6
5# 0x00007F942958E426 in /lib64/libc.so.6
6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-28 21:31:08 +08:00
ddf72ce09a
[Fix](branch-2.1) fix manual pick with mistake when handling decimal … ( #40008 )
...
…type
introduced by #39843
2024-08-28 10:15:52 +08:00
6915d76731
[opt](file-cache) add evict file number per round ( #39721 )
...
Previously, when getting block from file cache, it may try to evict
lots of blocks to reserve capacity for lru cache. This operation may
take long time
while hold the lock, causing other operation blocked.
This PR add a new BE config `file_cache_max_evict_num_per_round`,
default is 1000, so that it will not hold lock for a long time.
2024-08-28 08:49:12 +08:00
cb312cabb2
[Fix](tablet-meta) limit the data size of tablet meta ( #39455 ) ( #39974 )
...
pick master #39455
2024-08-27 20:10:17 +08:00
29088ebaed
[enhancement](compression) prints the specific exception when ZSTD compress fails ( #39433 ) ( #39978 )
...
cherry-pick #39433 to branch-2.1
2024-08-27 20:07:15 +08:00
4127eec9a7
[fix](function) fix error return type in mod(float32,BigInt) ( #39358 ) ( #39971 )
...
## Proposed changes
https://github.com/apache/doris/pull/39358
```
CREATE TABLE testdb (
K1 BIGINT,
K2 FLOAT
) properties("replication_num" = "1");
insert into testdb values(1,1.1);
select mod(k1,k2) from testdb;
mysql [test10]>select mod(k1,k2) from testdb;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Function mod get failed, expr is VectorizedFnCall[mod](arguments=K1, K2,return=Nullable(Float32)) and return type is Nullable(Float32).
```
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 18:32:52 +08:00
fc9936d923
[fix](funcs) fix map struct construct funcs ( #39973 )
...
## Proposed changes
backport: https://github.com/apache/doris/pull/39699
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 18:31:51 +08:00
173aafc86f
[Enhancement] add information_schema.table_properties #38745 ( #38746 ) ( #39886 )
...
bp #38746
---------
Co-authored-by: Vallish Pai <vallishpai@gmail.com >
2024-08-27 17:22:19 +08:00
06a2182d89
[fix](build index)Only get file size for inverted index ( #39966 )
...
## Proposed changes
Fix `NOT FOUND` index file error message in logs.
bp #39965
2024-08-27 15:30:51 +08:00
b7e0bfa1c2
[improve](function) opt aes_encrypt/decrypt function to handle const column ( #37194 ) ( #39954 )
...
pick #37194 to branch-2.1
---------
Co-authored-by: zhangstar333 <87313068+zhangstar333@users.noreply.github.com >
2024-08-27 14:56:38 +08:00
4cf769b39f
[Improve](table-function) explode json array with json args ( #39491 )
2024-08-27 14:53:17 +08:00
8256c6f0ba
[Fix](parquet-reader) Fix definition level rle decode dead loop in parquet-reader. ( #39523 ) ( #39945 )
...
bp #39523
Co-authored-by: Qi Chen <kaka11.chen@gmail.com >
2024-08-27 08:54:43 +08:00
db0724dfe0
[Fix-2.1](function) fix function covar core for not null input ( #39943 )
...
## Proposed changes
Issue Number: close #xxx
add testcases like:
```groovy
qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```
before they will all coredump in 2.1
2024-08-27 08:39:47 +08:00
21bd4a4ac8
[bug](function)fix json_replace check return type error ( #37014 ) ( #39938 )
...
1. fix the return type dcheck error:
```
mysql [test]>select (json_replace(a, '$.fparam.nested_2', "qwe")) from json_table_2 limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]Function json_replace get failed, expr is VectorizedFnCall[json_replace](arguments=a, String, String, String,return=Nullable(String)) and return type is Nullable(String).
```
2. improve the json_replace/json_insert/json_set function execute of not
convert const column, test about could faster 1s on 1000w table rows
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 08:03:48 +08:00
a5c8ed1cde
[branch2.1][fix](cache) Catch the directory_iterator's error_code ( #39922 )
...
## Proposed changes
Catch the directory_iterator's error_code to avoid exceptions causing
core dump
2024-08-27 08:00:52 +08:00
a6e485e54f
[fix](followup) Fix wrong partition num ( #39917 )
2024-08-26 18:25:36 +08:00
357394bb3e
[branch-2.1]Reset io limit default value ( #39898 )
...
pick #39842
2024-08-26 14:27:35 +08:00
5acd1279a9
[fix](2.1) Fix correctness in branch-2.1 ( #39901 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-26 14:12:59 +08:00
4c1c67e03a
[improvemen](overflow) Provide the user with a suggestion to avoid th… ( #39631 ) ( #39897 )
...
cherry-pick #39631 to branch-2.1
2024-08-26 08:10:32 +08:00
f2a37d58fb
[fix](stat) handle overflow of memory stat if load failed ( #39621 ) ( #39887 )
...
## Proposed changes
pick #39621
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-25 18:24:08 +08:00
8dbd73988a
[fix](recvr) catch exception of transmit_block ( #39882 )
...
BP #39881
2024-08-25 00:25:20 +08:00
2dea859bdb
[debug](rpc) debug rpc time consumption problem ( #39852 )
...
## Proposed changes
Issue Number: close #xxx
Add detail RPC time info for each channel, sorted by max rpc time of
channels:
```
DATA_STREAM_SINK_OPERATOR (id=1,dst_id=1):
- Partitioner: Crc32HashPartitioner(64)
- BlocksProduced: 74
- BrpcSendTime: 2.689us
- BrpcSendTime.Wait: 0ns
- BytesSent: 89.35 KB
- CloseTime: 680.152us
- CompressTime: 0ns
- ExecTime: 160.663ms
- InitTime: 263.608us
- InputRows: 32.512K (32512)
- LocalBytesSent: 0.00
- LocalSendTime: 0ns
- LocalSentRows: 0
- MemoryUsage:
- PeakMemoryUsage: 80.00 KB
- MergeBlockTime: 0ns
- OpenTime: 4.113ms
- OverallThroughput: 0.0 /sec
- PendingFinishDependency: 41.179ms
- RowsProduced: 32.512K (32512)
- RpcAvgTime: 11.850ms
- RpcCount: 10
- RpcMaxTime: 86.891ms
- RpcMinTime: 15.200ms
- RpcSumTime: 118.503ms
- SerializeBatchTime: 13.517ms
- SplitBlockDistributeByChannelTime: 38.923ms
- SplitBlockHashComputeTime: 2.659ms
- UncompressedRowBatchSize: 135.19 KB
- WaitForDependencyTime: 0ns
- WaitForRpcBufferQueue: 0ns
RpcInstanceDetails:
- Instance 85d4f75b72a9ea61: Count: 4, MaxTime: 36.238ms, MinTime: 12.107ms, AvgTime: 21.722ms, SumTime: 86.891ms
- Instance 85d4f75b72a9ea91: Count: 3, MaxTime: 11.107ms, MinTime: 2.431ms, AvgTime: 5.470ms, SumTime: 16.412ms
- Instance 85d4f75b72a9eac1: Count: 3, MaxTime: 7.554ms, MinTime: 3.160ms, AvgTime: 5.066ms, SumTime: 15.200m
```
2024-08-24 19:59:39 +08:00
5a810122a2
[debug](load) check the column type when string column is invalid ( #39337 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-24 18:14:21 +08:00
263746b04b
[fix](paimon) fix crash when enable cache with paimon deletion vector( #39877 ) ( #39875 )
...
bp #39877
2024-08-24 17:58:20 +08:00
14a2a66106
[fix](paimon) fix not able to read paimon data from hdfs with HA ( #39806 ) ( #39876 )
...
bp #39806
2024-08-24 17:51:15 +08:00
460605ae3c
[branch-2.1] pick some prs ( #39860 )
...
## Proposed changes
Issue Number: close #xxx
https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
2024-08-24 17:26:42 +08:00
564d3cd647
[Performance](opt) opt the order by performance in permutation ( #39089 )
...
## Proposed changes
Issue Number: cherry pick #38985
<!--Describe your changes.-->
2024-08-24 16:05:46 +08:00
a6f267c479
[pick](Variant) fix element_at should return nullable if result type is nullable ( #39846 )
...
#39732
2024-08-24 09:22:03 +08:00
ae4d747c13
[branch-2.1](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes ( #39834 )
...
pick #39611
2024-08-24 09:21:35 +08:00
8cf6c6a2b5
[fix](agg function) incorrect result of map agg( #39743 ) ( #39854 )
...
## Proposed changes
pick #39743
2024-08-24 09:19:52 +08:00
3103bb08dc
[pick](Variant) casting to decimal type may lost precision ( #39843 )
...
#39650
2024-08-23 22:47:32 +08:00
37443aa7e1
[improve](move-memtable) reuse connection in load_stream_stub ( #39231 ) ( #39762 )
...
backport #39231
2024-08-23 22:46:28 +08:00
5b124a03ba
[enhancement](err-msg) Add detailed column and schema info when failed to create a column iterator ( #38689 ) ( #39861 )
...
As title.
2024-08-23 21:39:19 +08:00
6ceb574aa0
[branch-2.1]Pick IO limit/workload group usage table ( #39839 )
2024-08-23 18:51:47 +08:00
baf5b71b39
[branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache ( #39829 )
...
pick #38185
2024-08-23 17:21:42 +08:00
c40246efa9
[bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 ( #39808 )( #39569 ) ( #39832 )
...
## Proposed changes
bp: #39808 #39569
2024-08-23 17:19:48 +08:00
7a7292ad5a
[branch-2.1][Refactor]use async to get be resource ( #38389 ) ( #39826 )
...
pick #38389
2024-08-23 17:16:19 +08:00
e03b887a97
[opt](MultiCast) Avoid copying while holding a lock ( #37462 ) ( #39816 )
...
Previously, copying was done while holding a lock; Now, get block while
holding the lock and then copy
https://github.com/apache/doris/pull/37462
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-23 17:15:34 +08:00
0934fbee7e
[improvement](query) prefer to chose tablet on alive disk #39467 ( #39654 )
...
cherry pick from #39467
2024-08-23 12:23:12 +08:00
1f16daa5f6
Revert "[bugfix](iceberg)clear block for partition values for 2.1 ( #39569 )" ( #39815 )
...
Reverts apache/doris#39729
2024-08-23 11:58:42 +08:00
9d5468d198
[branch-2.1](memory) BE memory info compatible with CgroupV2 ( #39799 )
...
pick #39256
2024-08-23 02:03:00 +08:00
1367f74e7a
[branch-2.1](memory) Optimize ClearCacheActionimplementation ( #39796 )
...
pick #38438
2024-08-23 01:51:14 +08:00
0f8bd33077
[fix](scan) fix predicate contains cast that results in null, the pr… ( #39809 )
...
…edicate will be miss. (#39550 )
https://github.com/apache/doris/pull/39550
```
drop table datetest;
create table datetest (
id int,
dt date
)
DUPLICATE key (id)
distributed by hash(id) buckets 1
properties(
"replication_num" = "1"
);
insert into datetest values (1, '2024-01-01');
mysql [test10]>select dt from datetest WHERE dt = 1 ;
+------------+
| dt |
+------------+
| 2024-01-01 |
+------------+
```
now
```
mysql [test10]>select dt from datetest WHERE dt = 1 ;
Empty set (0.16 sec)
```
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-23 01:46:22 +08:00
dc732fe33f
[bugfix](iceberg)clear block for partition values for 2.1 ( #39569 ) ( #39729 )
...
## Proposed changes
bp: #39569
clear block, or we will get wrong partition values.
2024-08-22 22:43:02 +08:00
06a0b35704
[chore] Comment for tv_nsec ( #39752 )
...
just comment.
2024-08-22 22:16:21 +08:00
04e993c1de
[refine](pipeline) refine some VDataStreamRecvr code ( #35063 ) ( #37802 )
...
## Proposed changes
https://github.com/apache/doris/pull/35063
https://github.com/apache/doris/pull/35428
2024-08-22 19:55:17 +08:00