a32a2d8bb4
[bugfix](iceberg)Restrictions on creating a database ( #39641 ) ( #40051 )
...
bp #39641
Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com >
2024-08-28 21:45:03 +08:00
b052f46944
[fix](json-quote) fix json quote func for not find the func ( #39931 ) ( #40061 )
...
1. fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;
[17:47]> select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
```
after pr , we can carry on it
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
1# 0x00007F9429595B50 in /lib64/libc.so.6
2# gsignal in /lib64/libc.so.6
3# __GI_abort in /lib64/libc.so.6
4# _nl_load_domain.cold.0 in /lib64/libc.so.6
5# 0x00007F942958E426 in /lib64/libc.so.6
6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-28 21:31:08 +08:00
799ad1d398
[fix](mv) Fix test create table like test unstable ( #39305 ) ( #40013 )
...
ddl_p0/test_create_table_like_nereids.groovy test result is unstable.
Should wait util rollup join finished then we can compare the explain
rollup result.
## Proposed changes
commitId: 3b7b29b8
pr: https://github.com/apache/doris/pull/39305
2024-08-28 18:42:35 +08:00
fa4d885487
[fix](regression) fix compaction_width_array_column case ( #37664 ) ( #40045 )
...
pick master #37664
2024-08-28 18:13:42 +08:00
ddf72ce09a
[Fix](branch-2.1) fix manual pick with mistake when handling decimal … ( #40008 )
...
…type
introduced by #39843
2024-08-28 10:15:52 +08:00
4127eec9a7
[fix](function) fix error return type in mod(float32,BigInt) ( #39358 ) ( #39971 )
...
## Proposed changes
https://github.com/apache/doris/pull/39358
```
CREATE TABLE testdb (
K1 BIGINT,
K2 FLOAT
) properties("replication_num" = "1");
insert into testdb values(1,1.1);
select mod(k1,k2) from testdb;
mysql [test10]>select mod(k1,k2) from testdb;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Function mod get failed, expr is VectorizedFnCall[mod](arguments=K1, K2,return=Nullable(Float32)) and return type is Nullable(Float32).
```
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 18:32:52 +08:00
d70fe130d7
[fix](explode-map) fix explode_map with alias ( #39972 )
...
## Proposed changes
backport: https://github.com/apache/doris/pull/39757
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 18:32:10 +08:00
fc9936d923
[fix](funcs) fix map struct construct funcs ( #39973 )
...
## Proposed changes
backport: https://github.com/apache/doris/pull/39699
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 18:31:51 +08:00
da292b6908
[Fix-2.1](testcase) fix an unstable semi-structure case result ( #39981 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 18:31:32 +08:00
173aafc86f
[Enhancement] add information_schema.table_properties #38745 ( #38746 ) ( #39886 )
...
bp #38746
---------
Co-authored-by: Vallish Pai <vallishpai@gmail.com >
2024-08-27 17:22:19 +08:00
bf7675dc94
[Pick](PreparedStatement) allow prepare mysql command in Nerids even … ( #39960 )
...
…if enableServeSidePreparedStatement is false to avoid fallback to
legacy planner
backport #39465
2024-08-27 14:58:28 +08:00
4cf769b39f
[Improve](table-function) explode json array with json args ( #39491 )
2024-08-27 14:53:17 +08:00
dd509d0d35
[cherry-pick](branch-2.1)fix regression test fail ( #39951 )
...
## Proposed changes
Issue Number: close #xxx
cherry-pick master #39935
<!--Describe your changes.-->
2024-08-27 13:20:31 +08:00
959ea278cb
[fix](replay) fix replay ReplicaPersistInfo no update lastFailedVersion and lastSuccVersion #39918 ( #39947 )
...
cherry pick from #39918
2024-08-27 11:58:13 +08:00
db0724dfe0
[Fix-2.1](function) fix function covar core for not null input ( #39943 )
...
## Proposed changes
Issue Number: close #xxx
add testcases like:
```groovy
qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```
before they will all coredump in 2.1
2024-08-27 08:39:47 +08:00
21bd4a4ac8
[bug](function)fix json_replace check return type error ( #37014 ) ( #39938 )
...
1. fix the return type dcheck error:
```
mysql [test]>select (json_replace(a, '$.fparam.nested_2', "qwe")) from json_table_2 limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]Function json_replace get failed, expr is VectorizedFnCall[json_replace](arguments=a, String, String, String,return=Nullable(String)) and return type is Nullable(String).
```
2. improve the json_replace/json_insert/json_set function execute of not
convert const column, test about could faster 1s on 1000w table rows
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-27 08:03:48 +08:00
44c565840d
[fix] (regression test) fix error tests ( #39907 )
...
## Proposed changes
fix error tests: `test_dup_table_inverted_index`,
`test_single_compaction_with_variant_inverted`
2024-08-26 17:48:52 +08:00
9412ba56ae
[fix](restore) Reset the db name of the materialized index stmt #39710 ( #39848 )
...
cherry pick from #39710 , #39855
2024-08-26 16:51:50 +08:00
e21d2b82d6
[chore](backup) Fix the db name of the restored view ( #38072 )
...
Cherry-pick #37412 , #39855
Previously, during restore, the database name in the CREATE VIEW
statement was not modified, causing the restored view to be unviewable
with the SHOW VIEW command. This PR retains the original cluster's
database name in the BackupMeta and manually replaces it with the new
cluster's database name in the CREATE VIEW statement during restore.
2024-08-26 14:12:47 +08:00
d87a220d2b
[opt](privilege) Grant check name ( #39597 ) ( #39856 )
...
pick https://github.com/apache/doris/pull/39597
2024-08-26 09:53:45 +08:00
f7b6ed772b
[Fix-2.1](testcase) Fix some testcases unstable results due to lack of orderby ( #39892 )
...
## Proposed changes
Issue Number: close #xxx
some sql's order-by column has same value, add more order-by column to
fix it.
2024-08-25 08:45:36 +08:00
9997911ec9
[feat](Nereids) support nereids hint position detaction ( #39113 ) ( #39417 )
...
cherry-pick: #39113
When use hint in wrong position or use unsupport hint, use channel(2) to
filter it out
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-24 23:59:54 +08:00
263746b04b
[fix](paimon) fix crash when enable cache with paimon deletion vector( #39877 ) ( #39875 )
...
bp #39877
2024-08-24 17:58:20 +08:00
460605ae3c
[branch-2.1] pick some prs ( #39860 )
...
## Proposed changes
Issue Number: close #xxx
https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
2024-08-24 17:26:42 +08:00
de2e8f0ae6
[fix](ctas) fix NPE when ctas with old planner and varchar issue ( #39744 ) ( #39871 )
...
bp #39744
2024-08-24 09:24:47 +08:00
b9da934b16
[fix](hive) report error with escape char and null format ( #39700 ) ( #39869 )
...
bp #39700
Co-authored-by: Socrates <suxiaogang223@icloud.com >
2024-08-24 09:23:03 +08:00
a6f267c479
[pick](Variant) fix element_at should return nullable if result type is nullable ( #39846 )
...
#39732
2024-08-24 09:22:03 +08:00
8cf6c6a2b5
[fix](agg function) incorrect result of map agg( #39743 ) ( #39854 )
...
## Proposed changes
pick #39743
2024-08-24 09:19:52 +08:00
9d597bde68
[fix](Nereids) producer to consumer should be multimap in cte ( #39850 ) ( #39867 )
2024-08-23 23:25:11 +08:00
508c7a7040
[fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. ( #39239 ) ( #39865 )
...
bp #39239
Co-authored-by: daidai <2017501503@qq.com >
2024-08-23 23:21:02 +08:00
3103bb08dc
[pick](Variant) casting to decimal type may lost precision ( #39843 )
...
#39650
2024-08-23 22:47:32 +08:00
19c51cc217
[fix](mtmv) use name instead of id in meta of MTMV ( #39355 ) ( #39748 )
...
pick from master #39355
2024-08-23 20:40:54 +08:00
6ceb574aa0
[branch-2.1]Pick IO limit/workload group usage table ( #39839 )
2024-08-23 18:51:47 +08:00
e716658fba
[branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections ( #39836 )
...
pick #39127
pick #39802
2024-08-23 17:27:34 +08:00
424ad2384a
[opt](nereids) refine left semi/anti cost under short-cut opt ( #39636 )
...
## Proposed changes
pick from https://github.com/apache/doris/pull/37951
---------
Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com >
2024-08-23 17:26:56 +08:00
baf5b71b39
[branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache ( #39829 )
...
pick #38185
2024-08-23 17:21:42 +08:00
c40246efa9
[bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 ( #39808 )( #39569 ) ( #39832 )
...
## Proposed changes
bp: #39808 #39569
2024-08-23 17:19:48 +08:00
8f15efdbb8
[cherry-pick](branch-2.1) fix delete random distributed tbl ( #39830 )
...
## Proposed changes
cherry-pick #37985
<!--Describe your changes.-->
2024-08-23 17:17:05 +08:00
67a8099991
[fix](multi-catalog)fix max compute array and map type read offset ( #39822 )
...
bp #39680
2024-08-23 16:53:52 +08:00
e0b65d0ff4
[fix](regression) Fix creating db for downstream url #39601 ( #39818 )
...
cherry pick from #39601
2024-08-23 16:15:19 +08:00
1f16daa5f6
Revert "[bugfix](iceberg)clear block for partition values for 2.1 ( #39569 )" ( #39815 )
...
Reverts apache/doris#39729
2024-08-23 11:58:42 +08:00
6c10c47f79
[fix](fe) LIST partition table support modify default bucket num ( #39688 )
...
## Proposed changes
bp #39696
Issue Number: close #39684
```sql
CREATE TABLE `test1` (
`id1` VARCHAR(255) NULL COMMENT 'id1',
`id2` VARCHAR(255) NULL COMMENT 'id2',
`event_time` VARCHAR(255) NULL COMMENT '事件时间',
`event_date` VARCHAR(255) NULL COMMENT '事件日期',
`event_ts` VARCHAR(256) NULL COMMENT '事件发生时间戳(毫秒)',
`dt` VARCHAR(255) NOT NULL COMMENT '日期分区',
`hr` VARCHAR(255) NOT NULL COMMENT '小时分区'
) ENGINE = OLAP DUPLICATE KEY(`id1`) COMMENT 'xxx' PARTITION BY LIST(`dt`, `hr`) (
PARTITION p2024082021 VALUES IN (("2024-08-20", "21"))
) DISTRIBUTED BY HASH(`dt`, `hr`) BUCKETS 2 PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"min_load_replica_num" = "-1",
"is_being_synced" = "false",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V1",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false",
"group_commit_interval_ms" = "10000",
"group_commit_data_bytes" = "134217728"
);
```
1. 修改前表已有分区的bucket num为2

2. 修改List分区表bucket num(从2 -> 4)

3. 修改List分区后,添加的新分区bucket num为4

Co-authored-by: tongyang.han <tongyang.han@jiduauto.com >
2024-08-23 11:52:16 +08:00
40a58b9e42
[branch-2.1][regression test](jdbc catalog) Enable CLICKHOUSE_ALWAYS_RUN_INITDB_SCRIPTS for clickhouse docker ( #39667 )
...
pick (#39425 ) #39693
2024-08-23 09:59:03 +08:00
0f8bd33077
[fix](scan) fix predicate contains cast that results in null, the pr… ( #39809 )
...
…edicate will be miss. (#39550 )
https://github.com/apache/doris/pull/39550
```
drop table datetest;
create table datetest (
id int,
dt date
)
DUPLICATE key (id)
distributed by hash(id) buckets 1
properties(
"replication_num" = "1"
);
insert into datetest values (1, '2024-01-01');
mysql [test10]>select dt from datetest WHERE dt = 1 ;
+------------+
| dt |
+------------+
| 2024-01-01 |
+------------+
```
now
```
mysql [test10]>select dt from datetest WHERE dt = 1 ;
Empty set (0.16 sec)
```
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-23 01:46:22 +08:00
dc732fe33f
[bugfix](iceberg)clear block for partition values for 2.1 ( #39569 ) ( #39729 )
...
## Proposed changes
bp: #39569
clear block, or we will get wrong partition values.
2024-08-22 22:43:02 +08:00
40500f2b10
[branch-2.1](thirdparty) upgrade arrow to 17.0.0 ( #39773 )
...
pick #38572
2024-08-22 18:35:32 +08:00
ba7baa7e6b
[fix](window_funnel) fix upgrading problem caused by behaviour change of window_funnel ( #39766 )
...
## Proposed changes
Issue Number: close #xxx
For the latest published 2.1 version `2.1.5`:
```
max_be_exec_version=4;
AGG_FUNCTION_NEW=2;
```
and `branch-2.1`:
```
max_be_exec_version=5;
AGG_FUNCTION_NEW=2;
```
It will cause problem when upgrading.
This PR fix the problem, set `AGG_FUNCTION_NEW` to `5`.
2024-08-22 17:26:51 +08:00
f553645a71
[fix](mtmv) transfer col in mysql varchar to text when create MTMV ( #37668 ) ( #39727 )
...
pick from master #37668
2024-08-22 15:20:59 +08:00
10f3e88f7a
[fix](nereids) fix distribution expr list ( #39435 )
...
pick from #39148
2024-08-22 15:19:51 +08:00
1c566253a8
[Pick][Improment]Query queued by be memory ( #37559 ) ( #39733 )
...
pick #37559
2024-08-22 15:14:47 +08:00