Commit Graph

7567 Commits

Author SHA1 Message Date
08fb91f161 Revert "[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value" (#40359)
Reverts apache/doris#40283
2024-09-04 17:55:25 +08:00
b4beec8ea8 [fix](OrcWriter) fix be core when upgrading BE without upgrading FE (#40303)
bp: #40282
2024-09-04 10:24:41 +08:00
db06c678a3 [Fix](planner) legacy planner repeat group by has grouping report error (#40281) (#40305)
cherry-pick #40281 to branch-2.1
2024-09-04 08:50:21 +08:00
3cf5d15b49 [fix](fe) avoid setting the value of batch_size excessively high (#35941) (#40312)
Issue Number: close #xxx

pick #35941

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-09-03 19:49:26 +08:00
c49b345342 [cherry-pick](branch-2.1) Fix enable_mow_light_delete default value (#40283)
## Proposed changes

Pick #40104 #40197

<!--Describe your changes.-->
2024-09-03 11:19:05 +08:00
92752b90e7 [feature](metacache) add system table catalog_meta_cache_statistics #40155 (#40210)
bp #40155
2024-09-02 23:23:35 +08:00
1527099e35 [opt](Nereids) forbid one step topn with distribute as child (#40066) (#40278)
pick from master #40066
2024-09-02 22:26:26 +08:00
a5efdc6085 [improve](statistics)Clean expired TableStatsMeta. (#39779) (#40253)
backport: https://github.com/apache/doris/pull/39779
2024-09-02 22:13:42 +08:00
8ff068eb7c [feature](statistics) Support get row count for JDBC external table. (#38889) (#40248)
backport: https://github.com/apache/doris/pull/38889
2024-09-02 16:39:42 +08:00
98e039ae31 [pick](Variant) truncate key when column is variant type when crea… (#40238)
#39988
2024-09-02 13:54:04 +08:00
5db2c8b8bc [branch-2.1](arrow-flight-sql) Fix return result from FE Arrow Flight server error 0.0.0.0:xxx, connection refused (#40229)
pick #40002
2024-09-02 09:45:27 +08:00
2a12251b83 [branch-2.1][Feat](Job)After a job is paused, it can be manually triggered to execute. (#40180)
…

pick (#39565)
2024-08-31 19:24:53 +08:00
32a32808a7 [improvement](nereids) support convert_tz in partition prune (#40047) (#40127)
cherry-pick #40047 to branch-2.1
2024-08-30 13:39:37 +08:00
80508fb595 [fix](restore) Reset next version for remote table when restore #40118 (#40165)
cherry pick from #40118

Co-authored-by: smallx <e9999e@163.com>
2024-08-30 13:22:23 +08:00
1288676b68 [fix](nereids)change lag/lead function signature (#40161)
## Proposed changes

pick from master https://github.com/apache/doris/pull/40060

<!--Describe your changes.-->
2024-08-30 12:05:47 +08:00
163193b1d4 [branch-2.1](function) fix random_bytes return same data for multi rows (#39891) (#40137)
pick https://github.com/apache/doris/pull/39891

Issue Number: close #xxx

before:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a                      |
+------------------------+
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
+------------------------+
```

now:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a                      |
+------------------------+
| 0xd82cf60825b29ef2a0fd |
| 0x6f8c808415bdbaa6d257 |
| 0x7c26b5214297a151c25c |
| 0x43f02c77293063900437 |
| 0x5e5727569dec5e24f96b |
| 0x434f20bf74d7759640b7 |
| 0x087ed96b739750c733a6 |
| 0xdf05f6d7ede4972eb846 |
| 0xcefab471912264b5c54f |
| 0x1bddc019409d1926aa10 |
+------------------------+
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-30 10:43:42 +08:00
e4d543f060 [branch-2.1](load) fix commit txn timeout when loading to table with many tablet (#40031) (#40139)
pick https://github.com/apache/doris/pull/40031
2024-08-30 10:34:43 +08:00
70daa1f85d [opt](inverted index) Controls whether the in_list can execute fast_execute. (#40141)
https://github.com/apache/doris/pull/40022
2024-08-30 10:32:43 +08:00
ca07a00c93 Revert "[branch-2.1](hive) support hive write text table (#38549) (#4… (#40157)
…0063)"

This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-08-30 10:25:38 +08:00
1ecc85b575 [Fix](ctas) forward the use_max_length_of_varchar_in_ctas (#39705) (#40138)
cherry-pick from master #39705
2024-08-30 10:11:29 +08:00
0b9817ea4b [cherry-pick](branch-2.1) Pick "[Enhancement](txn) Block new insert into if schema change happens during transaction (#39483)" (#40115)
## Proposed changes

Pick #39483 

<!--Describe your changes.-->
2024-08-30 10:01:10 +08:00
45e35e8a99 [improvement](mtmv) Optimize get available mv logic to avoid unnecessary rewrite (#39734) (#40136)
## Proposed changes

commitId: 55ed1277
pr: https://github.com/apache/doris/pull/39734
2024-08-30 08:02:27 +08:00
e072ce7f68 [fix](nereids) Remove Nondeterministic to avoid ambiguity when decide a expression is nondeterministic or not (#39801) (#40130)
## Proposed changes

commitId: 2b715a7e
pr: https://github.com/apache/doris/pull/39801
2024-08-30 07:51:42 +08:00
b86e4e8498 [fix](profile) fix possible coredump of rpc verbose profile (#40117)
## Proposed changes

Issue Number: close #xxx

`_instance_to_rpc_stats_vec` may be updated when sorting in
`ExchangeSinkBuffer<Parent>::update_profile`, which may cause coredump.
2024-08-29 23:54:52 +08:00
82fb747e6f [fix](Nereids) datetimev2 literal equals should compare microsecond (#40121) (#40150)
pick from master #40121
2024-08-29 23:41:51 +08:00
c6df7c21a3 [branch-2.1](hive) support hive write text table (#38549) (#40063)
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd

pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
2054cc9fc9 [fix](nereids) fix union all instance number (#40099)
pick from https://github.com/apache/doris/pull/39999

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
2024-08-29 16:47:55 +08:00
eb6a040a03 [branch-2.1] Picks "[Fix](partial update) Fix wrongly update autoinc column in partial update #39996" (#40073)
## Proposed changes

pick https://github.com/apache/doris/pull/39996
2024-08-29 16:41:51 +08:00
131238ff71 [fix](file-cache) change metric_value column in file_cache_statistics table to string (#40083)
Make it more flexible
followup #39552
2024-08-29 16:39:22 +08:00
7155711431 [cherry-pick](branch-2.1) Improve local shuffle strategy (#40030)
pick #34122 #35454 #35716 #37195
2024-08-29 14:16:16 +08:00
34d95cef1f [improve](ES Catalog)Only push down literal expr in binary predicate(#39845) (#39847)
## Proposed changes

bp #39845
2024-08-29 14:07:17 +08:00
93da0ebaf4 [chore](backup) limit the involved tablets in a backup job #39987 (#40080)
cherry pick from #39987
2024-08-29 12:03:14 +08:00
c5d9e178be [fix](mtmv) Fix result wrong when query rewrite by mv if query contains null_unsafe equals expression (#39629) (#40041)
## Proposed changes

commitId: 5d4ad028
pr: https://github.com/apache/doris/pull/39629
2024-08-29 00:31:36 +08:00
688b97e183 [enhancement](Nereids) support reuse sql cache between different comment (#40065)
cherry pick from #40049
2024-08-28 22:15:55 +08:00
c9aca7d852 [opt](profile) Add ExecutedByFrontend in profile #39942 (#40012)
cherry pick from #39942
2024-08-28 22:06:57 +08:00
a32a2d8bb4 [bugfix](iceberg)Restrictions on creating a database (#39641) (#40051)
bp #39641

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-08-28 21:45:03 +08:00
d1c3b857cf [fix](catalog) fix invalid getPartitionNames() methods (#39878) (#40054)
bp #39878
2024-08-28 21:44:30 +08:00
6ef1211585 [bugfix](paimon)Use the fs cache, otherwise it will cause OOM (#38552) (#40052)
bp #38552

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-08-28 21:42:36 +08:00
b052f46944 [fix](json-quote) fix json quote func for not find the func (#39931) (#40061)
1.  fix function not found before this pr:
```
drop table if exists t003;
create table t003 (a bigint, b json not null) properties ("replication_num"="1");
insert into t003 values (1, '{"a":1,"b":2}');
select a, map_agg("k1", json_quote(b)) from t003 group by a;

[17:47]>  select a, map_agg("k1", json_quote(b)) from t003 group by a;
(1105, 'errCode = 2, detailMessage = (172.20.48.119)[INTERNAL_ERROR]Function json_quote get failed, expr is VectorizedFnCall[json_quote](arguments=(CAST b(JSONB) TO String),return=Nullable(String)) and return type is Nullable(String).')
``` 
after pr , we can carry on it 
2. fix a core if we use json_quote with table column
```
[WARNING!] /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never, Doris not recommend turning on THP, which may cause the BE process to use more memory and cannot be freed in time. Turn off THP: `echo madvise | sudo tee /sys/kernel/mm/transparent_hugepage/enabled`
start BE in local mode
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
*** Query id: b7b94dead55e4090-8c8e94f65ec9efd3 ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1724744881 (unix time) try "date -d @1724744881" if you are using GNU date ***
*** Current BE git commitID: bd5844ea0d ***
*** SIGABRT unknown detail explain (@0x461003611e1) received by PID 3543521 (TID 3548858 OR 0x7f8a53c1a700) from PID 3543521; stack trace: ***
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
doris_be: /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488:void rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<>>>::Prefix(Type) [OutputStream = rapidjson::GenericStringBuffer<rapidjson::UTF8<>>, SourceEncoding = rapidjson::UTF8<>, TargetEncoding = rapidjson::UTF8<>, StackAllocator = rapidjson::CrtAllocator, writeFlags = 0]: 假设 ‘!hasRoot_’ 失败。
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
 1# 0x00007F9429595B50 in /lib64/libc.so.6
 2# gsignal in /lib64/libc.so.6
 3# __GI_abort in /lib64/libc.so.6
 4# _nl_load_domain.cold.0 in /lib64/libc.so.6
 5# 0x00007F942958E426 in /lib64/libc.so.6
 6# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::Prefix(rapidjson::Type) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:488
 7# rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>::String(char const*, unsigned int, bool) at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/writer.h:206
 8# bool rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> >::Accept<rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u> >(rapidjson::Writer<rapidjson::GenericStringBuffer<rapidjson::UTF8<char>, rapidjson::CrtAllocator>, rapidjson::UTF8<char>, rapidjson::UTF8<char>, rapidjson::CrtAllocator, 0u>&) const at /mnt/disk1/wangqiannan/amory/doris/thirdparty/installed/include/rapidjson/document.h:1974
 9# doris::vectorized::FunctionJsonQuoteImpl::execute(std::vector<doris::vectorized::ColumnStr<unsigned int> const*, std::allocator<doris::vectorized::ColumnStr<unsigned int> const*> > const&, doris::vectorized::ColumnStr<unsigned int>&, unsigned long) at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:826
10# doris::vectorized::FunctionJson<doris::vectorized::FunctionJsonQuoteImpl>::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function_json.cpp:985
11# doris::vectorized::DefaultExecutable::execute_impl(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.h:463
12# doris::vectorized::PreparedFunctionImpl::_execute_skipped_constant_deal(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:120
13# doris::vectorized::PreparedFunctionImpl::execute_without_low_cardinality_columns(doris::FunctionContext*, doris::vectorized::Block&, std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long, unsigned long, bool) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/functions/function.cpp:245
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-28 21:31:08 +08:00
b3b085c561 [fix](statistics)Use utf-8 charset for internal query result. (#39989) (#40026)
backport: https://github.com/apache/doris/pull/39989
2024-08-28 16:24:34 +08:00
42f30d4a48 [branch-2.1][Bug] test suite test_table_options may cause NPE in FE #39457 (#39509) (#40018)
## Proposed changes

pick #39509

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
2024-08-28 14:17:00 +08:00
34d80f6239 [fix](group commit) Fix group commit error log when decommission (#39899) (#39955)
pick https://github.com/apache/doris/pull/39899
2024-08-27 18:33:24 +08:00
d70fe130d7 [fix](explode-map) fix explode_map with alias (#39972)
## Proposed changes
backport: https://github.com/apache/doris/pull/39757
Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-27 18:32:10 +08:00
173aafc86f [Enhancement] add information_schema.table_properties #38745 (#38746) (#39886)
bp #38746

---------

Co-authored-by: Vallish Pai <vallishpai@gmail.com>
2024-08-27 17:22:19 +08:00
bbf7701046 [fix](catalog) hide some properties in show create catalog stmt (#39946) (#39970)
bp #39946
2024-08-27 16:46:12 +08:00
bf7675dc94 [Pick](PreparedStatement) allow prepare mysql command in Nerids even … (#39960)
…if enableServeSidePreparedStatement is false to avoid fallback to
legacy planner

backport #39465
2024-08-27 14:58:28 +08:00
4cf769b39f [Improve](table-function) explode json array with json args (#39491) 2024-08-27 14:53:17 +08:00
a4c5c03acf [fix](mtmv) Add debug log decide for performance when query rewrite by materialized view (#39914) (#39952)
## Proposed changes
In method `AbstractMaterializedViewRule#isMaterializationValid` Should
add `LOG.isDebugEnabled()` before print debug log. Because
`Plan#treeString` in debug log is performance consume.

## Proposed changes

pr: https://github.com/apache/doris/pull/39914 
commitId: 45ddb8ce
2024-08-27 14:27:44 +08:00
959ea278cb [fix](replay) fix replay ReplicaPersistInfo no update lastFailedVersion and lastSuccVersion #39918 (#39947)
cherry pick from #39918
2024-08-27 11:58:13 +08:00
db0724dfe0 [Fix-2.1](function) fix function covar core for not null input (#39943)
## Proposed changes

Issue Number: close #xxx

add testcases like:
```groovy
    qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
    qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
    qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```

before they will all coredump in 2.1
2024-08-27 08:39:47 +08:00