fc0b4b9ced
[fix](explain) fix NPE when explain verbose with partition batch mode ( #40969 ) ( #41231 )
...
bp #40969
2024-09-24 21:10:13 +08:00
5c5251405a
[improvement](statistics)Remove analyze retry logic. ( #41220 )
...
Remove analyze retry logic when task failed. Because usually retry would
fail again and retry would bring a long time of sleep, which cause the
analyze job running too slow.
Master pr: https://github.com/apache/doris/pull/33703
2024-09-24 20:26:01 +08:00
8bb57bcc3e
[fix] (inverted index) fix the error in the query result when using count on index ( #41200 )
...
## Proposed changes
Introduced by #39473
<!--Describe your changes.-->
2024-09-24 19:47:18 +08:00
78adf64353
[compability](variant) fix nullable serde during upgrading to 3.0 ( #41191 )
...
need to check nullable during serialization when upgrading to 3.0
2024-09-24 19:44:54 +08:00
d4c1b39d03
[fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables ( #39992 ) ( #41131 )
...
pick (#39992 )
BE node was killed by OOM-killer when use multi table load under high
concurrency with a large number of tables(128 concurrency and every
concurrency load 200 tables).
This pr restricts the multi tables load memory under this issue. If
memory reaches hard limit, new task will be rejected and return
directly.
2024-09-24 16:34:32 +08:00
3ad9dce067
[case](mtmv) Failed to fix the case of concurrent insert overwrite ex… ( #41163 )
...
…ecution (#41125 )
pick: https://github.com/apache/doris/pull/41125
2024-09-24 16:04:30 +08:00
a1429efc41
[fix](routine load) write edit log when rescheduled job ( #40728 ) ( #41134 )
...
pick (#40728 )
```
2024-09-11 20:00:53,079 ERROR (replayer|105) [RoutineLoadManager.replayChangeRoutineLoadJob():836] should not happened
org.apache.doris.common.DdlException: errCode = 2, detailMessage = Could not transform PAUSED to PAUSED
at org.apache.doris.load.routineload.RoutineLoadJob.checkStateTransform(RoutineLoadJob.java:855) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.load.routineload.RoutineLoadJob.unprotectUpdateState(RoutineLoadJob.java:1407) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.load.routineload.RoutineLoadJob.updateState(RoutineLoadJob.java:1394) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.load.routineload.RoutineLoadManager.replayChangeRoutineLoadJob(RoutineLoadManager.java:834) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:717) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env.replayJournal(Env.java:2913) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.Env$4.runOneCycle(Env.java:2675) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT]
```
`unprotectNeedReschedule()` will change job state to
`JobState.NEED_SCHEDULE` without `logOpRoutineLoadJob`.If job is paused
then rescheduled and paused finally, the record of two consecutive edit
logs will be 'PAUSED', the correct
replay sequence should be: `PAUSED` -> `NEED_SCHEDULE` ->` PAUSED`.
Therefore, it is need to write edit log when rescheduled job.
2024-09-24 16:03:55 +08:00
336a658f9a
[Fix](test)Fixing Kerberos Principal Not Mapping Correctly to a Local User ( #40905 ) ( #41128 )
...
## Proposed changes
error log
http://43.132.222.7:8111/buildConfiguration/Doris_External_Regression/529960?buildTab=tests&status=muted&focusLine=NaN&suite=&package=external_table_p0.kerberos&class=test_single_hive_kerberos&expandedTest=build%3A%28id%3A529960%29%2Cid%3A2000000060
## Changes
- If there are multiple realms, it's best to configure
hadoop.security.auth_to_local to correctly map Kerberos Principals to
local users.
- Thrift exceptions should use those from hive-shade, but to avoid
future changes in hive-shade, I omitted part of the package name prefix.
(cherry picked from commit 902cf1e913f1b01dccb5978e9c9d521a2e8c9350)
#40905
2024-09-24 16:02:51 +08:00
4237341e0d
[fix](move-memtable) do not execute close if create rowset failed when loading MOW table ( #40105 ) ( #41132 )
...
pick (#40105 )
Core dump happened when load to MOW table:
```
Check failure stack trace: ***
@ 0x55fae437d246 google::LogMessage::SendToLog()
@ 0x55fae4379c90 google::LogMessage::Flush()
@ 0x55fae437da89 google::LogMessageFatal::~LogMessageFatal()
@ 0x55faacf26bbf doris::BaseTablet::check_delete_bitmap_correctness()
@ 0x55fab05049ef doris::RowsetBuilder::commit_txn()
@ 0x55fab09026e8 doris::LoadStreamWriter::close()
@ 0x55fab089eff7 std::_Function_handler<>::_M_invoke()
@ 0x55fab0d14d7c doris::WorkThreadPool<>::work_thread()
@ 0x55fae76ae6f0 execute_native_thread_routine
@ 0x7fa32ea45ac3 (unknown)
@ 0x7fa32ead7850 (unknown)
@ (nil) (unknown)
Query id: a21981d5c8ef4113-84df9a5a8680e004 ***
is nereids: 0 ***
tablet id: 0 ***
Aborted at 1724668499 (unix time) try "date -d @1724668499" if you are using GNU date ***
Current BE git commitID: 2f848737c1 ***
SIGABRT unknown detail explain (@0x20db) received by PID 8411 (TID 9837 OR 0x7f9e42cfe640) from PID 8411; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# 0x00007FA32E9F3520 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill at ./nptl/pthread_kill.c:89
3# raise at ../sysdeps/posix/raise.c:27
4# abort at ./stdlib/abort.c:81
5# 0x000055FAE4387B1D in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
6# 0x000055FAE437A15A in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
8# google::LogMessage::Flush() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
10# doris::BaseTablet::check_delete_bitmap_correctness(std::shared_ptr, long, long, std::unordered_set, std::equal_to, std::allocator > const&, std::vector, std::allocator > >*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/base_tablet.cpp:1152
11# doris::RowsetBuilder::commit_txn() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset_builder.cpp:316
12# doris::LoadStreamWriter::close() at /home/zcp/repo_center/doris_master/doris/be/src/runtime/load_stream_writer.cpp:311
13# std::_Function_handler::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
14# doris::WorkThreadPool::work_thread(int) at /home/zcp/repo_center/doris_master/doris/be/src/util/work_thread_pool.hpp:159
15# execute_native_thread_routine at ../../../../../libstdc+-v3/src/c+11/thread.cc:84
16# start_thread at ./nptl/pthread_create.c:442
17# 0x00007FA32EAD7850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```
If create rowset failed,` calc_delete_bitmap_task` still could be
executed:
```
add segment failed load_id=5649413b98976f0d-a105b42749f561b0, txn_id=2, tablet_id=10088, status=[INTERNAL_ERROR]create row
set failed
...
submit calc delete bitmap task to executor, tablet_id: 10088, txn_id: 2
```
This PR skips close to avoid `submit_calc_delete_bitmap_task` if create
rowset failed when loading MOW table to solve this problem.
2024-09-24 16:02:22 +08:00
d7659ff34d
[fix](bytebuffer) fix allocate size improper in append_and_flush ( #40613 ) ( #41133 )
...
pick (#40613 )
fix allocate size improper in append_and_flush introduced by
https://github.com/apache/doris/pull/38960
2024-09-24 16:01:52 +08:00
602275d1fe
[improvement](stream load) display the time of receiving network data ( #40735 ) ( #41139 )
...
pick (#40735 )
Display the time of receiving network data.
For example:
```
Stream load result: {
"TxnId": 19,
"Label": "c10ee032-c19c-484f-80b2-16f6c1d0c209",
"Comment": "",
"TwoPhaseCommit": "false",
"Status": "Success",
"Message": "OK",
"NumberTotalRows": 6,
"NumberLoadedRows": 6,
"NumberFilteredRows": 0,
"NumberUnselectedRows": 0,
"LoadBytes": 60,
"LoadTimeMs": 32,
"BeginTxnTimeMs": 0,
"StreamLoadPutTimeMs": 4,
"ReadDataTimeMs": 0,
"WriteDataTimeMs": 4,
"ReceiveDataTimeMs": 6,
"CommitAndPublishTimeMs": 22
}
```
2024-09-24 16:00:28 +08:00
c81182aed8
[fix](restore) Fix clean restore with view #40620 ( #41186 )
...
cherry pick from #40620
2024-09-24 14:10:27 +08:00
ddd17be874
[fix](migrate) Fix iterator returning too early #41168 ( #41189 )
...
cherry pick from #41168
2024-09-24 14:10:10 +08:00
ff6f17c22c
[fix](external-p2) ignore external p2 cases( #41148 ) ( #41179 )
...
bp #41148
2024-09-24 09:58:50 +08:00
2b427c316a
[feature](functions) impl scalar functions normal_cdf,to_iso8601,from_iso8601_date ( #40695 ) ( #41049 )
...
bp #40695
2024-09-24 09:52:39 +08:00
8494496dc9
[fix](p2) fix assertion in test_broker_load_p2 ( #41116 ) ( #41117 )
...
cherry-pick #41116
2024-09-24 09:46:07 +08:00
0d38a9a36d
[feature](restore) support atomic restore ( #41107 )
...
Cherry-pick #40353 , #40734 , #40817 , #40876 , #40921 , #41017 , #41083
2024-09-24 09:41:41 +08:00
48e60f3ff3
[Fix](inverted index) fix wrong opt for count_on_index #41127 ( #41154 )
...
cherry pick from #41127
2024-09-23 22:45:52 +08:00
5bcea1983d
[opt](nereids) enable runtime filter use cte as target #40815 (2.1) ( #41090 )
...
## Proposed changes
pick #40815
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-23 22:34:03 +08:00
a6ef7e00e4
[fix](restore) Fix view signature #41120 ( #41150 )
...
cherry pick from #41120
2024-09-23 22:04:20 +08:00
d529f00dc0
[branch-2.1][Fix](hdfs-fs)The cache expiration should explicitly release the held fs ( #38610 ) ( #41108 )
...
bp #38610
2024-09-23 18:55:08 +08:00
fdd7ffc0bd
[branch-2.1][Fix](http)Ignore exceptions when getting system information errors #39671 ( #41106 )
...
bp #39671
2024-09-23 18:35:35 +08:00
e1057ac26d
[branch-2.1][fix](metadata)Add FE metadata-related file checks #40546 ( #41113 )
...
## Proposed changes
#40546
2024-09-23 17:13:35 +08:00
51bb4e9331
[fix](test-case) fix unstable test case about table_options sys table( #41006 ) ( #41121 )
...
bp #41006
2024-09-23 17:00:51 +08:00
76d62c42db
[pick](Variant) check enable light_schema_change when create table with variant type ( #41102 )
...
#40908
2024-09-23 14:30:16 +08:00
f6917acd6a
[cherry-pick](branch2.1) Impl translate and url encode 2.1 ( #41051 )
...
## Proposed changes
pick https://github.com/apache/doris/pull/40567
some code about const folding should wait the pr picked:
https://github.com/apache/doris/pull/40441
2024-09-23 14:26:27 +08:00
e175c63d41
[Fix](inverted index) Fix wrong need read data opt when enable_common_expr_pushdown is disabled #40689 ( #41101 )
...
cherry pick from #40689
2024-09-23 14:21:30 +08:00
059f822a3a
[fix](restore) Remove the sqlMode from the view signature #40615 ( #41110 )
...
cherry pick from #40615
2024-09-23 14:07:52 +08:00
9af2de5e6a
[regression-test](point query) add enable_mow_light_delete=false to related cases ( #41103 )
...
#40520
2024-09-23 12:32:29 +08:00
011baeb2d2
[bugfix](allocatebytes) ignore null ptr column in Block ( #41093 ) ( #41099 )
...
## Proposed changes
Sometimes if expr failed, then there will be a nullptr column left in
the block.
We should ignore the nullptr column, or exception will be thrown and
some profile will not computed correctly.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-09-23 12:18:22 +08:00
0e5c4281dc
[fix](function) fix Substring/SubReplace error result with input utf8… ( #40954 )
...
… string (#40929 )
https://github.com/apache/doris/pull/40929
```
mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1) |
+-------------------------------------+
| �a�好世界 |
+-------------------------------------+
mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647) |
+------------------------------------------+
| 中文测试 |
+------------------------------------------+
1 row in set (0.04 sec)
now
mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1) |
+-------------------------------------+
| 你a世界 |
+-------------------------------------+
1 row in set (0.05 sec)
mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647) |
+------------------------------------------+
| |
+------------------------------------------+
1 row in set (0.13 sec)
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-23 10:44:03 +08:00
d44ee1cddb
[branch-2.1](session-variable) Add a debug variable of constant fold ( #35584 ) ( #41032 )
...
pick https://github.com/apache/doris/pull/35584
2024-09-23 10:01:05 +08:00
7d64c8cbc6
[branch-2.1] Picks "[opt](autoinc) Remove some restrictions on schema change on table that has auto-increment column #40280 " ( #41096 )
...
picks https://github.com/apache/doris/pull/40280
2024-09-23 09:30:15 +08:00
9dc55f90eb
[opt](nereids) set lower bound for range-selectivity（2.1） ( #41061 )
...
## Proposed changes
pick #40089
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-22 07:32:22 +08:00
d1d52ae68c
[feature](compaction) Add an http action for visibility of compaction score on each tablet ( #38489 ) ( #40826 )
...
pick: #38489
Usage:
1. `curl http://be_ip:be_host/api/compaction_score?top_n=10 ` Returns a
json object contains compaction score for top n, n=top_n.
```
[
{
"compaction_score": "5",
"tablet_id": "42595"
},
{
"compaction_score": "5",
"tablet_id": "42587"
},
{
"compaction_score": "5",
"tablet_id": "42593"
},
{
"compaction_score": "5",
"tablet_id": "42597"
},
{
"compaction_score": "5",
"tablet_id": "42589"
},
{
"compaction_score": "5",
"tablet_id": "42599"
},
{
"compaction_score": "5",
"tablet_id": "42601"
},
{
"compaction_score": "5",
"tablet_id": "42591"
},
{
"compaction_score": "5",
"tablet_id": "42585"
},
{
"compaction_score": "4",
"tablet_id": "10034"
}
]
```
If top_n is not specified, return all compaction score for all tablets.
If top_n is illegal, raise an error.
```
invalid argument: top_n=wrong
```
2. `curl http://be_ip:be_host/api/compaction_score?sync_meta=true `
`sync_meta` is only available on cloud mode, will sync meta from meta
service. It can cooperate with top_n.
If add param `sync_meta` on non-cloud mode, will raise an error.
```
sync meta is only available for cloud mode
```
3. In the future, this endpoint may extend other utility, like fetching
tablet compaction score by table id, etc.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-21 20:35:55 +08:00
9877a08834
[feature](function) support ngram_search function #38226 ( #40893 )
...
https://github.com/apache/doris/pull/38226
mysql [test]>select ngram_search('123456789' , '12345' , 3);
+---------------------------------------+
| ngram_search('123456789', '12345', 3) |
+---------------------------------------+
| 0.6 |
+---------------------------------------+
1 row in set (0.01 sec)
mysql [test]>select ngram_search("abababab","babababa",2);
+-----------------------------------------+
| ngram_search('abababab', 'babababa', 2) |
+-----------------------------------------+
| 1 |
+-----------------------------------------+
1 row in set (0.01 sec)
```
doc https://github.com/apache/doris-website/pull/899
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-21 20:34:44 +08:00
d5115a21b5
[pick](ShortCircuit) Conjuncts outof key columns's order should be handled ( #41071 )
...
#37900
2024-09-21 20:34:05 +08:00
36a45c5ebf
[Fix](parquet-reader) Fix parquet reader crash in set_dict(). ( #41074 )
...
## Proposed changes
Backport #40643
2024-09-21 08:48:06 +08:00
e568ba3b4c
[bugfix](hive)Handle exceptions when submitting tasks fails to prevent dead loop for 2.1 ( #40708 ) ( #41052 )
...
bp: #40708
2024-09-21 08:47:31 +08:00
c6a6adb3a4
[Fix](topn) avoid missmatched row count when upgrading ( #40999 )
...
#41000
2024-09-21 08:46:57 +08:00
c9d71cdc64
[Enhancement](brpc)Added enable_brpc_builtin_services parameter in be.conf ( #40718 ) ( #41047 )
...
bp #40718
## Proposed changes
Since [brpc's built-in
services](https://brpc.apache.org/zh/docs/builtin-services/buildin_services/ )
will expose some internal server status, for security reasons, add the
parameter `enable_brpc_builtin_services` in `be.conf`. When the
parameter is false, this built-in service can be disabled.
2024-09-21 08:45:38 +08:00
4967635e54
[test](mtmv) Fix regression test unstable ( #40871 ) ( #41082 )
...
## Proposed changes
commitId: da6ac0c5
pr: https://github.com/apache/doris/pull/40871
2024-09-20 23:49:23 +08:00
63a1e0dccb
[fix](profile) Fix npe when updating profile for internal load task. #40990 ( #41077 )
...
cherry pick from #40990
2024-09-20 23:46:43 +08:00
ff22059d5d
[cherry-pick](branch-2.1)[fix](sys_table) fix potential NPE when query table_properties and partitions sys table ( #41081 )
...
## Proposed changes
Issue Number: close #xxx
bp #40972
2024-09-20 23:43:33 +08:00
f4b06e83eb
[bugfix](hive)Fix cache inconsistency issue for 2.1 ( #40729 ) ( #41050 )
...
## Proposed changes
bp: #40729
2024-09-20 20:28:55 +08:00
96b2fa1a24
[fix](move-memtable) fix bvar g_load_stream_file_writer_cnt ( #39075 ) ( #40985 )
...
backport #39075
2024-09-20 20:20:49 +08:00
f60a881425
[fix](nereids)adjust conjunct's nullable info in LogicalExternalRelation ( #41031 )
...
## Proposed changes
pick from master https://github.com/apache/doris/pull/41014
<!--Describe your changes.-->
2024-09-20 20:20:12 +08:00
2bb83302e4
[bugfix](hive)Remove redundant split operations for 2.1 ( #40683 ) ( #41055 )
...
## Proposed changes
bp: #40683
2024-09-20 20:09:11 +08:00
1686650a56
[bugfix](test)Set the enable_fallback_to_original_planner behavior for 2.1 ( #40913 )( #40989 ) ( #41065 )
...
bp: #40913 #40989
2024-09-20 20:08:38 +08:00
7e0d498043
[Fix](orc-reader) Fix StringRef nullptr data in orc-reader. ( #41073 )
...
## Proposed changes
backport #40857
2024-09-20 20:06:50 +08:00
