f528a8aa06
[Opt](log) change frequent INFO log to DEBUG log in ColumnObject ( #42336 ) ( #42434 )
...
(#42336 )
2024-10-24 23:03:58 +08:00
cbdaaa62b2
[feature](function) hour/minute/second functions support time as an a… 41008 ( #42232 )
...
…rgument. (#41008 )
## Proposed changes
```
mysql> select cast(4562632 as time),hour(cast(4562632 as time)), minute(cast(4562632 as time)),second(cast(4562632 as time));
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
| cast(4562632 as TIME) | hour(cast(4562632 as TIME)) | minute(cast(4562632 as TIME)) | second(cast(4562632 as TIME)) |
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
| 456:26:32 | 456 | 26 | 32 |
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
```
<!--Describe your changes.-->
---------
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Co-authored-by: Dongyang Li <hello_stephen@qq.com >
2024-10-24 11:09:36 +08:00
a3510d3b73
[cherry-pick](branch-21)fix the exchange not skip offset rows ( #42190 ) ( #42330 )
...
cherry-pick from master (#42190 )
2024-10-24 11:08:17 +08:00
64d0c55e1d
[fix](decimal256) support decimal256 for many functions ( #42136 ) ( #42353 )
...
BP #42136
2024-10-24 10:52:28 +08:00
e7395198d4
[fix](estimate) func call with not filter will estimate some statisti… ( #42302 )
...
…cs (#41989 )
before this pr: use ! or not in or condition when table has been
analyzed we will meet
```
SELECT
count(1)
FROM
table_30_un_pa_ke_pr_di4
where
col_int_undef_signed_not_null < -128
or not array_contains(col_array_bigint__undef_signed, col_int_undef_signed_not_null);
ERROR 1105 (HY000): errCode = 2, detailMessage = Not-predicate meet unexpected child:
array_contains(col_array_bigint__undef_signed, cast(col_int_undef_signed_not_null as BIGINT))
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-23 14:33:41 +08:00
157d67e7ca
[enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug #42200 ( #42273 )
...
cherry pick from #42200
Co-authored-by: Socrates <suxiaogang223@icloud.com >
2024-10-22 23:56:57 +08:00
bde8e2d474
[2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool ( #42255 )
...
pick (#41992 )
We initially introduced jdbc connection pool to improve the connection
performance of jdbc catalog, but we always found that connection pool
would bring some unexpected errors, so we chose to add a catalog
property: `enable_connection_pool` to choose whether to enable the jdbc
connection pool of jdbc catalog, and the default false.However, the
created catalog will still open the connection pool when it is upgraded,
and only the newly created catalog will be false
And we conducted performance tests on this, the performance loss is
within the expected range.
- Enable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM mysql.test.test
limit 1;' --create-schema=mysql --delimiter=";" --verbose
Benchmark
Average number of seconds to run all queries: 0.008 seconds
Minimum number of seconds to run all queries: 0.004 seconds
Maximum number of seconds to run all queries: 0.133 seconds
Number of clients running queries: 1
Average number of queries per client: 1
- Disable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM
mysql_no_pool.test.test limit 1;' --create-schema=mysql --delimiter=";"
--verbose
Benchmark
Average number of seconds to run all queries: 0.054 seconds
Minimum number of seconds to run all queries: 0.047 seconds
Maximum number of seconds to run all queries: 0.184 seconds
Number of clients running queries: 1
Average number of queries per client: 1
2024-10-22 23:28:28 +08:00
25d7d0b255
[fix](move-memtable) abstract multi-streams to one logical stream ( #42039 ) ( #42250 )
...
backport #42039
2024-10-22 20:26:42 +08:00
38e529cd29
[cherry-pick](branch-2.1) support decimal256 for parquet reader ( #42241 )
...
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41526
2024-10-22 19:42:09 +08:00
e2bdac39fb
[fix] Implementing match_phrase_edge without index query method ( #41658 ) ( #42098 )
...
pick from #41658
2024-10-22 18:44:21 +08:00
7eec0f8fbb
[branch-2.1](datetime) Fix date floor functions overflow ( #35477 ) ( #42238 )
...
pick https://github.com/apache/doris/pull/35477
2024-10-22 15:54:53 +08:00
47ff6f1300
[fix](OrcReader) fix the issue that orc_reader can not read DECIMAL(0,0) type of orc file #41795 ( #42220 )
...
cherry pick from #41795
Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com >
2024-10-22 10:10:25 +08:00
1f8d685f26
[fix](inverted index) multi_match supports any, all, phrase. ( #41663 ) ( #42097 )
...
https://github.com/apache/doris/pull/41663
2024-10-22 10:10:02 +08:00
e713b92321
[fix](multi-catalog) Disable string dictionary filtering when predicate express is not slot #42113 ( #42222 )
...
cherry pick from #42113
Co-authored-by: Socrates <suxiaogang223@icloud.com >
2024-10-22 09:43:29 +08:00
084434e25c
[Test](tvf) add regression tests for testing orc reader #41606 #42188 ( #42120 )
...
cherry pick from #42031 #42188
---------
Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com >
Co-authored-by: TieweiFang <ftw2139@163.com >
2024-10-21 21:31:18 +08:00
b9e2738ee6
[Fix](orc-reader) Fix StringRef nullptr data by add checking string_values empty. #42061 ( #42154 )
...
cherry pick from #42061
Co-authored-by: Qi Chen <kaka11.chen@gmail.com >
2024-10-21 16:26:23 +08:00
5db44a1b91
[fix](arrays_overlap) support arrays overlap with inverted index ( #42090 )
...
## Proposed changes
backport : https://github.com/apache/doris/pull/41286
https://github.com/apache/doris/pull/41495
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-18 22:08:39 +08:00
dde0bf92ce
[fix](inverted index) Fix incorrect usage of regexp compile_err ( #41944 ) ( #42085 )
...
https://github.com/apache/doris/pull/41944
2024-10-18 22:06:59 +08:00
460ff02997
[cherry-pick](branch-21)fix date_floor function return wrong result ( #41948 ) ( #42065 )
...
## Proposed changes
cherry-pick from master https://github.com/apache/doris/pull/41948
<!--Describe your changes.-->
2024-10-18 21:54:22 +08:00
03136baacf
[fix](scanner) Fix incorrect _max_thread_num in scanner context when many queries are running. #41273 ( #42016 )
...
cherry pick from #41273
2024-10-18 18:08:07 +08:00
fb12e10272
[fix](array-funcs)fix array agg func with decimal type ( #40839 ) ( #42023 )
...
## Proposed changes
backport: (https://github.com/apache/doris/pull/40839 )
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-17 20:47:39 +08:00
5806dae467
[fix](move-memtable) do not retry open streams ( #41550 ) ( #41999 )
...
backport #41550
2024-10-17 15:56:56 +08:00
b4875c2789
[fix](jni)fix jni use timezone_obj get timezone be core. ( #41956 ) ( #42003 )
...
bp #41956
This PR #40225 try to pass time zone info from BE to JNI, and it use
`_state->timezone_obj().name()`
to get the timezone name.
But when we do some rolling upgrade of BE, it may coredump like:
```
*** SIGSEGV address not mapped to object (@0x610) received by PID 72661 (TID 73538 OR 0x7f2e898d1640) from PID 1552; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
1# os::Linux::chained_handler(int, siginfo_t*, void*) in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo_t*, void*) in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so
4# 0x00007F3070D3E520 in /lib/x86_64-linux-gnu/libc.so.6
5# cctz::time_zone::name[abi:cxx11]() const in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
6# doris::vectorized::JniConnector::open(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/jni_connector.cpp:87
7# doris::vectorized::AvroJNIReader::init_fetch_table_schema_reader() at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/exec/format/avro/avro_jni_reader.cpp:119
8# std::_Function_handler::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
9# doris::WorkThreadPool::work_thread(int) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/work_thread_pool.hpp:159
10# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
11# start_thread at ./nptl/pthread_create.c:442
12# 0x00007F3070E22850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
172.20.50.206 last coredump sql: 2024-10-13 04:12:23,985 [query]
```
This PR use another method: `_state->timezone()`, which just return a
string, instead of reading and initializing
time zone info file, to avoid potential coredump.
2024-10-17 14:47:33 +08:00
67d057a711
[cherry-pick](branch-21) fix conv function parser string failure return wrong result ( #40530 ) ( #41964 )
...
## Proposed changes
Issue Number: close #39618
cherry-pick from master (#40530 )
2024-10-17 14:45:46 +08:00
0b41cd2472
[fix](serde)fix the bug in DataTypeNullableSerDe.deserialize_column_from_fixed_json ( #41217 ) ( #41960 )
...
bp #41217
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-17 14:36:01 +08:00
968e33f07e
[cherry-pick](branch-21) pick ( #39057 ) ( #41352 ) ( #41958 )
...
## Proposed changes
pick from master (#39057 ) (#41352 )
<!--Describe your changes.-->
---------
Co-authored-by: Mryange <59914473+Mryange@users.noreply.github.com >
2024-10-17 14:30:40 +08:00
1b901f6fcc
[cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet bug ( #41931 )
...
## Proposed changes
pick pr:
https://github.com/apache/doris/pull/41683
https://github.com/apache/doris/pull/41506
https://github.com/apache/doris/pull/41338
https://github.com/apache/doris/pull/39326
---------
Co-authored-by: morningman <morningman@163.com >
2024-10-17 14:20:58 +08:00
0b6447faeb
[Fix](SchemaChange) refactor variant root column iterator to make row… ( #41941 )
...
pick #41700
2024-10-17 10:39:07 +08:00
5bd33fc88c
[pick](branch-2.1) pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751 ( #41927 )
...
## Proposed changes
pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751
<!--Describe your changes.-->
---------
Co-authored-by: Pxl <pxl290@qq.com >
2024-10-16 15:41:28 +08:00
e6545a36a3
[improvement](iceberg)Parallelize splits for count(*) for 2.1 ( #41169 ) ( #41880 )
...
bp: #41169
2024-10-16 10:52:06 +08:00
b185dfcbf6
[pick](branch-2.1) pick #41676 #41740 #41857 ( #41904 )
...
pick #41676 #41740 #41857
2024-10-15 22:41:17 +08:00
94687a2f3c
[fix](array/map) fix resize impl in array/map ( #41595 ) ( #41699 )
...
backport: https://github.com/apache/doris/pull/41595
2024-10-15 09:50:11 +08:00
4888c632f4
[cherry-pick](branch2.1) support escape.delim and serialization.null.format for hive text ( #41684 )
...
## Proposed changes
pick from master:
https://github.com/apache/doris/pull/40291
2024-10-15 00:08:23 +08:00
ff52e73a07
[Fix](inverted index) fix match null for inverted index #41746 ( #41787 )
...
cherry pick from #41746
2024-10-14 14:45:36 +08:00
f112af0fd2
[pick](branch-2.1) pick #41555 #41592 #38204 ( #41781 )
...
pick #41555 #41592 #38204
2024-10-14 14:05:08 +08:00
e10458baad
[enhancement](err-msg) Output column info when size invalid in block data convertor ( #41535 ) ( #41764 )
...
## Proposed changes
pick: #41535
As title.
2024-10-12 21:08:04 +08:00
4ac07fe918
[Feature](json) Support json_search function in 2.1 ( #41590 )
...
cherry-pick #40948
Like mysql, json_search returns the path which point to a json string
witch match the pattern.
`SELECT JSON_SEARCH('["A",[{"B":"1"}],{"C":"AB"},{"D":"BC"}]', 'one',
'A_') as res;`
```
+----------+
| res |
+----------+
| "$[2].C" |
+----------+
```
Co-authored-by: liutang123 <liulijia@gmail.com >
2024-10-11 16:33:07 +08:00
8c0f73cb90
[Enhancement](MaxCompute)Refactoring maxCompute catalog using Storage API.( #40225 , #40888 ,#41386 ) ( #41610 )
...
bp #40225 , #40888 ,#41386
## Proposed changes
Among them, #40225 is the new api of mc,
#40888 is used to fix the bug when reading null between the new and old
apis,
#41386 is used for compatibility between the new and old versions
2024-10-11 11:55:41 +08:00
0fb42d3a48
[Enhancement](tvf)catalog tvf implements user permission checks and hides sensitive information ( #41497 ) ( #41604 )
...
bp #41497
before #21790
## Proposed changes
This PR unifies the duplicate parts of `catalog tvf` and `show
catalogs`, adds permission check when querying `catalog tvf`, and hides
sensitive information.
2024-10-10 17:55:40 +08:00
1db0aef9b7
[feature](array_agg) support array_agg with param is array/map/struct… ( #41651 )
...
… (#40697 )
this pr we support array_agg function support param with array map
struct type
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-10-10 17:54:54 +08:00
31b506c8cc
[Enhancement](inverted index) return OK instead of not supported in expr evaluate_inverted_index #41567 ( #41578 )
...
cherry pick from #41567
2024-10-09 15:14:38 +08:00
c24ff2ff81
[fix](upgrade) fix version check failure of window_funnel when upgrading ( #41542 )
...
## Proposed changes
Issue Number: close #xxx
Fix fix version check failure of window_funnel when upgrading from 2.1.6
and higher version to latest branch 2.1.
```
02:49:13 F20240930 02:47:48.546983 7581 block.cpp:89] Check failed: BeExecVersionManager::check_be_exec_version(be_exec_version)
02:49:13 *** Check failure stack trace: ***
02:49:13 @ 0x564640041856 google::LogMessage::SendToLog()
02:49:13 @ 0x56464003e2a0 google::LogMessage::Flush()
02:49:13 @ 0x564640042099 google::LogMessageFatal::~LogMessageFatal()
02:49:13 @ 0x56463922d106 doris::vectorized::Block::deserialize()
02:49:13 @ 0x5646390a82bf doris::vectorized::WindowFunnelState<>::read()
02:49:13 @ 0x5646390a6889 doris::vectorized::IAggregateFunctionDataHelper<>::deserialize_and_merge()
02:49:13 @ 0x5646390acdc3 doris::vectorized::IAggregateFunctionHelper<>::deserialize_and_merge_from_column_range()
02:49:13 @ 0x56463fa77152 doris::pipeline::AggSinkLocalState::_merge_without_key()
02:49:13 @ 0x56463fa9d114 doris::pipeline::AggSinkLocalState::Executor<>::execute()
02:49:13 @ 0x56463fa78569 doris::pipeline::AggSinkOperatorX::sink()
02:49:13 @ 0x564640013296 doris::pipeline::PipelineXTask::execute()
02:49:13 @ 0x56464001d41c doris::pipeline::TaskScheduler::_do_work()
02:49:13 @ 0x56463663e078 doris::ThreadPool::dispatch_thread()
02:49:13 @ 0x564636634901 doris::Thread::supervise_thread()
02:49:13 @ 0x7fb64cf58ac3 (unknown)
02:49:13 @ 0x7fb64cfea850 (unknown)
02:49:13 @ (nil) (unknown)
02:49:13 *** Query id: b0cd194940184766-961c310e833e92b1 ***
02:49:13 *** is nereids: 1 ***
02:49:13 *** tablet id: 0 ***
02:49:13 *** Aborted at 1727635668 (unix time) try "date -d @1727635668" if you are using GNU date ***
02:49:13 *** Current BE git commitID: 653e315ba5 ***
02:49:13 *** SIGABRT unknown detail explain (@0x1648) received by PID 5704 (TID 7581 OR 0x7fb354a9a640) from PID 5704; stack trace: ***
02:49:13 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
02:49:13 1# 0x00007FB64CF06520 in /lib/x86_64-linux-gnu/libc.so.6
02:49:13 2# pthread_kill at ./nptl/pthread_kill.c:89
02:49:13 3# raise at ../sysdeps/posix/raise.c:27
02:49:13 4# abort at ./stdlib/abort.c:81
02:49:13 5# 0x000056464004C06D in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 6# 0x000056464003E76A in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 8# google::LogMessage::Flush() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 10# doris::vectorized::Block::deserialize(doris::PBlock const&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/core/block.cpp:113
02:49:13 11# doris::vectorized::WindowFunnelState<(doris::vectorized::TypeIndex)14, long>::read(doris::vectorized::BufferReadable&) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function_window_funnel.h:363
02:49:13 12# doris::vectorized::IAggregateFunctionDataHelper<doris::vectorized::WindowFunnelState<(doris::vectorized::TypeIndex)14, long>, doris::vectorized::AggregateFunctionWindowFunnel<(doris::vectorized::TypeIndex)14, long> >::deserialize_and_merge(char*, char*, doris::vectorized::BufferReadable&, doris::vectorized::Arena*) const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:517
02:49:13 13# doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullVariadicInline<doris::vectorized::AggregateFunctionWindowFunnel<(doris::vectorized::TypeIndex)14, long>, false> >::deserialize_and_merge_from_column_range(char*, doris::vectorized::IColumn const&, unsigned long, unsigned long, doris::vectorized::Arena*) const at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:465
02:49:13 14# doris::pipeline::AggSinkLocalState::_merge_without_key(doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:389
02:49:13 15# doris::pipeline::AggSinkLocalState::Executor<true, true>::execute(doris::pipeline::AggSinkLocalState*, doris::vectorized::Block*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.h:73
02:49:13 16# doris::pipeline::AggSinkOperatorX::sink(doris::RuntimeState*, doris::vectorized::Block*, bool) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/exec/aggregation_sink_operator.cpp:744
02:49:13 17# doris::pipeline::PipelineXTask::execute(bool*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/pipeline_x/pipeline_x_task.cpp:332
02:49:13 18# doris::pipeline::TaskScheduler::_do_work(unsigned long) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/pipeline/task_scheduler.cpp:347
02:49:13 19# doris::ThreadPool::dispatch_thread() in /mnt/hdd01/ci/compatibility-deploy/be/lib/doris_be
02:49:13 20# doris::Thread::supervise_thread(void*) at /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thread.cpp:499
02:49:13 21# start_thread at ./nptl/pthread_create.c:442
02:49:13 22# 0x00007FB64CFEA850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
02:49:13 [2024-09-30 02:49:13,147 __main__:796] [INFO]: 172.20.50.73 last coredump sql: 2024-09-30 02:48:18,328 [query] Query b0cd194940184766-961c310e833e92b1 1 times with new query id: 2e0e00de0e7548dd-95f9abc9d8d11c3a
```
2024-10-08 17:13:33 +08:00
98a1311aa2
[Opt](scanner-scheduler) Opt scanner scheduler starvation issue. ( #41484 )
...
## Proposed changes
Backport #40641
2024-09-30 15:40:20 +08:00
2b9c963edb
[fix](scanner) Check query status when iterating through rowsets and segments #41363 ( #41452 )
...
cherry pick from #41363
2024-09-30 09:49:46 +08:00
676c4e3b43
[Fix](inverted index) fix comparison for string after refactor compound #40338 ( #41441 )
...
cherry pick from #40338
2024-09-29 22:10:01 +08:00
34429bfa0e
[Chore](inverted index) remove useless code of compound filters for inverted index #40258 ( #41448 )
...
cherry pick from #40258
2024-09-29 17:27:29 +08:00
ddbb436e36
[Chore][inverted index] remove duplicate null bitmap reader in function array index #40907 ( #41444 )
...
cherry pick from #40907
2024-09-29 12:09:12 +08:00
b3bb4f4402
[Enhancement](inverted index) improve expr evaluate_inverted_index performace and remove useless code #40600 ( #41442 )
...
cherry pick from #40600
2024-09-29 12:00:58 +08:00
d659750fd9
[pick](Serde-2.1) fix variant serde may lost num_rows when subcolumns empty ( #41438 )
...
serialization object with empty subcolumns may lost num_rows, so need to
record num_rows and set back num_rows in serdes
backport #38413
2024-09-29 09:45:37 +08:00
727f0374be
[Refactor](inverted index) refactor inverted index compound predicates evaluate logic #38908 ( #41385 )
...
cherry pick from #38908
2024-09-29 09:19:17 +08:00
