### What problem does this PR solve?
Problem Summary:
In HiveMetaStoreCache, the function FileInputFormat.setInputPaths is
used to set input paths. However, this function splits paths using
commas, which is not the expected behavior. As a result, when partition
values contain commas, it leads to incorrect path parsing and potential
errors.
```java
public static void setInputPaths(JobConf conf, String org.apache.hadoop.shaded.com.aSeparatedPaths) {
setInputPaths(conf, StringUtils.stringToPath(
getPathStrings(org.apache.hadoop.shaded.com.aSeparatedPaths)));
}
```
To prevent FileInputFormat.setInputPaths from splitting paths by commas,
we use another overloaded version of the method. Instead of passing a
comma-separated string, we explicitly pass a Path object, ensuring that
partition values containing commas are handled correctly.
```java
public static void setInputPaths(JobConf conf, Path... inputPaths) {
Path path = new Path(conf.getWorkingDirectory(), inputPaths[0]);
StringBuffer str = new StringBuffer(StringUtils.escapeString(path.toString()));
for(int i = 1; i < inputPaths.length;i++) {
str.append(StringUtils.COMMA_STR);
path = new Path(conf.getWorkingDirectory(), inputPaths[i]);
str.append(StringUtils.escapeString(path.toString()));
}
conf.set(org.apache.hadoop.shaded.org.apache.hadoop.mapreduce.lib.input.
FileInputFormat.INPUT_DIR, str.toString());
}
```
### Release note
None
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
pick: #49087
Related PR: #40441
Problem Summary:
wrong calculation of emoji character length in some String function when
do constant folding in FE. For example:
select STRLEFT('😊😉👍', 2);
should return 😊😉, but fe return 😊 only when folding constant
fixed functions:
- left
- strleft
- right
- strright
- locate
- character_length
- split_by_string
- overlay
- replace_empty
…8872)
https://github.com/apache/doris/pull/48872
before
```
mysql> select STR_TO_DATE (' ', '%Y-%m-%d %H:%i:%s');
+-----------------------------------------+
| STR_TO_DATE (' ', '%Y-%m-%d %H:%i:%s') |
+-----------------------------------------+
| |
+-----------------------------------------+
```
now
```
mysql> select STR_TO_DATE (' ', '%Y-%m-%d %H:%i:%s');
+-----------------------------------------+
| STR_TO_DATE (' ', '%Y-%m-%d %H:%i:%s') |
+-----------------------------------------+
| NULL |
+-----------------------------------------+
```
Problem Summary:
None
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
when nereids cast invalid date literal to date like type, it will throws
exceptions:
```
select '' = cast('2020-10-20' as date);
(1105, 'errCode = 2, detailMessage = date/datetime literal [] is invalid')
```
But old planner will not throw exceptions, so let neredis don't throw
exceptions too.
This PR is pick code from: #40202
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: Pick #48554
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
…ession #48103 (#48456)"
This reverts commit 4979d706127044598b1af853d1c2d69ab3f2cebb.
cherry-pick: #48688
revert: #48103
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
if we set session variable for fold_constant_for_be = 1 and
enable_decimal256 = true
here will meet error in sql
```
SELECT ARRAY_AVG(CAST([] AS ARRAY < DECIMALV3(1,0) > ));
```
with core
```
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /mnt/disk1/wangqiannan/amory/doris/be/src/vec/columns/column_decimal.h:200:15 in
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /mnt/disk1/wangqiannan/amory/doris/be/src/common/signal_handler.h:421
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so
2# JVM_handle_linux_signal in /mnt/disk1/wangqiannan/tool/jdk-17.0.10/lib/server/libjvm.so
3# 0x00007FD98C1A2B50 in /lib64/libc.so.6
4# doris::vectorized::ColumnDecimal<doris::vectorized::Decimal<wide::integer<256ul, int> > >::get_data_at(unsigned long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/columns/column_decimal.h:201
5# doris::vectorized::DataTypeDecimalSerDe<doris::vectorized::Decimal<wide::integer<256ul, int> > >::write_column_to_pb(doris::vectorized::IColumn const&, doris::PValues&, long, long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_decimal_serde.h:158
6# doris::vectorized::DataTypeNullableSerDe::write_column_to_pb(doris::vectorized::IColumn const&, doris::PValues&, long, long) const at /mnt/disk1/wangqiannan/amory/doris/be/src/vec/data_types/serde/data_type_nullable_serde.cpp:237
7# doris::FoldConstantExecutor::fold_constant_vexpr(doris::TFoldConstantParams const&, doris::PConstantExprResult*) at /mnt/disk1/wangqiannan/amory/doris/be/src/runtime/fold_constant_executor.cpp:118
8# doris::PInternalService::_fold_constant_expr(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::PConstantExprResult*) at /mnt/disk1/wangqiannan/amory/doris/be/src/service/internal_service.cpp:1537
9# doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0::operator()() const at /mnt/disk1/wangqiannan/amory/doris/be/src/service/internal_service.cpp:1515
10# void std::__invoke_impl<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>(std::__invoke_other, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&) at /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61
11# std::enable_if<is_invocable_r_v<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>, void>::type std::__invoke_r<void, doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&>(doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0&) at /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:117
12# std::_Function_handler<void (), doris::PInternalService::fold_constant_expr(google::protobuf::RpcController*, doris::PConstantExprRequest const*, doris::PConstantExprResult*, google::protobuf::Closure*)::$_0>::_M_invoke(std::_Any_data const&) at /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
13# std::function<void ()>::operator()() const at /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:560
14# doris::WorkThreadPool<false>::work_thread(int) at /mnt/disk1/wangqiannan/amory/doris/be/src/util/work_thread_pool.hpp:158
15# void std::__invoke_impl<void, void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>(std::__invoke_memfun_deref, void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&) at /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74
16# std::__invoke_result<void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>::type std::__invoke<void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&>(void (doris::WorkThreadPool<false>::* const&)(int), doris::WorkThreadPool<false>*&, int&) at /mnt/disk1/wangqiannan/tool/ldb_toolchain_16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96
```
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
sql
```
SELECT *, ROW_NUMBER() OVER ( PARTITION BY orderid ORDER BY v DESC ) AS row_idfirst FROM test_v_legacy order by orderid;
```
core:
```
F20250225 11:23:08.735848 1788347 column_object.h:439] should not call the method in column object
*** Check failure stack trace: ***
F20250225 11:23:08.736686 1788345 column_object.h:439] should not call the method in column object
*** Check failure stack trace: ***
@ 0x5555fa0821b6 google::LogMessage::SendToLog()
@ 0x5555fa0821b6 google::LogMessage::SendToLog()
@ 0x5555fa07ec00 google::LogMessage::Flush()
@ 0x5555fa07ec00 google::LogMessage::Flush()
@ 0x5555fa0829f9 google::LogMessageFatal::~LogMessageFatal()
@ 0x5555fa0829f9 google::LogMessageFatal::~LogMessageFatal()
@ 0x5555dd722b1c doris::vectorized::ColumnObject::get_permutation()
@ 0x5555dd722b1c doris::vectorized::ColumnObject::get_permutation()
@ 0x5555dd6b933c doris::vectorized::ColumnNullable::get_permutation()
@ 0x5555dd6b933c doris::vectorized::ColumnNullable::get_permutation()
@ 0x5555dde3997f doris::vectorized::sort_block()
@ 0x5555dde3997f doris::vectorized::sort_block()
@ 0x5555e2f13646 doris::vectorized::Sorter::partial_sort()
@ 0x5555e2f13646 doris::vectorized::Sorter::partial_sort()
@ 0x5555e2f1503b doris::vectorized::FullSorter::_do_sort()
@ 0x5555e2f1503b doris::vectorized::FullSorter::_do_sort()
@ 0x5555e2f15a2f doris::vectorized::FullSorter::prepare_for_read()
@ 0x5555e2f15a2f doris::vectorized::FullSorter::prepare_for_read()
@ 0x5555f97fb76a doris::pipeline::SortSinkOperatorX::sink()
@ 0x5555f97fb76a doris::pipeline::SortSinkOperatorX::sink()
@ 0x5555fa018798 doris::pipeline::PipelineXTask::execute()
@ 0x5555fa018798 doris::pipeline::PipelineXTask::execute()
@ 0x5555fa04efe5 doris::pipeline::TaskScheduler::_do_work()
@ 0x5555fa04efe5 doris::pipeline::TaskScheduler::_do_work()
@ 0x5555fa052f2b doris::pipeline::TaskScheduler::start()::$_0::operator()()
@ 0x5555fa052f2b doris::pipeline::TaskScheduler::start()::$_0::operator()()
```
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
pick:#47966
Related PR: #40744， #47228
Problem Summary:
When numeric function input or output out of boundary when fe folding
constant, it would throw an exception before. Now
we change it to match with be execution result, which is NullLiteral
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
