Commit Graph

1515 Commits

Author SHA1 Message Date
aad189cf40 [feature](function) upper lower support utf8 input (#49756)
### What problem does this PR solve?
https://github.com/apache/doris/pull/49231
2025-04-07 12:00:31 +08:00
c0bc16d88f [fix](function) wrong result of arrays_overlap (#49403) (#49707)
Pick #49403
If the two arrays have the same non-null elements, they are considered
overlapping, and the result is 1.
If the two arrays have no common non-null elements and either array
contains a null element, the result is null.
Otherwise, the result is 0.

```
select arrays_overlap([1, 2, 3], [1, null]);  -- result should be 1

select arrays_overlap([2, 3], [1, null]);  -- result should be null

select arrays_overlap([2, 3], [1]);   -- result should be 0
```

### What problem does this PR solve?
2025-04-04 20:58:01 +08:00
145e393d3d branch-2.1: [fix](function) check return type is nullptr in FunctionBasePtr::build #49737 (#49761) 2025-04-02 20:23:41 +08:00
1259ee5088 branch-2.1: [Feature](function) support year of week #48870 (#49012) 2025-03-29 11:24:45 +08:00
4a31fc4e09 [Bug](fix) fix the percentile func result do not equal the percentile array rewrite result (#49379)
cherry pick https://github.com/apache/doris/pull/49351
2025-03-29 08:56:24 +08:00
ce49f37a5e branch-2.1: [fix](core) fix subreplace when inputting a large number of empty strings #49241 (#49303)
Cherry-picked from #49241

Co-authored-by: Mryange <yanxuecheng@selectdb.com>
2025-03-20 22:56:44 +08:00
8f79742f7d branch-2.1: [fix](arrow) Fix Arrow serialization and deserialization of Date/Datetime/Array/Map/Struct/Bitmap/HLL/Decimal256 types (#49244)
### What problem does this PR solve?

pick #48944 [fix](arrow) Fix UT DataTypeSerDeArrowTest of
Array/Map/Struct/Bitmap/HLL/Decimal256 types
pick #48398  [fix](arrow) Fix UT DataTypeSerDeArrowTest of Date type
2025-03-20 09:57:04 +08:00
f771a422a9 branch-2.1: [fix](column) fix ColumnWithTypeAndName::get_nested use-after-free when input Const(Nullable) column #48288 (#49258) 2025-03-20 09:53:20 +08:00
3b61f840f4 [fix](function) Undefined behavior in parse_url (#49149) (#49226) 2025-03-19 17:32:47 +08:00
e5a2b0eea8 Revert "[cherry-pick](jsonb) add a check for jsonb value to avoid invalid jsonb value write into segment file " (#49058)
Reverts apache/doris#48729
temp revert this pr for
PartialUpdateInfo::_generate_default_values_for_missing_cids using empty
string , which will make this check fail.
2025-03-14 17:41:06 +08:00
ad6cf63a28 branch-2.1: [opt](inverted index) uniform profile naming convention #48826 (#48975)
Cherry-picked from #48826

Co-authored-by: zzzxl <yangsiyu@selectdb.com>
2025-03-14 14:04:46 +08:00
ed2e1ac34a branch-2.1: [fix](variant) update least common type in ColumnObject::pop_back #48935 (#48979)
Cherry-picked from #48935

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
2025-03-13 17:41:17 +08:00
e455bceb91 [fix](function) fix error result when STR_TO_DATE input all space (#4… (#48920)
…8872)
https://github.com/apache/doris/pull/48872
before
```
mysql> select STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s');
+-----------------------------------------+
| STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s') |
+-----------------------------------------+
|                                         |
+-----------------------------------------+
```
now
```
mysql> select STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s');
+-----------------------------------------+
| STR_TO_DATE ('  ', '%Y-%m-%d %H:%i:%s') |
+-----------------------------------------+
| NULL                                    |
+-----------------------------------------+
```

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
2025-03-11 19:30:38 +08:00
3f684f2899 branch-2.1:[fix] (inverted index) Fix UTF-8 4-byte truncation issue and add configuration to control correct term writing (#48657) (#48741)
Cherry-picked from #48657
2025-03-06 21:28:24 +08:00
7b2899a7ff [cherry-pick](jsonb) add a check for jsonb value to avoid invalid jsonb value write into segment file (#48729)
…ke select core (#48625)

fix invalid jsonb value write into segment file which make select core,
so we add a check for jsonb value when convert_to_olap which value will
be written into segment file
2025-03-06 15:50:35 +08:00
621944d487 [InvertedIndex](Variant) supoort inverted index for array type in variant (#48594)
cherry-pick from #47688
2025-03-05 10:02:13 +08:00
08e7d920db branch-2.1: [fix](index build) Correct inverted index behavior after dynamically adding a column #48389 (#48546)
Cherry-picked from #48389

---------

Co-authored-by: airborne12 <jiangkai@selectdb.com>
2025-03-05 09:26:54 +08:00
cd3e1dce74 [feature](inverted index) Add profile statistics for each condition in inverted index filters (#48459)
https://github.com/apache/doris/pull/47504
2025-03-01 11:00:19 +08:00
1aa57a3b13 branch-2.1: [fix](array index) Correct null bitmap writing for inverted index #47846 (#48214)
cherry pick from #47846 #48231
2025-02-25 20:31:18 +08:00
470030b878 [feat](clone) Speed clone tablet via batch small file downloading #45061 (#45218)
cherry pick from #45061
2025-02-10 19:38:40 +08:00
3ec723f2cb branch-2.1: [fix](prepared statement) fix protocol with TIME datatype #47389 (#47543)
Cherry-picked from #47389

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2025-02-08 13:00:49 +08:00
701aec6b21 branch-2.1: [opt](jsonb) add ut for the jsonb parser #47181 (#47388)
Cherry-picked from #47181

Co-authored-by: Sun Chenyang <sunchenyang@selectdb.com>
2025-01-24 17:29:33 +08:00
Pxl
58415c3591 [Chore](case) add test case for cityhash #46928 (#46957)
pick from #46928
2025-01-14 14:03:19 +08:00
4472648c07 [branch-2.1] pick workload group usage metrics (#46177)
pick #45284  #44870
2024-12-31 10:09:48 +08:00
df26475e1a [Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run (#45411) (#46160)
pick master #45411
2024-12-31 09:51:43 +08:00
Pxl
43c646363e [Bug](runtime-filter) support ip rf and use exception to replace dche… (#41531)
…ck when PrimitiveType to PColumnType (#39985)

use exception to replace dcheck when PrimitiveType to PColumnType
```cpp
*** SIGABRT unknown detail explain (@0x11d3f) received by PID 73023 (TID 74292 OR 0x7fd758225640) from PID 73023; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
 1# 0x00007FDDBE6B9520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill at ./nptl/pthread_kill.c:89
 3# raise at ../sysdeps/posix/raise.c:27
 4# abort at ./stdlib/abort.c:81
 5# 0x000056123F81A94D in /root/output/be/lib/doris_be
 6# 0x000056123F80CF8A in /root/output/be/lib/doris_be
 7# google::LogMessage::SendToLog() in /root/output/be/lib/doris_be
 8# google::LogMessage::Flush() in /root/output/be/lib/doris_be
 9# google::LogMessageFatal::~LogMessageFatal() in /root/output/be/lib/doris_be
10# doris::to_proto(doris::PrimitiveType) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:114
11# doris::IRuntimeFilter::push_to_remote(doris::TNetworkAddress const*) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:1143
12# doris::IRuntimeFilter::publish(bool)::$_0::operator()(doris::IRuntimeFilter*) const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:959
13# doris::IRuntimeFilter::publish(bool)::$_2::operator()() const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:983
14# doris::IRuntimeFilter::publish(bool) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:997
```

## Proposed changes
pick from #39985
2024-12-30 20:56:11 +08:00
d2c108726d [opt](bloomfilter index) optimize memory usage for bloom filter index writer #45833 (#46047)
cherry pick from #45833
2024-12-27 12:10:56 +08:00
df8bc8f23d branch-2.1: [fix](parquet) impl has_dict_page to replace old logic and fix write empty parquet row group bug #45740 (#45954)
Cherry-picked from #45740

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-12-26 15:17:49 +08:00
1cf6986cea [pick](branch-2.1) pick #44092 (#45836) 2024-12-25 23:11:19 +08:00
64195d79ee [refactor](metrics) Remove IntAtomicCounter & CoreLocal #45742 (#45870)
cherry pick from #45742
2024-12-24 23:13:48 +08:00
02f15a8ef0 [fix](inverted index) Fix Null Pointer Exception in function match(#45456)(#45774)
pick: https://github.com/apache/doris/pull/45456
2024-12-24 11:27:13 +08:00
79662fcc94 [branch-2.1](functions) clean some ip functions code and make IS_IP_ADDRESS_IN_RANGE DEPENDS_ON_ARGUMENT (#45358)
pick https://github.com/apache/doris/pull/35239


add special logic to deal smooth upgrade

The origin PR is https://github.com/apache/doris/pull/35239. for
branch-3.0 it was merged in 3.0.0 but forgot to register old version.
now in branch-3.0 we fix it in
https://github.com/apache/doris/pull/45428 which must be merged in
3.0.4. and do same thing in this PR which must be merged in 2.1.8.
then:
```
FROM    TO    result
217-    218+    
217-    303-    💥
218+    303-    
218+    304+    
303-    304+    
```
this is our best result.
2024-12-17 11:51:07 +08:00
667f5e6e6a [feat](iceberg)Supports using rest type catalog to read tables in unity catalog for 2.1 (#43525) (#45217)
bp: #43525
2024-12-12 00:49:36 -08:00
5d3f0a267a [opt](scan) unify the local and remote scan bytes stats for all scanners for 2.1 (#45167)
pick part of #40493

TODO: not working with s3 reader
2024-12-10 14:19:19 +08:00
0bcb99710b [branch-2.1]Add customStdAllocator for vector/map (#41193) (#45124) 2024-12-09 18:33:51 +08:00
d4a6fd1850 Revert #43255 & #44615 (#45096)
Revert "branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for
OR-connected Predicate #43255 (#44438)"
Revert "[fix](orc) check all the cases before build_search_argument
(#44615) (#44801)"
2024-12-06 21:14:13 +08:00
94746b764d branch-2.1: [fix](memory) Fix UT ThreadMemTrackerMgrTest #44147 (#44936)
Cherry-picked from #44147

Co-authored-by: Xinyi Zou <zouxinyi@selectdb.com>
2024-12-06 10:54:12 +08:00
0c3bc1bcd0 [fix](auth)fix be enable http auth, some request link never return. (#44959) (#45063)
bp #44959

if you `enable_all_http_auth = true` in be.conf, then restart be, and
keep using `curl -u "xxxx:xxxx" http://127.0.0.1:8040/api/health` while
be is starting. You may encounter a situation where the link does not
return.
Reason:
When be is still starting, there is no information about fe master. When
you make an api request to be http port, be needs to request
authentication information from fe, which will cause it to request a
machine with empty ip and port 0. This rpc call will definitely fail
(this is not equivalent to a password error). After receiving this
failure, be does not `send_reply` to the api requester, so this api
request cannot be returned.
2024-12-06 09:39:13 +08:00
5f952cf6ed branch-2.1: [fix](iceberg)Bring field_id with parquet files And fix map type's key optional #44470 (#44828)
Cherry-picked from #44470

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
2024-12-02 10:24:07 +08:00
4b15b1f263 [fix](orc) check all the cases before build_search_argument (#44615) (#44801)
cherry-pick #44615

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-30 09:17:56 +08:00
cf2ee23e68 Revert "[fix] (compaction) fix compaction score in time series policy (#40242)" (#44527)
revert #40242
2024-11-26 12:18:59 +08:00
dceaf97381 branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255 (#44438)
Cherry-picked from #43255

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-22 22:52:53 +08:00
702abbff0f [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) (#44239)
bp #42004

Co-authored-by: kaka11chen <kaka11.chen@gmail.com>
2024-11-22 11:01:41 +08:00
ea61206233 [pick](branch-2.1) pick #43281 (#44020)
pick #43281
2024-11-16 21:53:21 +08:00
10009dc062 [fix](variant) fix index in variant (#43375) (#43971)
Problem Summary:

1. Fixing error checks,
`InvertedIndexColumnWriter::check_support_inverted_index`, It is not
appropriate to determine support for other indexes by checking the
inverted index.
2. Fix `TableSchema::update_index()`

pick from master #43375
2024-11-16 16:29:00 +08:00
a519702fac branch-2.1: [Bug](bitmap-filter) fix wrong type cast on BitmapFilterColumnPredicate::evaluate #43877 (#43886)
Cherry-picked from #43877

Co-authored-by: Pxl <xl@selectdb.com>
2024-11-15 00:02:51 +08:00
d4712aed1a branch-2.1: [fix](string64) fix coredump caused by ColumnArray<ColumnStr<uint64_t>>::insert_indices_from (#43862)
Cherry-picked from #43624

Co-authored-by: TengJianPing <tengjianping@selectdb.com>
2024-11-13 19:31:11 +08:00
1101fbaf04 [fix](column_complex) wrong type of Field returned by ColumnComplex (#43515) (#43860) 2024-11-13 19:07:00 +08:00
6ecd55fa9e [cherry-pick](branch-2.1) Pick "[Fix](table size) Fix MoW table merge data fault (#40880)" (#43610) 2024-11-13 14:43:18 +08:00
15f85e2cfb [fix] (bloom filter) Fix the bloom filter calculation for date and datetime (#43351) (#43622)
pick from master #43351

---------

Co-authored-by: csun5285 <sunchenyang@selectdb.com>
2024-11-12 10:56:55 +08:00