Commit Graph

1492 Commits

Author SHA1 Message Date
4472648c07 [branch-2.1] pick workload group usage metrics (#46177)
pick #45284  #44870
2024-12-31 10:09:48 +08:00
df26475e1a [Enhancement](compaction) enable the compaction producer to generate multiple compaction tasks in a single run (#45411) (#46160)
pick master #45411
2024-12-31 09:51:43 +08:00
Pxl
43c646363e [Bug](runtime-filter) support ip rf and use exception to replace dche… (#41531)
…ck when PrimitiveType to PColumnType (#39985)

use exception to replace dcheck when PrimitiveType to PColumnType
```cpp
*** SIGABRT unknown detail explain (@0x11d3f) received by PID 73023 (TID 74292 OR 0x7fd758225640) from PID 73023; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
 1# 0x00007FDDBE6B9520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill at ./nptl/pthread_kill.c:89
 3# raise at ../sysdeps/posix/raise.c:27
 4# abort at ./stdlib/abort.c:81
 5# 0x000056123F81A94D in /root/output/be/lib/doris_be
 6# 0x000056123F80CF8A in /root/output/be/lib/doris_be
 7# google::LogMessage::SendToLog() in /root/output/be/lib/doris_be
 8# google::LogMessage::Flush() in /root/output/be/lib/doris_be
 9# google::LogMessageFatal::~LogMessageFatal() in /root/output/be/lib/doris_be
10# doris::to_proto(doris::PrimitiveType) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:114
11# doris::IRuntimeFilter::push_to_remote(doris::TNetworkAddress const*) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:1143
12# doris::IRuntimeFilter::publish(bool)::$_0::operator()(doris::IRuntimeFilter*) const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:959
13# doris::IRuntimeFilter::publish(bool)::$_2::operator()() const at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:983
14# doris::IRuntimeFilter::publish(bool) at /home/zcp/repo_center/doris_master/doris/be/src/exprs/runtime_filter.cpp:997
```

## Proposed changes
pick from #39985
2024-12-30 20:56:11 +08:00
d2c108726d [opt](bloomfilter index) optimize memory usage for bloom filter index writer #45833 (#46047)
cherry pick from #45833
2024-12-27 12:10:56 +08:00
df8bc8f23d branch-2.1: [fix](parquet) impl has_dict_page to replace old logic and fix write empty parquet row group bug #45740 (#45954)
Cherry-picked from #45740

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-12-26 15:17:49 +08:00
1cf6986cea [pick](branch-2.1) pick #44092 (#45836) 2024-12-25 23:11:19 +08:00
64195d79ee [refactor](metrics) Remove IntAtomicCounter & CoreLocal #45742 (#45870)
cherry pick from #45742
2024-12-24 23:13:48 +08:00
02f15a8ef0 [fix](inverted index) Fix Null Pointer Exception in function match(#45456)(#45774)
pick: https://github.com/apache/doris/pull/45456
2024-12-24 11:27:13 +08:00
79662fcc94 [branch-2.1](functions) clean some ip functions code and make IS_IP_ADDRESS_IN_RANGE DEPENDS_ON_ARGUMENT (#45358)
pick https://github.com/apache/doris/pull/35239


add special logic to deal smooth upgrade

The origin PR is https://github.com/apache/doris/pull/35239. for
branch-3.0 it was merged in 3.0.0 but forgot to register old version.
now in branch-3.0 we fix it in
https://github.com/apache/doris/pull/45428 which must be merged in
3.0.4. and do same thing in this PR which must be merged in 2.1.8.
then:
```
FROM    TO    result
217-    218+    
217-    303-    💥
218+    303-    
218+    304+    
303-    304+    
```
this is our best result.
2024-12-17 11:51:07 +08:00
667f5e6e6a [feat](iceberg)Supports using rest type catalog to read tables in unity catalog for 2.1 (#43525) (#45217)
bp: #43525
2024-12-12 00:49:36 -08:00
5d3f0a267a [opt](scan) unify the local and remote scan bytes stats for all scanners for 2.1 (#45167)
pick part of #40493

TODO: not working with s3 reader
2024-12-10 14:19:19 +08:00
0bcb99710b [branch-2.1]Add customStdAllocator for vector/map (#41193) (#45124) 2024-12-09 18:33:51 +08:00
d4a6fd1850 Revert #43255 & #44615 (#45096)
Revert "branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for
OR-connected Predicate #43255 (#44438)"
Revert "[fix](orc) check all the cases before build_search_argument
(#44615) (#44801)"
2024-12-06 21:14:13 +08:00
94746b764d branch-2.1: [fix](memory) Fix UT ThreadMemTrackerMgrTest #44147 (#44936)
Cherry-picked from #44147

Co-authored-by: Xinyi Zou <zouxinyi@selectdb.com>
2024-12-06 10:54:12 +08:00
0c3bc1bcd0 [fix](auth)fix be enable http auth, some request link never return. (#44959) (#45063)
bp #44959

if you `enable_all_http_auth = true` in be.conf, then restart be, and
keep using `curl -u "xxxx:xxxx" http://127.0.0.1:8040/api/health` while
be is starting. You may encounter a situation where the link does not
return.
Reason:
When be is still starting, there is no information about fe master. When
you make an api request to be http port, be needs to request
authentication information from fe, which will cause it to request a
machine with empty ip and port 0. This rpc call will definitely fail
(this is not equivalent to a password error). After receiving this
failure, be does not `send_reply` to the api requester, so this api
request cannot be returned.
2024-12-06 09:39:13 +08:00
5f952cf6ed branch-2.1: [fix](iceberg)Bring field_id with parquet files And fix map type's key optional #44470 (#44828)
Cherry-picked from #44470

Co-authored-by: wuwenchi <wuwenchi@selectdb.com>
2024-12-02 10:24:07 +08:00
4b15b1f263 [fix](orc) check all the cases before build_search_argument (#44615) (#44801)
cherry-pick #44615

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-30 09:17:56 +08:00
cf2ee23e68 Revert "[fix] (compaction) fix compaction score in time series policy (#40242)" (#44527)
revert #40242
2024-11-26 12:18:59 +08:00
dceaf97381 branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255 (#44438)
Cherry-picked from #43255

Co-authored-by: Socrates <suyiteng@selectdb.com>
2024-11-22 22:52:53 +08:00
702abbff0f [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) (#44239)
bp #42004

Co-authored-by: kaka11chen <kaka11.chen@gmail.com>
2024-11-22 11:01:41 +08:00
ea61206233 [pick](branch-2.1) pick #43281 (#44020)
pick #43281
2024-11-16 21:53:21 +08:00
10009dc062 [fix](variant) fix index in variant (#43375) (#43971)
Problem Summary:

1. Fixing error checks,
`InvertedIndexColumnWriter::check_support_inverted_index`, It is not
appropriate to determine support for other indexes by checking the
inverted index.
2. Fix `TableSchema::update_index()`

pick from master #43375
2024-11-16 16:29:00 +08:00
a519702fac branch-2.1: [Bug](bitmap-filter) fix wrong type cast on BitmapFilterColumnPredicate::evaluate #43877 (#43886)
Cherry-picked from #43877

Co-authored-by: Pxl <xl@selectdb.com>
2024-11-15 00:02:51 +08:00
d4712aed1a branch-2.1: [fix](string64) fix coredump caused by ColumnArray<ColumnStr<uint64_t>>::insert_indices_from (#43862)
Cherry-picked from #43624

Co-authored-by: TengJianPing <tengjianping@selectdb.com>
2024-11-13 19:31:11 +08:00
1101fbaf04 [fix](column_complex) wrong type of Field returned by ColumnComplex (#43515) (#43860) 2024-11-13 19:07:00 +08:00
6ecd55fa9e [cherry-pick](branch-2.1) Pick "[Fix](table size) Fix MoW table merge data fault (#40880)" (#43610) 2024-11-13 14:43:18 +08:00
15f85e2cfb [fix] (bloom filter) Fix the bloom filter calculation for date and datetime (#43351) (#43622)
pick from master #43351

---------

Co-authored-by: csun5285 <sunchenyang@selectdb.com>
2024-11-12 10:56:55 +08:00
e073b575cc [Opt](TabletSchema) reuse TabletColumn info to reduce mem (#42448) (#43349)
(#42448)
2024-11-11 10:38:42 +08:00
d933956449 [branch-2.1](timezone) Preload time offset in datetime (#42395) (#42607)
pick https://github.com/apache/doris/pull/42395
2024-11-10 00:30:28 +08:00
9d7bc5b765 [pick](branch-2.1) pick #38215 (#43386)
pick #38215

---------

Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com>
2024-11-09 22:13:05 +08:00
90da65c7b8 [fix](block-reader) Make rowsets union iterating work (#40877) (#43175)
pick: #40877
2024-11-08 10:05:10 +08:00
46afbfca01 branch-2.1: [fix](ip) fix datatype serde for ipv6 with rowstore (#43252)
Cherry-picked from #43065

Co-authored-by: amory <wangqiannan@selectdb.com>
2024-11-05 20:09:14 +08:00
72bb6e79e4 [fix](index compaction)Skip writing terms with a doc frequency of 0(#43113) (#43115)
bp #43113
2024-11-04 17:49:56 +08:00
25d7d0b255 [fix](move-memtable) abstract multi-streams to one logical stream (#42039) (#42250)
backport #42039
2024-10-22 20:26:42 +08:00
38e529cd29 [cherry-pick](branch-2.1) support decimal256 for parquet reader (#42241)
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41526
2024-10-22 19:42:09 +08:00
7eec0f8fbb [branch-2.1](datetime) Fix date floor functions overflow (#35477) (#42238)
pick https://github.com/apache/doris/pull/35477
2024-10-22 15:54:53 +08:00
d5fef266ec [fix](inverted index) Fix incorrect exception handling (#42094)
https://github.com/apache/doris/pull/41874
2024-10-19 10:45:32 +08:00
1b901f6fcc [cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet bug (#41931)
## Proposed changes
pick pr:
  https://github.com/apache/doris/pull/41683
  https://github.com/apache/doris/pull/41506
  https://github.com/apache/doris/pull/41338
  https://github.com/apache/doris/pull/39326

---------

Co-authored-by: morningman <morningman@163.com>
2024-10-17 14:20:58 +08:00
5bd33fc88c [pick](branch-2.1) pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751 (#41927)
## Proposed changes

pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751

<!--Describe your changes.-->

---------

Co-authored-by: Pxl <pxl290@qq.com>
2024-10-16 15:41:28 +08:00
6dddd4c499 [function](cast)Make string casting to integers more like MySQL's beh… (#41541)
…avior (#38847)
https://github.com/apache/doris/pull/38847
## Proposed changes

There are two issues here. First, the results of casting are
inconsistent between FE and BE .
```
FE
mysql [(none)]>select cast('3.000' as int); 
+----------------------+
| cast('3.000' as INT) |
+----------------------+
|                    3 |
+----------------------+

mysql [(none)]>set debug_skip_fold_constant = true;

BE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
|                 NULL |
+----------------------+
```
The second issue is that casting on BE converts '3.0' to null. Here, the
casting logic for FE and BE has been unified

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

---------

Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
2024-10-11 09:32:00 +08:00
0b4552f74b [cherry-pick](branch-2.1) pick hive text write from master (#40537)
## Proposed changes
pick prs:
https://github.com/apache/doris/pull/38549
https://github.com/apache/doris/pull/40183
https://github.com/apache/doris/pull/40315

---------

Co-authored-by: Calvin Kirs <kirs@apache.org>
2024-09-27 20:57:07 +08:00
d4c1b39d03 [fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables (#39992) (#41131)
pick (#39992)

BE node was killed by OOM-killer when use multi table load under high
concurrency with a large number of tables(128 concurrency and every
concurrency load 200 tables).

This pr restricts the multi tables load memory under this issue. If
memory reaches hard limit, new task will be rejected and return
directly.
2024-09-24 16:34:32 +08:00
b52b572ade [branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850)
pick
#38960
#39908
#40043
#40092
#40016
#40439

---------

Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
2024-09-15 23:47:53 +08:00
cecd214345 [branch-2.1](Column) refactor ColumnNullable to provide flags safety (#40769) (#40848)
pick https://github.com/apache/doris/pull/40769

Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
2024-09-14 16:27:43 +08:00
873f70c262 [fix] (compaction) fix compaction score in time series policy (#40242) (#40779)
## Proposed changes

pick from master #40242

<!--Describe your changes.-->
2024-09-13 14:35:16 +08:00
023317e8a0 [fix](ut) fix SegmentTest and SegmentMoWTest asan core (#40287) (#40622)
## Proposed changes

pick #40287
2024-09-10 18:37:00 +08:00
354967c09f [branch-2.1](memory) pick reserve memory and workload group (#40543)
1. pick #38494
2. pick #39862
3. remove vdata_stream_test, master has been removed
2024-09-09 21:16:06 +08:00
87ac378c4a [branch-2.1](be-ut) wait lazy open in ut (#40453)
## Proposed changes

LRUFileCache test need to wait lazy open done
2024-09-06 09:47:47 +08:00
cc20ecd738 Revert "[fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728) (#39262)" (#40375)
This reverts commit c9949f24e5c15e9529285f0e99b7ffdb1095558b.

This pr may increase the probability of full clone failure, so revert it
first.
2024-09-05 00:01:03 +08:00
ca07a00c93 Revert "[branch-2.1](hive) support hive write text table (#38549) (#4… (#40157)
…0063)"

This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-08-30 10:25:38 +08:00