94746b764d
branch-2.1: [fix](memory) Fix UT ThreadMemTrackerMgrTest #44147 ( #44936 )
...
Cherry-picked from #44147
Co-authored-by: Xinyi Zou <zouxinyi@selectdb.com >
2024-12-06 10:54:12 +08:00
0c3bc1bcd0
[fix](auth)fix be enable http auth, some request link never return. ( #44959 ) ( #45063 )
...
bp #44959
if you `enable_all_http_auth = true` in be.conf, then restart be, and
keep using `curl -u "xxxx:xxxx" http://127.0.0.1:8040/api/health ` while
be is starting. You may encounter a situation where the link does not
return.
Reason:
When be is still starting, there is no information about fe master. When
you make an api request to be http port, be needs to request
authentication information from fe, which will cause it to request a
machine with empty ip and port 0. This rpc call will definitely fail
(this is not equivalent to a password error). After receiving this
failure, be does not `send_reply` to the api requester, so this api
request cannot be returned.
2024-12-06 09:39:13 +08:00
5f952cf6ed
branch-2.1: [fix](iceberg)Bring field_id with parquet files And fix map type's key optional #44470 ( #44828 )
...
Cherry-picked from #44470
Co-authored-by: wuwenchi <wuwenchi@selectdb.com >
2024-12-02 10:24:07 +08:00
4b15b1f263
[fix](orc) check all the cases before build_search_argument ( #44615 ) ( #44801 )
...
cherry-pick #44615
Co-authored-by: Socrates <suyiteng@selectdb.com >
2024-11-30 09:17:56 +08:00
cf2ee23e68
Revert "[fix] (compaction) fix compaction score in time series policy ( #40242 )" ( #44527 )
...
revert #40242
2024-11-26 12:18:59 +08:00
dceaf97381
branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255 ( #44438 )
...
Cherry-picked from #43255
Co-authored-by: Socrates <suyiteng@selectdb.com >
2024-11-22 22:52:53 +08:00
702abbff0f
[Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. ( #42004 ) ( #44239 )
...
bp #42004
Co-authored-by: kaka11chen <kaka11.chen@gmail.com >
2024-11-22 11:01:41 +08:00
ea61206233
[pick](branch-2.1) pick #43281 ( #44020 )
...
pick #43281
2024-11-16 21:53:21 +08:00
10009dc062
[fix](variant) fix index in variant ( #43375 ) ( #43971 )
...
Problem Summary:
1. Fixing error checks,
`InvertedIndexColumnWriter::check_support_inverted_index`, It is not
appropriate to determine support for other indexes by checking the
inverted index.
2. Fix `TableSchema::update_index()`
pick from master #43375
2024-11-16 16:29:00 +08:00
a519702fac
branch-2.1: [Bug](bitmap-filter) fix wrong type cast on BitmapFilterColumnPredicate::evaluate #43877 ( #43886 )
...
Cherry-picked from #43877
Co-authored-by: Pxl <xl@selectdb.com >
2024-11-15 00:02:51 +08:00
d4712aed1a
branch-2.1: [fix](string64) fix coredump caused by ColumnArray<ColumnStr<uint64_t>>::insert_indices_from ( #43862 )
...
Cherry-picked from #43624
Co-authored-by: TengJianPing <tengjianping@selectdb.com >
2024-11-13 19:31:11 +08:00
1101fbaf04
[fix](column_complex) wrong type of Field returned by ColumnComplex ( #43515 ) ( #43860 )
2024-11-13 19:07:00 +08:00
6ecd55fa9e
[cherry-pick](branch-2.1) Pick "[Fix](table size) Fix MoW table merge data fault ( #40880 )" ( #43610 )
2024-11-13 14:43:18 +08:00
15f85e2cfb
[fix] (bloom filter) Fix the bloom filter calculation for date and datetime ( #43351 ) ( #43622 )
...
pick from master #43351
---------
Co-authored-by: csun5285 <sunchenyang@selectdb.com >
2024-11-12 10:56:55 +08:00
e073b575cc
[Opt](TabletSchema) reuse TabletColumn info to reduce mem ( #42448 ) ( #43349 )
...
(#42448 )
2024-11-11 10:38:42 +08:00
d933956449
[branch-2.1](timezone) Preload time offset in datetime ( #42395 ) ( #42607 )
...
pick https://github.com/apache/doris/pull/42395
2024-11-10 00:30:28 +08:00
9d7bc5b765
[pick](branch-2.1) pick #38215 ( #43386 )
...
pick #38215
---------
Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com >
2024-11-09 22:13:05 +08:00
90da65c7b8
[fix](block-reader) Make rowsets union iterating work ( #40877 ) ( #43175 )
...
pick: #40877
2024-11-08 10:05:10 +08:00
46afbfca01
branch-2.1: [fix](ip) fix datatype serde for ipv6 with rowstore ( #43252 )
...
Cherry-picked from #43065
Co-authored-by: amory <wangqiannan@selectdb.com >
2024-11-05 20:09:14 +08:00
72bb6e79e4
[fix](index compaction)Skip writing terms with a doc frequency of 0( #43113 ) ( #43115 )
...
bp #43113
2024-11-04 17:49:56 +08:00
25d7d0b255
[fix](move-memtable) abstract multi-streams to one logical stream ( #42039 ) ( #42250 )
...
backport #42039
2024-10-22 20:26:42 +08:00
38e529cd29
[cherry-pick](branch-2.1) support decimal256 for parquet reader ( #42241 )
...
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41526
2024-10-22 19:42:09 +08:00
7eec0f8fbb
[branch-2.1](datetime) Fix date floor functions overflow ( #35477 ) ( #42238 )
...
pick https://github.com/apache/doris/pull/35477
2024-10-22 15:54:53 +08:00
d5fef266ec
[fix](inverted index) Fix incorrect exception handling ( #42094 )
...
https://github.com/apache/doris/pull/41874
2024-10-19 10:45:32 +08:00
1b901f6fcc
[cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet bug ( #41931 )
...
## Proposed changes
pick pr:
https://github.com/apache/doris/pull/41683
https://github.com/apache/doris/pull/41506
https://github.com/apache/doris/pull/41338
https://github.com/apache/doris/pull/39326
---------
Co-authored-by: morningman <morningman@163.com >
2024-10-17 14:20:58 +08:00
5bd33fc88c
[pick](branch-2.1) pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751 ( #41927 )
...
## Proposed changes
pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751
<!--Describe your changes.-->
---------
Co-authored-by: Pxl <pxl290@qq.com >
2024-10-16 15:41:28 +08:00
6dddd4c499
[function](cast)Make string casting to integers more like MySQL's beh… ( #41541 )
...
…avior (#38847 )
https://github.com/apache/doris/pull/38847
## Proposed changes
There are two issues here. First, the results of casting are
inconsistent between FE and BE .
```
FE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
| 3 |
+----------------------+
mysql [(none)]>set debug_skip_fold_constant = true;
BE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
| NULL |
+----------------------+
```
The second issue is that casting on BE converts '3.0' to null. Here, the
casting logic for FE and BE has been unified
<!--Describe your changes.-->
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
---------
Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com >
2024-10-11 09:32:00 +08:00
0b4552f74b
[cherry-pick](branch-2.1) pick hive text write from master ( #40537 )
...
## Proposed changes
pick prs:
https://github.com/apache/doris/pull/38549
https://github.com/apache/doris/pull/40183
https://github.com/apache/doris/pull/40315
---------
Co-authored-by: Calvin Kirs <kirs@apache.org >
2024-09-27 20:57:07 +08:00
d4c1b39d03
[fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables ( #39992 ) ( #41131 )
...
pick (#39992 )
BE node was killed by OOM-killer when use multi table load under high
concurrency with a large number of tables(128 concurrency and every
concurrency load 200 tables).
This pr restricts the multi tables load memory under this issue. If
memory reaches hard limit, new task will be rejected and return
directly.
2024-09-24 16:34:32 +08:00
b52b572ade
[branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 ( #40850 )
...
pick
#38960
#39908
#40043
#40092
#40016
#40439
---------
Co-authored-by: hui lai <1353307710@qq.com >
Co-authored-by: yiguolei <676222867@qq.com >
2024-09-15 23:47:53 +08:00
cecd214345
[branch-2.1](Column) refactor ColumnNullable to provide flags safety ( #40769 ) ( #40848 )
...
pick https://github.com/apache/doris/pull/40769
Co-authored-by: Jerry Hu <mrhhsg@gmail.com >
2024-09-14 16:27:43 +08:00
873f70c262
[fix] (compaction) fix compaction score in time series policy ( #40242 ) ( #40779 )
...
## Proposed changes
pick from master #40242
<!--Describe your changes.-->
2024-09-13 14:35:16 +08:00
023317e8a0
[fix](ut) fix SegmentTest and SegmentMoWTest asan core ( #40287 ) ( #40622 )
...
## Proposed changes
pick #40287
2024-09-10 18:37:00 +08:00
354967c09f
[branch-2.1](memory) pick reserve memory and workload group ( #40543 )
...
1. pick #38494
2. pick #39862
3. remove vdata_stream_test, master has been removed
2024-09-09 21:16:06 +08:00
87ac378c4a
[branch-2.1](be-ut) wait lazy open in ut ( #40453 )
...
## Proposed changes
LRUFileCache test need to wait lazy open done
2024-09-06 09:47:47 +08:00
cc20ecd738
Revert "[fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets ( #38728 ) ( #39262 )" ( #40375 )
...
This reverts commit c9949f24e5c15e9529285f0e99b7ffdb1095558b.
This pr may increase the probability of full clone failure, so revert it
first.
2024-09-05 00:01:03 +08:00
ca07a00c93
Revert "[branch-2.1](hive) support hive write text table ( #38549 ) (#4… ( #40157 )
...
…0063)"
This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-08-30 10:25:38 +08:00
c6df7c21a3
[branch-2.1](hive) support hive write text table ( #38549 ) ( #40063 )
...
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd
pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
ac8cff34ce
[fix](ut)fix be enable_http_auth ut ( #40071 ) ( #40088 )
...
bp #40071
## Proposed changes
before pr #39577
2024-08-29 16:42:53 +08:00
9d5468d198
[branch-2.1](memory) BE memory info compatible with CgroupV2 ( #39799 )
...
pick #39256
2024-08-23 02:03:00 +08:00
8ce8887b75
[branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth ( #39760 )
...
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
2024-08-22 17:33:11 +08:00
0e694f19db
[fix](merge-on-write) segcompaction should process delete bitmap if necessary ( #38369 ) ( #39707 )
...
## Proposed changes
Issue Number: close #xxx
cherry-pick #38369 and #38800
2024-08-22 00:42:56 +08:00
bb687bd69c
[cherry-pick](branch-2.1) add function regexp_extract_or_null ( #39561 )
...
# Proposed changes
pick https://github.com/apache/doris/pull/38296
2024-08-21 09:14:58 +08:00
fb17f204d7
[fix](http) fix http url with incorrect character notation ( #38420 ) ( #39535 )
...
## Proposed changes
pick from master #38420
2024-08-19 15:03:19 +08:00
021678c7c3
[fix](window_funnel) fix wrong result of window_funnel #38954 ( #39270 )
...
## Proposed changes
BP #38954
2024-08-16 09:59:31 +08:00
a44a274563
[Fix](parquet-reader) Fix and optimize parquet min-max filtering. ( #39375 )
...
Backport #38277 .
2024-08-15 14:12:54 +08:00
c9949f24e5
[fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets ( #38728 ) ( #39262 )
...
pick master #38728
2024-08-13 17:41:11 +08:00
5f77f909d9
[cherry-pick](branch-2.1) Pick "[feature](function) support ip functions named ipv4_to_ipv6 and cut_ipv6" ( #39058 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
pick https://github.com/apache/doris/pull/36883 and
https://github.com/apache/doris/pull/35239
2024-08-10 18:37:11 +08:00
8a682d43ec
[fix](ut) repair segcompaction ut ( #38165 ) ( #38225 )
...
cherry-pick #38165
2024-08-09 15:52:18 +08:00
773008d6fa
[Fix](Json) fix some cast issue ( #38683 ) ( #39025 )
...
#38683
2024-08-07 22:05:43 +08:00
