database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,815 Commits 1 Branch 0 Tags
fa3d6f0ffd8632c1dfbfcd8e259583df3de4bb6d
Commit Graph

1475 Commits

Author SHA1 Message Date
Sun Chenyang
cf2ee23e68 Revert "[fix] (compaction) fix compaction score in time series policy (#40242)" (#44527) 
revert #40242
 2024-11-26 12:18:59 +08:00
github-actions[bot]
dceaf97381 branch-2.1: [enhance](orc) Optimize ORC Predicate Pushdown for OR-connected Predicate #43255 (#44438) 
Cherry-picked from #43255

Co-authored-by: Socrates <suyiteng@selectdb.com>
 2024-11-22 22:52:53 +08:00
daidai
702abbff0f [Opt](orc)Optimize the merge io when orc reader read multiple tiny stripes. (#42004) (#44239) 
bp #42004

Co-authored-by: kaka11chen <kaka11.chen@gmail.com>
 2024-11-22 11:01:41 +08:00
Xinyi Zou
ea61206233 [pick](branch-2.1) pick #43281 (#44020) 
pick #43281
 2024-11-16 21:53:21 +08:00
Sun Chenyang
10009dc062 [fix](variant) fix index in variant (#43375) (#43971) 
Problem Summary:

1. Fixing error checks,
`InvertedIndexColumnWriter::check_support_inverted_index`, It is not
appropriate to determine support for other indexes by checking the
inverted index.
2. Fix `TableSchema::update_index()`

pick from master #43375
 2024-11-16 16:29:00 +08:00
github-actions[bot]
a519702fac branch-2.1: [Bug](bitmap-filter) fix wrong type cast on BitmapFilterColumnPredicate::evaluate #43877 (#43886) 
Cherry-picked from #43877

Co-authored-by: Pxl <xl@selectdb.com>
 2024-11-15 00:02:51 +08:00
github-actions[bot]
d4712aed1a branch-2.1: [fix](string64) fix coredump caused by ColumnArray<ColumnStr<uint64_t>>::insert_indices_from (#43862) 
Cherry-picked from #43624

Co-authored-by: TengJianPing <tengjianping@selectdb.com>
 2024-11-13 19:31:11 +08:00
Jerry Hu
1101fbaf04 [fix](column_complex) wrong type of Field returned by ColumnComplex (#43515) (#43860) 2024-11-13 19:07:00 +08:00
abmdocrt
6ecd55fa9e [cherry-pick](branch-2.1) Pick "[Fix](table size) Fix MoW table merge data fault (#40880)" (#43610) 2024-11-13 14:43:18 +08:00
Sun Chenyang
15f85e2cfb [fix] (bloom filter) Fix the bloom filter calculation for date and datetime (#43351) (#43622) 
pick from master #43351

---------

Co-authored-by: csun5285 <sunchenyang@selectdb.com>
 2024-11-12 10:56:55 +08:00
lihangyu
e073b575cc [Opt](TabletSchema) reuse TabletColumn info to reduce mem (#42448) (#43349) 
(#42448)
 2024-11-11 10:38:42 +08:00
zclllhhjj
d933956449 [branch-2.1](timezone) Preload time offset in datetime (#42395) (#42607) 
pick https://github.com/apache/doris/pull/42395
 2024-11-10 00:30:28 +08:00
Xinyi Zou
9d7bc5b765 [pick](branch-2.1) pick #38215 (#43386) 
pick #38215

---------

Co-authored-by: Zou Xinyi <zouxinyi@selectdb.com>
 2024-11-09 22:13:05 +08:00
Siyang Tang
90da65c7b8 [fix](block-reader) Make rowsets union iterating work (#40877) (#43175) 
pick: #40877
 2024-11-08 10:05:10 +08:00
github-actions[bot]
46afbfca01 branch-2.1: [fix](ip) fix datatype serde for ipv6 with rowstore (#43252) 
Cherry-picked from #43065

Co-authored-by: amory <wangqiannan@selectdb.com>
 2024-11-05 20:09:14 +08:00
qiye
72bb6e79e4 [fix](index compaction)Skip writing terms with a doc frequency of 0(#43113) (#43115) 
bp #43113
 2024-11-04 17:49:56 +08:00
Kaijie Chen
25d7d0b255 [fix](move-memtable) abstract multi-streams to one logical stream (#42039) (#42250) 
backport #42039
 2024-10-22 20:26:42 +08:00
Socrates
38e529cd29 [cherry-pick](branch-2.1) support decimal256 for parquet reader (#42241) 
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41526
 2024-10-22 19:42:09 +08:00
zclllhhjj
7eec0f8fbb [branch-2.1](datetime) Fix date floor functions overflow (#35477) (#42238) 
pick https://github.com/apache/doris/pull/35477
 2024-10-22 15:54:53 +08:00
zzzxl
d5fef266ec [fix](inverted index) Fix incorrect exception handling (#42094) 
https://github.com/apache/doris/pull/41874
 2024-10-19 10:45:32 +08:00
Socrates
1b901f6fcc [cherry-pick](branch-2.1) add parquet tvf cases and fix some parquet bug (#41931) 
## Proposed changes
pick pr:
  https://github.com/apache/doris/pull/41683
  https://github.com/apache/doris/pull/41506
  https://github.com/apache/doris/pull/41338
  https://github.com/apache/doris/pull/39326

---------

Co-authored-by: morningman <morningman@163.com>
 2024-10-17 14:20:58 +08:00
Gabriel
5bd33fc88c [pick](branch-2.1) pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751 (#41927) 
## Proposed changes

pick #41292 #41350 #41589 #41628 #41743 #41601 #41667 #41751

<!--Describe your changes.-->

---------

Co-authored-by: Pxl <pxl290@qq.com>
 2024-10-16 15:41:28 +08:00
Mryange
6dddd4c499 [function](cast)Make string casting to integers more like MySQL's beh… (#41541) 
…avior (#38847)
https://github.com/apache/doris/pull/38847
## Proposed changes

There are two issues here. First, the results of casting are
inconsistent between FE and BE .
```
FE
mysql [(none)]>select cast('3.000' as int); 
+----------------------+
| cast('3.000' as INT) |
+----------------------+
|                    3 |
+----------------------+

mysql [(none)]>set debug_skip_fold_constant = true;

BE
mysql [(none)]>select cast('3.000' as int);
+----------------------+
| cast('3.000' as INT) |
+----------------------+
|                 NULL |
+----------------------+
```
The second issue is that casting on BE converts '3.0' to null. Here, the
casting logic for FE and BE has been unified

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

---------

Co-authored-by: Xinyi Zou <zouxinyi02@gmail.com>
 2024-10-11 09:32:00 +08:00
Socrates
0b4552f74b [cherry-pick](branch-2.1) pick hive text write from master (#40537) 
## Proposed changes
pick prs:
https://github.com/apache/doris/pull/38549
https://github.com/apache/doris/pull/40183
https://github.com/apache/doris/pull/40315

---------

Co-authored-by: Calvin Kirs <kirs@apache.org>
 2024-09-27 20:57:07 +08:00
hui lai
d4c1b39d03 [fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables (#39992) (#41131) 
pick (#39992)

BE node was killed by OOM-killer when use multi table load under high
concurrency with a large number of tables(128 concurrency and every
concurrency load 200 tables).

This pr restricts the multi tables load memory under this issue. If
memory reaches hard limit, new task will be rejected and return
directly.
 2024-09-24 16:34:32 +08:00
Xinyi Zou
b52b572ade [branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850) 
pick
#38960
#39908
#40043
#40092
#40016
#40439

---------

Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
 2024-09-15 23:47:53 +08:00
zclllhhjj
cecd214345 [branch-2.1](Column) refactor ColumnNullable to provide flags safety (#40769) (#40848) 
pick https://github.com/apache/doris/pull/40769

Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
 2024-09-14 16:27:43 +08:00
Sun Chenyang
873f70c262 [fix] (compaction) fix compaction score in time series policy (#40242) (#40779) 
## Proposed changes

pick from master #40242

<!--Describe your changes.-->
 2024-09-13 14:35:16 +08:00
zhannngchen
023317e8a0 [fix](ut) fix SegmentTest and SegmentMoWTest asan core (#40287) (#40622) 
## Proposed changes

pick #40287
 2024-09-10 18:37:00 +08:00
Xinyi Zou
354967c09f [branch-2.1](memory) pick reserve memory and workload group (#40543) 
1. pick #38494
2. pick #39862
3. remove vdata_stream_test, master has been removed
 2024-09-09 21:16:06 +08:00
Socrates
87ac378c4a [branch-2.1](be-ut) wait lazy open in ut (#40453) 
## Proposed changes

LRUFileCache test need to wait lazy open done
 2024-09-06 09:47:47 +08:00
Luwei
cc20ecd738 Revert "[fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728) (#39262)" (#40375) 
This reverts commit c9949f24e5c15e9529285f0e99b7ffdb1095558b.

This pr may increase the probability of full clone failure, so revert it
first.
 2024-09-05 00:01:03 +08:00
yiguolei
ca07a00c93 Revert "[branch-2.1](hive) support hive write text table (#38549) (#4… (#40157) 
…0063)"

This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-08-30 10:25:38 +08:00
Socrates
c6df7c21a3 [branch-2.1](hive) support hive write text table (#38549) (#40063) 
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd

pick from https://github.com/apache/doris/pull/38549
 2024-08-29 16:50:40 +08:00
daidai
ac8cff34ce [fix](ut)fix be enable_http_auth ut (#40071) (#40088) 
bp #40071 

## Proposed changes
before pr #39577
 2024-08-29 16:42:53 +08:00
Xinyi Zou
9d5468d198 [branch-2.1](memory) BE memory info compatible with CgroupV2 (#39799) 
pick #39256
 2024-08-23 02:03:00 +08:00
Xinyi Zou
8ce8887b75 [branch-2.1](memory) Refactor refresh workload groups weighted memory ratio and record refresh interval memory growth (#39760) 
pick #38168
overwrites changes in #37221 on workload_group_manager.cpp. If need to
pick 37221, ignore it.
 2024-08-22 17:33:11 +08:00
zhannngchen
0e694f19db [fix](merge-on-write) segcompaction should process delete bitmap if necessary (#38369) (#39707) 
## Proposed changes

Issue Number: close #xxx

cherry-pick #38369 and #38800
 2024-08-22 00:42:56 +08:00
Socrates
bb687bd69c [cherry-pick](branch-2.1) add function regexp_extract_or_null (#39561) 
# Proposed changes

pick https://github.com/apache/doris/pull/38296
 2024-08-21 09:14:58 +08:00
Sun Chenyang
fb17f204d7 [fix](http) fix http url with incorrect character notation (#38420) (#39535) 
## Proposed changes

pick from master #38420
 2024-08-19 15:03:19 +08:00
TengJianPing
021678c7c3 [fix](window_funnel) fix wrong result of window_funnel #38954 (#39270) 
## Proposed changes

BP #38954
 2024-08-16 09:59:31 +08:00
Qi Chen
a44a274563 [Fix](parquet-reader) Fix and optimize parquet min-max filtering. (#39375) 
Backport #38277.
 2024-08-15 14:12:54 +08:00
Luwei
c9949f24e5 [fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728) (#39262) 
pick master #38728
 2024-08-13 17:41:11 +08:00
yangshijie
5f77f909d9 [cherry-pick](branch-2.1) Pick "[feature](function) support ip functions named ipv4_to_ipv6 and cut_ipv6" (#39058) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
pick https://github.com/apache/doris/pull/36883 and
https://github.com/apache/doris/pull/35239
 2024-08-10 18:37:11 +08:00
zhannngchen
8a682d43ec [fix](ut) repair segcompaction ut (#38165) (#38225) 
cherry-pick #38165
 2024-08-09 15:52:18 +08:00
lihangyu
773008d6fa [Fix](Json) fix some cast issue (#38683) (#39025) 
#38683
 2024-08-07 22:05:43 +08:00
Luwei
0603ec1d9d [enhancement](compaction) optimizing memory usage for compaction (#37099) (#37486) 2024-08-04 10:49:18 +08:00
zclllhhjj
79a6496bb6 [branch-2.1](function) fix wrong result when convert_tz is out of bound (#37358) (#38313) 
## Proposed changes

pick https://github.com/apache/doris/pull/37358

before:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001'  as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| q535-12-31 08:01:19                                                                                                                               |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.12 sec)
```
now:
```sql
mysql> select CONVERT_TZ(cast('0000-01-01 00:00:00.00001'  as DATETIMEV1), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533)));
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| convert_tz(cast('0000-01-01 00:00:00.00001' as DATETIME), cast('Asia/Shanghai' as VARCHAR(65533)), cast('America/Los_Angeles' as VARCHAR(65533))) |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
| NULL                                                                                                                                              |
+---------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.09 sec)
```
 2024-07-25 11:32:44 +08:00
bobhan1
c30c1d2436 [branch-2.1] Picks "[opt](delete) Delete job should retry for failure that is not DELETE_INVALID_XXX #37834" (#38032) 
## Proposed changes

picks https://github.com/apache/doris/pull/37834 and
https://github.com/apache/doris/pull/38043
 2024-07-18 14:50:30 +08:00
zhiqiang
02716598d4 [Fix](sql function) memory overflow to the left of string address when do_money_format has small negative value #36226 (#37870) 
cherry pick from #36226

Co-authored-by: sparrow <38098988+biohazard4321@users.noreply.github.com>
 2024-07-16 15:04:42 +08:00