database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,239 Commits 1 Branch 0 Tags
4e4f3d204ecf30166e177a02ef38ec3e007e7dfd
Commit Graph

7895 Commits

Author SHA1 Message Date
daidai
c8978fc9d1 [fix](HadoopLz4BlockCompression)Fixed the bug that HadoopLz4BlockCompression creates _decompressor every time it decompresses.(#37187) (#37299) 
bp : #37187
 2024-07-04 20:22:27 +08:00
Pxl
e2c2702dff [Bug](runtime-filter) fix some rf error problems (#37155) 
## Proposed changes
pick from #37273
 2024-07-04 20:03:46 +08:00
wangbo
b272247a57 [pick]log thread num (#37258) 
## Proposed changes

pick #37159
 2024-07-04 15:27:52 +08:00
Mingyu Chen
ceef9ee123 [feature](serde) support presto compatible output format (#37039) (#37253) 
bp #37039
 2024-07-04 13:56:05 +08:00
TengJianPing
fb344b66ca [fix](hash join) fix numeric overflow when calculating hash table bucket size #37193 (#37213) 
## Proposed changes

Bp #37193
 2024-07-04 11:12:52 +08:00
Gabriel
4532ba990a [fix](pipeline) Avoid to close task twice (#36747) (#37115) 2024-07-04 10:02:56 +08:00
Pxl
70e1c563b3 [Chore](runtime-filter) enlarge sync filter size rpc timeout limit (#37103) (#37225) 
pick from #37103
 2024-07-03 21:02:26 +08:00
Pxl
ffc57c9ef4 [Bug](runtime-filter) fix brpc ctrl use after free (#37223) 
part of #35186
 2024-07-03 21:01:50 +08:00
zhannngchen
97945af947 [fix](merge-on-write) when full clone failed, duplicate key might occur (#37001) (#37229) 
cherry-pick #37001
 2024-07-03 19:48:10 +08:00
Tiewei Fang
0aeb768bf9 [Fix](export/outfile) Support compression when exporting data to Parquet / ORC. (#37167) 
bp: #36490
 2024-07-03 10:53:57 +08:00
Tiewei Fang
bd24a8bdd9 [Fix](csv_reader) Add a session variable to control whether empty rows in CSV files are read as NULL values (#37153) 
bp: #36668
 2024-07-02 22:12:17 +08:00
Mingyu Chen
e25717458e [opt](catalog) add some profile for parquet reader and change meta cache config (#37040) (#37146) 
bp #37040
 2024-07-02 20:58:43 +08:00
wangbo
f5572ac732 [pick]reset memtable flush thread num (#37092) 
## Proposed changes

pick #37028
 2024-07-02 19:20:17 +08:00
camby
239bc1a7e0 [fix](compile) fix compile failed on MacOS due to ambiguous std::abs (#37136) 
cherry-pick #35125 to branch-2.1

Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
 2024-07-02 17:45:33 +08:00
Mingyu Chen
f5d0cdeeb4 [branch-2.1] avoid glog coredump when running with ASAN (#37134) 
## Proposed changes

This is just a workround try avoid coredump like this:
```
#0 0x56414f0e8ed1 in __asan::CheckUnwind() crtstuff.c
    #1 0x56414f1009a2 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) crtstuff.c
    #2 0x56414f0ecbf3 in __asan::AsanThread::GetStackFrameAccessByAddr(unsigned long, __asan::AsanThread::StackFrameAccess*) crtstuff.c
    #3 0x56414f050d87 in __asan::AddressDescription::AddressDescription(unsigned long, unsigned long, bool) crtstuff.c
    #4 0x56414f052a73 in __asan::ErrorGeneric::ErrorGeneric(unsigned int, unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long) crtstuff.c
    #5 0x56414f0e6a9e in __asan::ReportGenericError(unsigned long, unsigned long, unsigned long, unsigned long, bool, unsigned long, unsigned int, bool) crtstuff.c
    #6 0x56414f066885 in gmtime_r (/mnt/hdd01/ci/branch21-deploy/be/lib/doris_be+0x17ef3885) (BuildId: f58eb5e327529636)
    #7 0x564177940521 in google::LogMessage::Init(char const*, int, int, void (google::LogMessage::*)()) crtstuff.c
    #8 0x564151de36fc in doris::Status doris::ThriftRpcHelper::rpc(std::__cxx11::basic_string, std::allocator> const&, int, std::function&)>, int) /home/zcp/repo_center/doris_branch-2.1/doris/be/src/util/thrift_rpc_helper.cpp:76:13
    #9 0x56417603cda7 in doris::vectorized::VRowDistribution::automatic_create_partition() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/vrow_distribution.cpp:99:5
    #10 0x56417614cffa in doris::vectorized::VTabletWriter::_send_new_partition_batch() /home/zcp/repo_center/doris_branch-2.1/doris/be/src/vec/sink/writer/vtablet_writer.cpp:1346:9
....
```
 2024-07-02 17:45:04 +08:00
Ashin Gau
d0eea3886d [fix](multi-catalog) Revert #36575 and check nullptr of data column (#37086) 
Revert #36575, because `VScanner::get_block` will check
`DCHECK(block->rows() == 0)`, so block should be cleared when `eof =
true`.
 2024-07-02 15:32:52 +08:00
Sun Chenyang
7443e8fcf2 [cherry-pick](branch-2.1) fix single compaction test p2 #34568 #36881 (#37075) 2024-07-02 15:22:04 +08:00
Ashin Gau
859a7c80b8 [fix](parquet) prevent parquet page reader print much warning logs (#37012) 
bp #37011
 2024-07-02 14:33:01 +08:00
TengJianPing
6789f5bc80 [fix](null safe equal join) fix coredump if both sides of the conjunct is not nullable #36263 (#37073) 2024-07-02 11:01:55 +08:00
Ashin Gau
e686e85f27 [opt](split) add max wait time of getting splits (#36842) 
bp: #36843
 2024-07-01 22:05:25 +08:00
zclllyybb
72c20d3ccc [branch-2.1](function) fix date_format and from_unixtime core when meet long format string (#35883) (#36158) 
pick #35883
 2024-07-01 20:35:31 +08:00
camby
798d9d6fc6 [pick21][opt](mow) reduce memory usage for mow table compaction (#36865) (#36968) 
cherry-pick https://github.com/apache/doris/pull/36865 to branch-2.1
 2024-07-01 15:33:18 +08:00
bobhan1
4210a6a8d6 [branch-2.1] PIck "[Fix](autoinc) Hanlde the processing of auto_increment column on exchange node rather than on TabletWriter when using TABLET_SINK_SHUFFLE_PARTITIONED #36836" (#37029) 
## Proposed changes

pick https://github.com/apache/doris/pull/36836
 2024-07-01 09:56:30 +08:00
deardeng
92cbbd2b75 [fix](clone) Fix clone and alter tablet use same tablet path #34889 (#36858) 
cherry pick from #34889
 2024-06-30 20:40:54 +08:00
Yongqiang YANG
07278e9dcb [improvement](segmentcache) limit segment cache by memory or segment … (#37035) 
…num (#37026)

pick ##37026
 2024-06-30 20:34:13 +08:00
bobhan1
12dddfc26c [branch-2.1] Pick "[Fix](autoinc) try fix concurrent load problem with auto inc column #36421" (#37027) 
## Proposed changes

pick https://github.com/apache/doris/pull/36421
 2024-06-30 13:10:03 +08:00
amory
d237a4d303 [fix](array)fix array_except/union for left const return only one row result #36776 (#36986) 2024-06-30 12:25:17 +08:00
amory
021ed1d690 [fix](map)fix upgrade behavior from 1.2 version #36635 (#36983) 2024-06-30 12:11:03 +08:00
lihangyu
d0c0a7b9ae [Fix](variant) ignore serialization of nothing type (#37006) 
picked from #36997
 2024-06-28 18:41:40 +08:00
Pxl
cb80ae906f [Bug](runtime-filter) disable sync filter when pipeline engine is off (#36994) 
## Proposed changes
1. disable sync filter when pipeline engine is off
2. reduce some warning log
 2024-06-28 16:59:26 +08:00
Jerry Hu
f27ae8fa09 [fix](bitmap) incorrect type of BitmapValue with fastunion (#36834) (#36896) 2024-06-28 11:29:03 +08:00
yujun
22cb7b8fcb [improvement](compaction) be do not compact invisible version to avoid query error -230 #28082 (#36222) 
cherry pick from #28082
 2024-06-27 13:45:21 +08:00
Lightman
23cf494b48 [fix](schema-change) Fix schema-change from non-null to null (#36389) 
https://github.com/apache/doris/pull/32913
 2024-06-26 20:20:50 +08:00
TengJianPing
25fb30c723 [fix](intersect) fix coredump caused by intersect of nullable and not nullable children #36401 (#36441) 
## Proposed changes

Pick #36765
 2024-06-26 17:45:21 +08:00
zhangstar333
695d58f354 [cherry-pick](scan)scanner could eos early when reached limit (#36535) (#36736) 
## Proposed changes
cherry-pick from master #36535
 2024-06-25 17:22:43 +08:00
TengJianPing
11201feae5 [fix](spill join) fix coredump of debug_string (#36723) 
## Proposed changes

Pick #36715

<!--Describe your changes.-->
 2024-06-25 16:33:33 +08:00
yujun
785a1f49f5 [fix](txn) Fix coordidator be restart not abort txn #35342 (#36437) 
cherry pick from #35342
 2024-06-25 13:35:01 +08:00
plat1ko
3652fc31c3 [Pick 2.1] "Fix data loss when node channel been cancelled before close wait (#36662)" (#36744) 
## Proposed changes

Pick from https://github.com/apache/doris/pull/36662
 2024-06-25 11:36:31 +08:00
zhannngchen
6ec9a731e8 [branch-2.1](cherry-pick) partial update should not read old fileds from rows with delete sign (#36210) (#36755) 
cherry-pick #36210
 2024-06-24 21:13:24 +08:00
Ashin Gau
e4b6dac0c1 [fix](ubsan) reinterpret_cast fix length types to int8 is not safe (#36725) 
## Proposed changes

Fix type check of ubsan. 
```
/root/doris/be/src/vec/exec/format/parquet/fix_length_plain_decoder.h:75:78: runtime error: member call on address 0x5582f35db5c0 which does not point to an object of type 'doris::vectorized::ColumnVector<signed char>'
0x5582f35db5c0: note: object is of type 'doris::vectorized::ColumnVector<int>'
 83 55 00 00  78 c0 b0 5a 82 55 00 00  02 00 00 00 00 00 00 00  10 a0 00 d7 83 55 00 00  10 a0 00 d7
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'doris::vectorized::ColumnVector<int>'
doris::Status doris::vectorized::FixLengthPlainDecoder::_decode_values<false>(COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>&, std::shared_ptr<doris::vectorized::IDataType const>&, doris::vectorized::ColumnSelectVector&, bool) at fix_length_plain_decoder.h:75:78
```
 2024-06-24 14:03:41 +08:00
Pxl
c6205783fa [Bug](function) fix wrong output_char_size on hll_to_base64 (#36572) 
## Proposed changes
pick from #36529
 2024-06-24 13:19:28 +08:00
abmdocrt
02fad48870 [Fix](upgrade) Fix fields not handled correctly during upgrade and downgrade (#36691) 
master version is #36690
 2024-06-22 14:23:04 +08:00
Qi Chen
17cf34b244 [Fix](multi-catalog) Fix core in orc and parquet reader sometimes after low mem exception. (#36575) 
## Proposed changes

Backport #36574.
 2024-06-22 11:28:21 +08:00
HappenLee
90a4dd09f3 [Fix](func) CoreDump and Result Error in percentile function (#36647) 
cherry pick #36643
 2024-06-21 23:42:45 +08:00
lihangyu
445d42a57d [fix](topn-opt) remove redundant check for fetch phase (#36676) 
#36629
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-06-21 22:28:38 +08:00
wangbo
c8e4c404fa [Fix]check if fe set thrift field current_connect_fe (#36681) 
bp #36678
 2024-06-21 22:15:25 +08:00
airborne12
c939781411 [Pick 2.1](inverted index) fix wrong no need read data when need_remaining_after_evaluate (#36684) 
When using an equal predicate on a column that applies an inverted index
with a parser, it requires remaining_after_evaluate. In this situation,
we cannot optimize the column without reading the data.

## Proposed changes

From (#36637)
 2024-06-21 22:01:39 +08:00
Mingyu Chen
0cff539810 [feature](function) support new function replace_empty (#36283) (#36656) 
#36283
 2024-06-21 16:46:22 +08:00
zhiqiang
c8f2a3f952 [fix](eq_for_null) fix incorrect logic in function eq_for_null #36004 (#36124) 
cherry pick from #36004
cherry pick from #36164
 2024-06-21 14:31:21 +08:00
airborne12
8105dc7de8 [Pick 2.1](inverted index) fix wrong opt for pk no need read data (#36634) 
## Proposed changes
 
Pick from #36618
 2024-06-21 00:57:23 +08:00