database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,634 Commits 1 Branch 0 Tags
79b07d0b8a4e179c463cf450f0b991e7e194fd60
Commit Graph

19634 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
hui lai
79b07d0b8a [fix](routine load) fix enclose and escape can not set in routine load job (#38402) (#38825) 
pick (#38402)
 2024-08-04 22:17:12 +08:00
hui lai
6035edad0b [fix](multi table) fix single stream multi table memory leak (#38255) (#38824) 
pick (#38255)

We meet OOM when using single stream multi table


![image](https://github.com/user-attachments/assets/748e9914-d591-4f41-8b28-412d3cecc841)

It exist memory leak, and heap profile like:


![image](https://github.com/user-attachments/assets/af30c593-88ea-44f6-bba1-82436b13f99f)

The stream load context will not release in some exception conditions as
plan failed for high concurrency causing timeout when obtaining read
lock. It is introduced by https://github.com/apache/doris/pull/35458

The solution effect is shown in the following figure, which can run
stably with a small amount of memory


![image](https://github.com/user-attachments/assets/4483e0a5-6c0c-4cdc-b8ed-3408da6a86b2)
 2024-08-04 22:12:44 +08:00
hui lai
8e4fad99a1 [test](routine load) add routine load case with timestamp as offset(#38567) (#38822) 
pick (#38567)
 2024-08-04 22:05:19 +08:00
hui lai
eef8c87fb5 [chore](test) disable fault injection to make pipeline task check happy (#38665) (#38821) 
pick (#38665)

test_delta_writer_v2_back_pressure_fault_injection would make pipeline
task can not finish, disable it temporarily to make pipeline task check
happy.
 2024-08-04 11:18:56 +08:00
Calvin Kirs
7c70f75198 [Fix](Load)Audit logs avoid recording certain sensitive information #38769 (#38784) 
…

## Proposed changes

#38769

<!--Describe your changes.-->
 2024-08-04 10:53:03 +08:00
Luwei
0603ec1d9d [enhancement](compaction) optimizing memory usage for compaction (#37099) (#37486) 2024-08-04 10:49:18 +08:00
HappenLee
7bdc508ac7 [Bug](fix) fix coredump case in (not null, null) execpt (not null, not null) case (#38756) 
## Proposed changes

Issue Number: close #38612

<!--Describe your changes.-->
 2024-08-04 10:44:10 +08:00
qiye
c0caca7c55 [fix](ES Catalog)Fix unstable test test_es_query (#38801) (#38802) 
## Proposed changes

bp #38801
 2024-08-03 23:49:00 +08:00
qiye
fe3e3d0fab [fix](test)Fix build index fault test (#38736) (#38762) 
## Proposed changes

backport #38736
 2024-08-03 23:48:29 +08:00
qiye
74908c123a [fix](test)Fix unstable test drop index fault #38768 (#38772) 
## Proposed changes

bp #38768

<!--Describe your changes.-->
 2024-08-03 23:47:55 +08:00
bobhan1
64b69ed1ba [branch-2.1] Picks "[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487" (#38682) 
## Proposed changes

picks https://github.com/apache/doris/pull/38487
 2024-08-02 20:05:31 +08:00
amory
556f0fc784 [pick](json-keys) support json_keys function (#38631) 
## Proposed changes
backport: https://github.com/apache/doris/pull/36411
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 19:10:00 +08:00
zhangdong
2425730609 [enhance](auth)support cache ranger datamask and row filter (#37723) (#38575) 
pick: https://github.com/apache/doris/pull/37723
 2024-08-02 14:59:32 +08:00
morrySnow
f24d55fc94 [fix](syntax) multi statements must delim with semicolon (#38670) (#38753) 
pick from master #38670
 2024-08-02 14:49:51 +08:00
Mingyu Chen
da7b2cf578 [refactor](catalog) set "use_meta_cache" default to true (#38244)(#38352)(#38619) (#38355) 
bp #38244 #38352 #38619

---------

Co-authored-by: Yulei-Yang <yulei.yang0699@gmail.com>
 2024-08-02 14:13:38 +08:00
amory
9b07cd2069 [pick](json-serde)pick jsonb string deserialize with spec char (#38711) 
## Proposed changes
backport: https://github.com/apache/doris/pull/37176
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 13:37:41 +08:00
Calvin Kirs
d800434859 [Chore](Fe)Upgrade dependencies (#38509) (#38747) 
## Proposed changes

upgrade spring-boot to 2.7.18
upgrade zookeeper to 3.9.2
upgrade jetty to 9.4.55.v20240627
upgrade ivy to 2.5.2
upgrade icu4j to 75.1
upgrade ini4j to 0.5.4

(cherry picked from commit 3f633c2018e86c6c842647262853d88ad63672bf)

pick #38509
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 12:34:17 +08:00
amory
7bcda89881 [pick](case) fix one_nested_types cases (#38723) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38410
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 12:07:14 +08:00
qiye
f21d7e3833 [test](inverted index)Add cases for inverted index format v2 (#38132)(#38443) (#38222) 
## Proposed changes

backport #38132 #38443
 2024-08-02 12:04:26 +08:00
qiye
b3f335ba5f [enhancement](index compaction) Enable index compaction by default (#36812) (#38676) 
## Proposed changes

bp #36812
 2024-08-02 12:03:57 +08:00
Mingyu Chen
4f2ca43917 [minor](fe) simplfy some code in HMSExternalTable (#32344) (#38675) 
bp #32344

Co-authored-by: DuRipeng <453243496@qq.com>
 2024-08-02 11:27:10 +08:00
amory
1d982ada45 [pick](array-funcs)pick array func array_enumerate_uniq bugfix (#38721) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38384
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 11:25:17 +08:00
amory
f5bc65989c [pick](array-range)improve array_range func for large param (#38707) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38284
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 11:22:46 +08:00
amory
b7e1588be9 [pick](upgrade)fix log message (#38710) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38254
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 11:20:20 +08:00
amory
84d9b2fcf4 [pick](nestedtypes) support nested type with agg replace_if_not_null (#38719) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38304
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-02 11:18:33 +08:00
yujun
327069fdbc [branch-2.1](log) add tablet clear cache log (#38713) 2024-08-02 08:40:02 +08:00
morrySnow
e140613ae1 [fix](Nereids) remove db readlock before get table from db (#38660) (#38729) 
pick from master #38660

insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
 2024-08-02 08:34:59 +08:00
zzzxl
0da388ade5 [fix](inverted index) fix match_phrase_ edge query result error #38327 (#38740) 2024-08-01 23:17:53 +08:00
qiye
4d980b8235 [feature](http action)Add http action to show nested inverted index file (#38272) (#38672) 
backport #38272
 2024-08-01 19:30:59 +08:00
qiye
555dccb1a4 [fix](bloom filter)Fix rename column with bloom filter (#38431) (#38662) 
backport #38431
 2024-08-01 19:01:14 +08:00
Lijia Liu
2562cf33a7 [fix](mtmv) Choose a valid partition column when there are both valid and invalid expressions (#38367) (#38684) 
## Proposed changes
pick #38367 

<!--Describe your changes.-->
 2024-08-01 19:00:28 +08:00
qiye
e59a866d6a [fix](test)Fix unstable index compaction fault injection case (#38501) (#38673) 
## Proposed changes

bp #38501
 2024-08-01 18:58:51 +08:00
zhangdong
60091f072a [fix](auth)fix create table like need create_priv of existed table (#… (#38570) 
…37879)

pick: https://github.com/apache/doris/pull/37879
 2024-08-01 18:57:44 +08:00
wuwenchi
a4e793752f [bugfix](iceberg)revert count(*) directly returned by fe for 2.1 (#38566) (#38655) 
bp: #38566
 2024-08-01 18:56:19 +08:00
Gabriel
3e5255a862 [pipeline](fix) Fix blocking task which is not triggered by 2nd RPC (… (#38694) 
…#38568)

Once a query is cancelled due to any reason, BE may not receive 2nd RPC
from FE. If so, we must ensure the execution dependency is ready so
tasks will not be blocked.
 2024-08-01 18:23:41 +08:00
Gabriel
82c681595e [fix](local exchange) Fix local exchange blocked by a huge data block… (#38693) 
… (#38657)

If a huge block is push into local exchanger, it will be blocked due to
concurrent problems. This PR use a unique lock to resolve it .
 2024-08-01 18:04:19 +08:00
meiyi
e8690b62ee [fix](group commit) Pick add debug log show why group commit not work; delete wal when replay success (#38611) (#38659) 
Pick https://github.com/apache/doris/pull/38611
 2024-08-01 16:59:54 +08:00
Siyang Tang
cafcf7acc1 [cherry-pick](SSL) Fix ssl connection close 2.1 (#38587) (#38677) 
## Proposed changes

Issue Number: close #38590 

If SSL connection closed, a specified packet will sent to indicate the
closing of connection. The SSL engine will be shut down and output an
empty unwrapped result.

Therefore, handle this case correctly to avoid buffer overflow by
breaking the reading flow and do the cleanup stuff initiatively.
 2024-08-01 16:06:30 +08:00
Gabriel
9d23ccf1f2 [Improvement](schema scan) Use async scanner for schema scanners (#38… (#38666) 
…403)
 2024-08-01 16:05:24 +08:00
slothever
b0943064e0 [fix](kerberos)fix and refactor ugi login for kerberos and simple authentication (#38607) 
pick from  (#37301)
 2024-08-01 14:01:32 +08:00
Qi Chen
4042cdf553 [Fix](memory) Fix allocator.h compiling failed on mac. (#38646) 
Backport #38562. Fix allocator.h compiling failed on mac which
introduced by #37257.
 2024-08-01 13:56:53 +08:00
Xin Liao
63a3ff570b [Opt](load) print tablet id when memtable flush coredump #38618 (#38656) 
cherry pick from #38618
 2024-08-01 13:52:50 +08:00
HappenLee
28998300d4 [Bug](fix) fix ubsan use int32_t pointer access bool value (#38621) 
## Proposed changes

Issue Number: close #38617

<!--Describe your changes.-->
 2024-08-01 13:52:12 +08:00
wuwenchi
c310b92727 [bugfix](external)Modify the default value of pushdowncount for 2.1 (#37754) (#38591) 
bp: #37754
 2024-08-01 12:10:54 +08:00
wuwenchi
057ee1905f [bugfix](hudi)add timetravel for nereids for 2.1 (#38324) (#38582) 
## Proposed changes

bp #38324
 2024-08-01 11:37:57 +08:00
amory
338fa32303 [pick](simdjson) fix simdjson with object array when jsonroot is not empty (#38633) 
## Proposed changes
backport: https://github.com/apache/doris/pull/38490
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-01 11:04:54 +08:00
wuwenchi
41fa7bc9fd [bugfix](paimon)Fixed the reading of timestamp with time zone type data for 2.1 (#37716) (#38592) 
bp: #37716
 2024-08-01 10:23:06 +08:00
abmdocrt
9f1e41c623 [Cherry-pick](branch-2.1) Pick "[Enhancement](audit log) Add print audit log sesssion variable #38419" (#38624) 
## Proposed changes

<!--Describe your changes.-->

Pick #38419
 2024-08-01 10:19:19 +08:00
amory
184b8cbbe4 [pick](json)fix jsonb deseriaze (#38630) 
## Proposed changes
backport: https://github.com/apache/doris/pull/37251
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-01 10:18:27 +08:00
amory
6bd93b119f [pick](cast)Feature cast complexttype2 json (#38632) 
## Proposed changes
backport: https://github.com/apache/doris/pull/36548
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-01 09:18:15 +08:00