Commit Graph

22 Commits

Author SHA1 Message Date
Pxl
ca71048f7f [Chore](status) avoid empty error msg on status (#21454)
avoid empty error msg on status
2023-07-11 13:48:16 +08:00
ee9822fa7e [Fix](pipeline) fix ExchangeSinkBuffer request id memory alloc problem (#21647)
Co-authored-by: airborne12 <airborne12@gmail.com>
fix ExchangeSinkBuffer request id memory alloc problem
2023-07-09 23:45:28 +08:00
Pxl
f8cfe5e579 [Bug](pipeline) add DCHECK for _instance_to_sending_by_pipeline = false on _send_rpc (#21169)
add DCHECK for _instance_to_sending_by_pipeline = false on _send_rpc
2023-06-29 10:03:57 +08:00
601120db04 [Bug](pipeline) access map may cause coredump in sink buffer (#21108) 2023-06-24 23:03:59 +08:00
661e1ae7c5 [fix](memory) no switch bthread context in UBSAN compile (#21064)
When UBSAN is compiled, all memory will be tracked to the orphan (unknown) mem tracker, and the bthread context and mem tracker will no longer be switched.

The supplementary fixes are as follows: #20999
2023-06-21 21:14:07 +08:00
622ef63c69 [fix](memory) fix bthread_setspecific error in rpc done.run() (#20999) 2023-06-20 21:00:45 +08:00
14f59bef1d [improvement](profile)add sum/avg rpc time (#20511) 2023-06-12 11:34:49 +08:00
05438eab0d remove DCHECK for rpc time (#20621) 2023-06-09 13:38:12 +08:00
65100d8083 [improvement](profile)add max/min rpc time (#20339) 2023-06-06 12:03:01 +08:00
8bec2b41db [pipeline](rpc) support closure reuse in pipeline exec engine (#20278) 2023-06-02 09:50:21 +08:00
488c9ba7c2 [improvement](exchange) test: data stream sender stop sending data to receiver if it returns eos early (#20081) 2023-05-26 16:05:38 +08:00
3598518e59 [fix](revert) data stream sender stop sending data to receiver if it returns eos early (#19847)" (#20040)
* Revert "[fix](sink) fix END_OF_FILE error for pipeline caused by VDataStreamSender eof (#20007)"

This reverts commit 2ec1d282c5e27b25d37baf91cacde082cca4ec31.

* [fix](revert) data stream sender stop sending data to receiver if it returns eos early (#19847)"

This reverts commit c73003359567067ea7d44e4a06c1670c9ec37902.
2023-05-25 16:50:17 +08:00
c730033595 [improvement](exchange) data stream sender stop sending data to receiver if it returns eos early (#19847)
For broadcast join, only one build fragment instance will build hash table, other fragment instances just receive and throw away build side data, this is waste of memory and cpu.

This PR improve this condition, data stream receiver tells sender that it does not need data from sender, and sender stops sending anydata to it.
2023-05-24 15:11:32 +08:00
693a3651c1 [bugfix](rpc) fix read-after-free problem of DeleteClosure (#19250)
1. fix read-after-free problem of DeleteClosure.
    2. modified fresh_exec_timer for operators
2023-05-05 09:57:54 +08:00
e412dd12e8 [chore](build) Use include-what-you-use to optimize includes (PART II) (#18761)
Currently, there are some useless includes in the codebase. We can use a tool named include-what-you-use to optimize these includes. By using a strict include-what-you-use policy, we can get lots of benefits from it.
2023-04-19 23:11:48 +08:00
633f2d52a4 [minor](log) add some logs (#17287) 2023-03-01 22:41:50 +08:00
784c27deeb [Bug](shuffle) fix mem leak in data stream sender (#16685) 2023-02-14 16:40:13 +08:00
a038fdaec6 [Bug](pipeline) Fix bug in non-local exchange on pipeline engine (#16463)
Currently, for broadcast shuffle, we serialize a block once and then send it by RPC through multiple channel. After this, we will serialize next block in the same memory for consideration of memory reuse. However, since the RPC is asynchronized, maybe the next block serialization will happen before sending the previous block.

So, in this PR, I use a ref count to identify if the serialized block can be reuse in broadcast shuffle.
2023-02-09 19:22:40 +08:00
84d6938a73 [Bug](pipeline) Fix BE crash caused by pipeline (#15890)
* [Bug](pipeline) Fix BE crash caused by pipeline

* update
2023-01-14 18:37:19 +08:00
f3aea7f0f0 [Enhancement](status) Unify error code and enable customed err msg for BE internal errors (#14744) 2022-12-11 23:33:18 +08:00
Pxl
48a9166aa4 [Pipeline](sink) support olap table sink operator (#14872)
* support olap table sink operator

* update config
2022-12-07 15:29:56 +08:00
12304bc0ee [Pipeline](exec) Support pipeline exec engine (#14736)
Co-authored-by: Lijia Liu <liutang123@yeah.net>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Pxl <952130278@qq.com>
Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>

## Problem Summary:

### 1. Design

DSIP: https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine

### 2. How to use:

Set the environment variable `set enable_pipeline_engine = true; `
2022-12-02 17:11:34 +08:00