Commit Graph

222 Commits

Author SHA1 Message Date
86fa6a37c7 [fix](load) Fix the issue of high-concurrency single-replica load getting stuck #42297 (#42317)
cherry pick from #42297
2024-10-23 23:28:05 +08:00
dc438649d9 [bugfix](handshake) brpc handshake should not use light pool (#42115) (#42127)
The light pool may be full. Handshake is used to check the connection
state of brpc. Should not be interfered by the thread pool logic.

---------
pick #42115

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-10-19 16:19:17 +08:00
cb24ccc112 [bugfix](brpc) Should use status to generate protobuf message, because it will encoding Backend Info (#41515) (#41522)
Should use status to generate protobuf message, because it will encoding
Backend Info

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-04 23:03:55 +08:00
b52b572ade [branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850)
pick
#38960
#39908
#40043
#40092
#40016
#40439

---------

Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
2024-09-15 23:47:53 +08:00
f16615a1fc [branch-2.1](memory) Allocator support address sanitizers (#40836)
pick
#33396
#33862
#33853
#33732
#33841
#33933
#34901
#35014

---------

Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-09-14 12:12:44 +08:00
7851563829 [fix](brpc_client_cache) resolve hostname in DNS cache before passing to brpc (#40074) (#40786)
backport #40074
2024-09-13 14:28:01 +08:00
7a7292ad5a [branch-2.1][Refactor]use async to get be resource (#38389) (#39826)
pick #38389
2024-08-23 17:16:19 +08:00
830f250a80 [opt](query cancel) cancel query if it has pipeline task leakage #39223 (#39537)
pick #39223 with some modifications. Optimization will only be applied
to pipeline x.
2024-08-19 14:33:59 +08:00
eea3676791 [fix](group commit) fix group commit insert rpc may stuck (#39391) (#39458)
pick https://github.com/apache/doris/pull/39391
2024-08-16 13:19:00 +08:00
fff26fe2fc [fix](group commit) fix group commit core if be inject FragmentMgr.exec_plan_fragment.failed (#39339) (#39396)
pick https://github.com/apache/doris/pull/39339
2024-08-15 17:54:11 +08:00
187461e2fd [Fix](Export) Export delete multiple times when specify the delete_existing_files property () (#39304)
bp: #38400

When the `Export` statement specifies the `delete_existing_files`
property, each `Outfile` statement generated by the `Export` will carry
this property. This causes each `Outfile` statement to delete existing
files, so only the result of the last Outfile statement will be
retained.

So, we add a rpc method which can delete existing files for `Export`
statement and the `Outfile` statements generated by the `Export` will
not carry `delete_existing_files` property any more.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-13 22:26:02 +08:00
91f0301b43 [fix](group commit) Pick some group commit pr (#38320)
Pick https://github.com/apache/doris/pull/38292,
https://github.com/apache/doris/pull/34021,
https://github.com/apache/doris/pull/38228, some modify of
https://github.com/apache/doris/pull/37260, some modify of
https://github.com/apache/doris/pull/37595
2024-07-25 17:32:44 +08:00
a8139d9f39 [Improvement](profile) Provide more info for schedule time (#38290) (#38338)
## Proposed changes

pick #38290

<!--Describe your changes.-->
2024-07-25 14:43:52 +08:00
294157dafc [fix](core) fix closure use after release #38087 (#38094)
## Proposed changes
https://github.com/apache/doris/pull/38087
<!--Describe your changes.-->
2024-07-18 22:44:30 +08:00
1f779ba9de [branch-2.1](arrow-flight-sql) Open regression-test/pipeline/p0/arrow_flight_sql (#37727)
pick #36854
2024-07-16 16:23:43 +08:00
dd18652861 [branch-2.1](routine-load) make get Kafka meta timeout configurable (#37399)
pick #36619
2024-07-08 10:39:17 +08:00
89857d3780 [cherry-pick](branch-2.1) Pick "Use async group commit rpc call (#36499)" (#37380)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Pick #36499
2024-07-07 18:28:19 +08:00
612f2ae961 [feature](api) add BE HTTP /api/load_streams (#36312) (#36338)
cherry-pick #36312
2024-06-16 22:09:04 +08:00
0b28420e1c [pick](Variant) make remote schema fetch rpc timeout configurable (#35296) (#36174) 2024-06-12 19:51:53 +08:00
c033c71aed [enhancement](cancel) add cancel query ability on be 2024-06-03 11:43:57 +08:00
7058b31edd [fix](move-memtable) clear load streams before shutdown SegmentFileWriterThreadPool (#35217) 2024-05-28 13:12:03 +08:00
7ba66c5890 [branch-2.1](routine-load) do not schedule task when there is no data (#34654) 2024-05-11 11:01:18 +08:00
315f6e44c2 [Branch-2.1](Outfile) Fixed the problem that the concurrent Outfile wrote multiple Success files (#33870)
backport: #33016
2024-04-19 12:09:53 +08:00
e0ec2da29b [fix](routine-load) fix get kafka offset timeout may too long (#33502) 2024-04-17 23:42:12 +08:00
af95302088 fix compile 2024-04-11 13:10:24 +08:00
69fc8cf06d [branch-2.1](memory) Fix rowid storage reader memory tracker (#33521)
fix:
F20240411 10:26:06.693233 1368925 thread_context.h:293] __builtin_unreachable, If you crash here, it means that SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not used correctly. starting position of each thread is expected to use SCOPED_ATTACH_TASK to bind a MemTrackerLimiter belonging to Query/Load/Compaction/Other Tasks, otherwise memory alloc using Doris Allocator in the thread will crash. If you want to switch MemTrackerLimiter during thread execution, please use SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER, do not repeat Attach. Of course, you can modify enable_memory_orphan_check=false in be.conf to avoid this crash.
*** Check failure stack trace: ***
    @     0x562d9b5aa6a6  google::LogMessage::SendToLog()
    @     0x562d9b5a70f0  google::LogMessage::Flush()
    @     0x562d9b5aaee9  google::LogMessageFatal::~LogMessageFatal()
    @     0x562d7ebd1b7e  doris::thread_context()
    @     0x562d7ec203b8  Allocator<>::sys_memory_check()
    @     0x562d7ec255a3  Allocator<>::memory_check()
    @     0x562d7ec274a1  Allocator<>::alloc_impl()
    @     0x562d7ec27227  Allocator<>::alloc()
    @     0x562d67a12207  doris::vectorized::PODArrayBase<>::alloc<>()
    @     0x562d67a11fde  doris::vectorized::PODArrayBase<>::realloc<>()
    @     0x562d67a11e26  doris::vectorized::PODArrayBase<>::reserve<>()
    @     0x562d77331ee3  doris::vectorized::ColumnVector<>::reserve()
    @     0x562d7e64328e  doris::vectorized::ColumnNullable::reserve()
    @     0x562d7ec79a84  doris::vectorized::Block::Block()
    @     0x562d6b86b81b  doris::PInternalServiceImpl::_multi_get()
    @     0x562d6b8a4a07  doris::PInternalServiceImpl::multiget_data()::$_0::operator()()
2024-04-11 13:10:24 +08:00
Pxl
3081fc584d [Improvement](runtime-filter) support sync join node build side's size to init bloom runtime filter (#32180)
support sync join node build side's size to init bloom runtime filter
2024-04-11 09:31:50 +08:00
0e262ba0e4 [improvement](spill) improve cancel of spill and improve log printing (#33229)
* [improvement](spill) improve cancel of spill and improve log printing

* fix
2024-04-10 16:23:20 +08:00
cf7595d423 [opt](memory) Optimize mem tracker accuracy (#32039) (#33140) 2024-04-10 11:42:19 +08:00
95da52b9d8 [fix](avro) avoid BE crash if avro scanner's dependency jars is mssing (#33031)
1. Check the return value of avro reader's init_fetch_table_schema_reader()
2. Also fix a bug but the parse exception of Nereids may suppress the real exception from old planner
    It will result unable to see the real error msg.
2024-04-07 22:17:16 +08:00
4bf5a21ba3 [pipelineX](cancel) Remove lock for mapping query ctx to fragment (#32346) 2024-03-21 14:07:23 +08:00
85b2c42f76 [Enhancement](jdbc catalog) Add a property to test the connection when creating a Jdbc catalog (#32125) (#32531) 2024-03-21 14:05:59 +08:00
ecadb60bcd [Pick 2.1](inverted index) support inverted index format v2 (#30145) (#32418) 2024-03-19 08:11:33 +08:00
0da010603e [Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path (#31141)
Both could be reference to related field in TabletColumn.And use shared_ptr for TabletColumn in TabletSchema for later memory reuse
2024-03-09 19:44:42 +08:00
fc53c7210b [fix](chmod) change chmod to filesystem::permission to avoid race condition (#31032) 2024-02-18 11:50:16 +08:00
09ef78402e fix build error 2024-02-05 22:15:17 +08:00
be31b8dc61 [Refactor](exchange) remove unless code in exchange and opt some code (#30813) 2024-02-05 21:59:52 +08:00
5a0764b288 [log](rpc) print log when offer_failed in internal service (#30454) 2024-02-01 19:01:08 +08:00
620cfc3cd7 [fix](move-memtable) set idle timeout equal to load timeout (#29839) 2024-01-16 18:33:51 +08:00
0b731800a0 [enhancement](group_commit) refector wal manager code (#29560) 2024-01-07 18:54:41 +08:00
e8e1a6ff03 [refactor](fold) refactor fold executor could get error status (#29541)
before _get_result return std::string, so ignore the error status.
2024-01-05 14:23:56 +08:00
329d57fdd7 [regression](move-memtable) test LoadStream on_idle_timeout (#29354)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2024-01-03 14:07:51 +08:00
706463781c [refactor](group commit) refactor group commit wal code (#29375) 2024-01-02 15:52:03 +08:00
b8cdbf4ffe Fix](schema change) fix metadata inconsistency bug caused by enable light schema change (#28255) (#29060) 2023-12-30 00:02:01 +08:00
e9e1e2894b [performance](variant) support topn 2phase read for variant column (#28318)
[performance](variant) support topn 2phase read for variant column
2023-12-25 11:50:41 +08:00
b1c5747f56 [improve](load) remove extra layer of heavy work pool in tablet_writer_add_block (#28550) 2023-12-22 20:10:50 +08:00
877935442f [feature](pipelineX)use markFragments instead of markInstances in pipelineX (#27829) 2023-12-11 17:59:53 +08:00
573b594df3 [improvement](Variant Type) Support displaying subcolumns expanded for the variant column (#27764) 2023-12-08 20:34:58 +08:00
d96e2dfefb [feature-wip](arrow-flight)(step5) Support JDBC and PreparedStatement and Fix Bug (#27661) 2023-11-29 21:17:20 +08:00
4633a5c49b [chore](log) Warning log to trace send fragment #27738 2023-11-29 16:43:25 +08:00