85dd606fd1
[fix](group_commit) Fix group_commit ut ( #29587 )
2024-01-06 18:11:13 +08:00
a0c3ddf902
[fix](memory) Fix LRUCacheType::NUMBER charge ( #29588 )
...
if LRUCacheType::NUMBER, charge not add handle_size, because charge at this time is no longer the memory size, but an independent weight.
2024-01-06 10:37:56 +08:00
eea6219920
[improvement](group commit) Add an option of enable relay wal on all regression test ( #28757 )
2024-01-05 18:27:31 +08:00
e2e9b9d8a3
[improve](insert-into) record rows info in log for check ( #29581 )
2024-01-05 17:28:07 +08:00
e8e1a6ff03
[refactor](fold) refactor fold executor could get error status ( #29541 )
...
before _get_result return std::string, so ignore the error status.
2024-01-05 14:23:56 +08:00
bfd23e30f6
[improve](load) handle EAGAIN in load stream ( #29437 )
2024-01-04 23:02:11 +08:00
96acef908a
[fix](move-memtable) check eos when close stream ( #29547 )
2024-01-04 22:56:52 +08:00
bd8113f424
[bugfix](scannerscheduler) should minus num_of_scanners before check should schedule #28926 ( #29331 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-01-03 20:47:35 +08:00
c84cd30223
[pipelineX](fix) Fix query cancel timeout ( #29460 )
...
There are 2 potential reasons to cancel pipelineX query timeout.
Cancel fragment context first and set ready to execute will set cancel flag to false.
Dead lock.
2024-01-03 20:29:04 +08:00
e3c9f535dc
[refactor](wal) refactor some wal code ( #29434 )
2024-01-03 14:45:57 +08:00
329d57fdd7
[regression](move-memtable) test LoadStream on_idle_timeout ( #29354 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2024-01-03 14:07:51 +08:00
1e8bb75182
[improve](move-memtable) add log on idle timeout ( #29438 )
2024-01-03 11:26:26 +08:00
be1d9c3358
[fix](memory) Fix mem tracker web page notice #29361
2024-01-03 11:25:00 +08:00
14e7eb7624
[Opt](rf) Opt broadcast join remote runtime filter merge and wait ( #29439 )
2024-01-03 11:21:28 +08:00
69524ccf98
[improve](group commit) Group commit support commit by data size ( #29428 )
2024-01-02 23:20:23 +08:00
243b075eef
[improve](move-memtable) report failed tablets with status ( #29351 )
2024-01-02 20:24:53 +08:00
b3f69211b7
[refactor](wal) move group commit load content length to runtime state ( #29188 )
2024-01-02 15:53:38 +08:00
706463781c
[refactor](group commit) refactor group commit wal code ( #29375 )
2024-01-02 15:52:03 +08:00
2ed122b787
[improvement](task exec context) add parent class HasTaskExecutionCtx to own the task ctx ( #29388 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-01-02 15:28:27 +08:00
cec69f4cbd
[Enhancement](wal) Add timout for wal memory back pressure ( #29178 )
2024-01-02 11:02:17 +08:00
4cbbd25d8c
[fix](ctx) manager the lifecycle of connection context ( #29346 )
...
In FrontendService, we may create some connection context and set it as a thread local varaible.
These context should be removed from thread local after call.
Otherwise, it may be reused by other thread incorrectly.
2024-01-01 23:32:28 +08:00
738abac9ed
[minor](context) duplicate query context in fragment ctx ( #29364 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-01-01 22:08:23 +08:00
cc40cdf2d4
[improve](load) don't limit add_segment running count in load stream ( #29332 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2023-12-31 16:47:46 +08:00
7623b5cc31
[cleanup](move-memtable) remove namespace stream_load ( #27441 )
2023-12-30 20:08:23 +08:00
03901b9a7a
[enhancement](group_commit): refector relay wal code ( #29183 )
2023-12-30 12:59:46 +08:00
e62857e5a4
[improve](move-memtable) return error when back pressure cost too much time ( #29327 )
2023-12-30 01:31:05 +08:00
82635d4b59
[opt](memory) All LRU Cache inherit from LRUCachePolicy ( #28940 )
...
After all LRU Cache inherits from LRUCachePolicy, this will allow prune stale entry, eviction when memory exceeds limit, and define common properties. LRUCache constructor change to private, only allow LRUCachePolicy to construct it.
Impl DummyLRUCache, when LRU Cache capacity is 0, will no longer be meaningless insert and evict.
2023-12-29 16:15:56 +08:00
c3c34e10bb
[feature](executor) Add some check when create workload group/workload schedule policy ( #29236 )
2023-12-29 15:41:16 +08:00
a525d5c5a3
[refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion ( #29265 )
...
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
2023-12-29 10:11:44 +08:00
9be0f04506
(improv)[group commit] refactor some group commit code ( #29180 )
2023-12-29 00:26:10 +08:00
8a491e7b1d
Fix workload scheduler start too early may cause npe ( #29258 )
2023-12-28 22:41:42 +08:00
5129ab5738
[fix](decimalv2) fix decimalv2 agg errors ( #29246 )
2023-12-28 21:17:16 +08:00
b31494b18c
[test](regression) add fault injection cases for LoadStream ( #29101 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-12-28 16:16:26 +08:00
31b3be456c
add workload scheduler in be ( #29116 )
2023-12-28 15:04:22 +08:00
0562999f91
[fix](doc) spell errors fixes and align with code log for memory tracker. ( #28000 )
...
Spell corrected for LastestSuccessChannelCache and aligned that with the docs
2023-12-28 11:12:35 +08:00
9ff8bd2e9c
[Enhancement](Wal)Support dynamic wal space limit ( #27726 )
2023-12-27 11:51:32 +08:00
6440fbfab6
[feature](scan) Implement parallel scanning by dividing the tablets based on the row range ( #28967 )
...
* [feature](scan) parallel scann on dup/mow mode
* fix bugs
2023-12-26 17:18:41 +08:00
4a60d01dc7
[improve](move-memtable) increase load_stream_flush_token_max_tasks ( #29011 )
2023-12-26 17:08:49 +08:00
e9e1e2894b
[performance](variant) support topn 2phase read for variant column ( #28318 )
...
[performance](variant) support topn 2phase read for variant column
2023-12-25 11:50:41 +08:00
f374beaa4e
[fix](log) regularise some BE error type and fix a load task check #28729
2023-12-25 10:45:19 +08:00
3b830f89a7
[improve](move-memtable) avoid using heavy work pool during append data ( #28745 )
2023-12-22 22:51:30 +08:00
18c9ebce95
[improve](move-memtable) tweak load stream flush token num and max tasks ( #28884 )
2023-12-22 20:08:47 +08:00
fa0ad56817
[exec](compress) use FragmentTransmissionCompressionCodec control the exchange compress behavior ( #28818 )
2023-12-22 19:50:57 +08:00
aca8406e31
[refactor](executor)remove scan group #28847
2023-12-22 17:05:50 +08:00
8c59e16f81
[opt](query cancel) optimization for query cancel #28778
2023-12-22 12:48:37 +08:00
453e3c18f4
[refactor](buffer) remove download buffer since it is no longer useful ( #28832 )
...
remove download buffer since it is no longer useful
2023-12-22 11:53:31 +08:00
0af6bd6390
[fix](group-commit) check if wal need recovery is abnormal ( #28769 )
2023-12-22 11:06:11 +08:00
172f68480b
[Enhancement](load) Limit the number of incorrect data drops and add documents ( #27727 )
...
In the load process, if there are problems with the original data, we will store the error data in an error_log file on the disk for subsequent debugging. However, if there are many error data, it will occupy a lot of disk space. Now we want to limit the number of error data that is saved to the disk.
Be familiar with the usage of doris' import function and internal implementation process
Add a new be configuration item load_error_log_limit_bytes = default value 200MB
Use the newly added threshold to limit the amount of data that RuntimeState::append_error_msg_to_file writes to disk
Write regression cases for testing and verification
Co-authored-by: xy720 <22125576+xy720@users.noreply.github.com >
2023-12-22 10:43:18 +08:00
0b9b1be1f1
[fix](function) Fix from_second functions overflow and wrong result ( #28685 )
2023-12-22 10:22:49 +08:00
0070909d30
[fix](group commit)Fix the issue of duplicate addition of wal path when encouter exception ( #28691 )
2023-12-21 20:27:33 +08:00
