2ed122b787
[improvement](task exec context) add parent class HasTaskExecutionCtx to own the task ctx ( #29388 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-01-02 15:28:27 +08:00
4581618b09
[improvement](disk) pick disk randomly when usage is less than 0.7 ( #29368 )
2024-01-02 14:08:09 +08:00
b7487430da
Revert "[improve](move-memtable) cancel load rapidly when stream close wait ( #29322 )" ( #29371 )
...
This reverts commit bbf58c5aa42d40e66bc6ccc9ed91a4fcb4bdfff7.
2024-01-02 11:32:14 +08:00
7cb3239599
[fix](sink) add writting restriction for OlapTableSinkV2Operator ( #29340 )
2024-01-02 11:16:06 +08:00
cec69f4cbd
[Enhancement](wal) Add timout for wal memory back pressure ( #29178 )
2024-01-02 11:02:17 +08:00
4cbbd25d8c
[fix](ctx) manager the lifecycle of connection context ( #29346 )
...
In FrontendService, we may create some connection context and set it as a thread local varaible.
These context should be removed from thread local after call.
Otherwise, it may be reused by other thread incorrectly.
2024-01-01 23:32:28 +08:00
738abac9ed
[minor](context) duplicate query context in fragment ctx ( #29364 )
...
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-01-01 22:08:23 +08:00
b07ce175de
Revert "[fix](move-memtable) only check missing tablets when commit i… ( #29352 )
...
* Revert "[fix](move-memtable) only check missing tablets when commit info is not empty (#29326 )"
This reverts commit 4634c723d7fb86f434060c0e6fa6168d92630cb9.
* Revert "[fix](move-memtable) check missing tablets before commit (#29223 )"
This reverts commit fb0ed8c253f5241c23c71fb88f031604b0bf5a1f.
2023-12-31 22:25:25 +08:00
cc40cdf2d4
[improve](load) don't limit add_segment running count in load stream ( #29332 )
...
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com >
2023-12-31 16:47:46 +08:00
9a90c1cd57
[improve](move-memtable)add multi replica fault injection ( #29348 )
2023-12-31 16:30:27 +08:00
bbf58c5aa4
[improve](move-memtable) cancel load rapidly when stream close wait ( #29322 )
2023-12-31 16:26:41 +08:00
4634c723d7
[fix](move-memtable) only check missing tablets when commit info is not empty ( #29326 )
...
Some sink may not get any success tablets, because load stream will only report them at the last CLOSE_LOAD.
For these sinks (no commit info), we should not check missing tablets since it's expected to be empty.
2023-12-31 13:50:13 +08:00
3661c316c9
Revert "[improvement](create tablet) backend create tablet round robin among disks ( #23218 )" ( #29347 )
...
This reverts commit df5b5ae0cb2f30f026ec104a64b4d9a5ce2904f3.
2023-12-31 12:51:21 +08:00
cc6013bfab
[fix](partial update) Fix error message when doing strict mode partial update on a table with column that is non-nullable and has no default value #29218
2023-12-31 11:18:37 +08:00
fcc4cfb900
[Fix](Variant) add more info before crash in serialization ( #29344 )
2023-12-31 11:17:36 +08:00
9a7c15b635
[fix](log) PipelineFragmentContext has already been destoried in try_close_task before logging it #29237
2023-12-30 21:29:58 +08:00
018d8b1326
[fix](doc) spell errors fixes and align with code log for check-rpc-channel. ( #28151 )
2023-12-30 20:56:56 +08:00
56648e0bd6
[Bug](security) BE SingleReplicaCompaction download_files function exists log print sensitive msg #28929 ( #28930 )
...
Co-authored-by: Rohit Satardekar <rohitrs1983@gmail.com >
2023-12-30 20:50:08 +08:00
673a0137ce
[fix] (schema change) fix missing data bug in schema change ( #28938 ) ( #29064 )
2023-12-30 20:45:58 +08:00
7623b5cc31
[cleanup](move-memtable) remove namespace stream_load ( #27441 )
2023-12-30 20:08:23 +08:00
03901b9a7a
[enhancement](group_commit): refector relay wal code ( #29183 )
2023-12-30 12:59:46 +08:00
30b7357604
[improve](logging) make glog FLAGS_v configurable ( #29182 )
2023-12-30 12:47:47 +08:00
e7d67e9411
[fix](be) resolves some unused-raii and used-after-moved issues ( #29285 )
2023-12-30 12:14:49 +08:00
51cb15d032
[improve](move-memtable) cancel load immediately when back pressure in delta writer v2 ( #29280 )
2023-12-30 10:45:06 +08:00
2c4e52e44e
[fix](es catalog) only es_query function can push down to ES ( #29320 )
...
Issue Number: close #29318
1. Only push down `es_query` function to ES
2. Add null check where ES query result not have `_source` or `fields` fields.
2023-12-30 09:33:26 +08:00
56058623f6
[fix](expr) Re fix BE core dump while common expr filter delete condition column ( #29328 )
...
Additional deleted filter condition will be materialized column be at the end of the block,
after _output_column_by_sel_idx will be erase, we not need to filter it,
so erase it from _columns_to_filter in the first next_batch.
Eg:
delete from table where a = 10;
select b from table;
a column only effective in segment iterator, the block from query engine only contain the b column,
so no need to filter a column by expr.
2023-12-30 08:40:03 +08:00
5c9e0d906a
[Fix](segment iterator) fix wrong segment cache remove ( #29333 )
2023-12-30 08:36:15 +08:00
5bc72bda78
[improve](move-memtable) add fault injection in writer v2 ( #29177 )
2023-12-30 01:32:26 +08:00
e62857e5a4
[improve](move-memtable) return error when back pressure cost too much time ( #29327 )
2023-12-30 01:31:05 +08:00
41d907912a
[improve](move-memtable) add fault injection in load stream stub ( #29105 )
2023-12-30 01:29:28 +08:00
d7001ce6e1
[fix](partition) Comment out some exceptions with partition equal to … ( #28702 )
...
* [fix](partition) Comment out some exceptions with partition equal to 0 and return
In some scenarios, the partition id in the table meta is 0, but the rowset meta partition is not 0, (#28055 ) PR check partition id will caused be failed to start when upgrade
meta like this
```
{
"table_id": 1266302,
"partition_id": 0,
"tablet_id": 1289222,
"schema_hash": 816078096,
"shard_id": 949,
"creation_time": 1684805433,
"cumulative_layer_point": -1,
"tablet_state": "PB_RUNNING",
"schema": {
"keys_type": "DUP_KEYS",
"column": [
...
],
"num_short_key_columns": 1,
"num_rows_per_row_block": 1024,
...
"skip_write_index_on_load": false
},
"rs_metas": [
{
"rowset_id": 0,
"partition_id": 1289193,
"tablet_id": 1289222,
"tablet_schema_hash": 816078096,
"newest_write_timestamp": 1684805433,
...
"segments_overlap_pb": "NONOVERLAPPING"
}
],
"in_restore_mode": false,
"tablet_uid": {
"hi": -4301918259617109775,
"lo": 2820124814688031668
},
"preferred_rowset_type": "BETA_ROWSET",
"tablet_type": "TABLET_TYPE_DISK",
"replica_id": 1812028,
"enable_unique_key_merge_on_write": false,
"binlog_config": {
"enable": false,
"ttl_seconds": 9223372036854775807,
"max_bytes": 9223372036854775807,
"max_history_nums": 9223372036854775807
},
"compaction_policy": "size_based",
"time_series_compaction_goal_size_mbytes": 1024,
"time_series_compaction_file_count_threshold": 2000,
"time_series_compaction_time_threshold_seconds": 3600
}
```
* fix
2023-12-30 00:07:43 +08:00
b8cdbf4ffe
Fix](schema change) fix metadata inconsistency bug caused by enable light schema change ( #28255 ) ( #29060 )
2023-12-30 00:02:01 +08:00
59fdd5e42b
[fix](compaction) time series compaction policy -> compact consecutive empty rowsets ( #27299 )
...
Sometimes we need to merge a large number of empty versions to reduce meta-information.
2023-12-29 22:20:41 +08:00
07bd65abe0
[pipelineX](local shuffle) remove unused code in local shuffle to improve performance #29292
2023-12-29 20:30:28 +08:00
f0629da10a
[pipelineX](fix) Fix illegal memory access ( #29283 )
2023-12-29 18:35:41 +08:00
82635d4b59
[opt](memory) All LRU Cache inherit from LRUCachePolicy ( #28940 )
...
After all LRU Cache inherits from LRUCachePolicy, this will allow prune stale entry, eviction when memory exceeds limit, and define common properties. LRUCache constructor change to private, only allow LRUCachePolicy to construct it.
Impl DummyLRUCache, when LRU Cache capacity is 0, will no longer be meaningless insert and evict.
2023-12-29 16:15:56 +08:00
c3c34e10bb
[feature](executor) Add some check when create workload group/workload schedule policy ( #29236 )
2023-12-29 15:41:16 +08:00
411f819306
[Opt](join) opt the join copy column from build side ( #29007 )
2023-12-29 14:39:33 +08:00
738a1644ff
[refactor](operator) We dont need OperatorBuilder to instantiate Operator ( #29133 )
2023-12-29 13:54:54 +08:00
b9572f9de0
[pipelineX](fix) Fix pip scanner context bug ( #29229 )
2023-12-29 13:24:39 +08:00
70f5a26f44
[pipelineX](fix) Fix heap-use-after-free for AggSource dependency ( #29272 )
2023-12-29 12:50:41 +08:00
253846a99d
[improve](compaction) enable compaction priority scheduling ( #29261 )
2023-12-29 10:13:08 +08:00
a525d5c5a3
[refactor](decimal) change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion ( #29265 )
...
change type name Decimal128 to Decimal128V2, Decimal128I to Decimal128V3 to avoid confusion
2023-12-29 10:11:44 +08:00
269c1b189d
[improve](vtablet_writer) check runtime state is cancel when back pressure ( #29260 )
2023-12-29 09:11:24 +08:00
2f29dda5aa
[Fix](core) Fix file system scan deleted file ( #29266 )
2023-12-29 09:07:59 +08:00
fb0ed8c253
[fix](move-memtable) check missing tablets before commit ( #29223 )
2023-12-29 00:33:58 +08:00
efea006f3a
[ut](move-memtable) add CLOSE_LOAD before EOS ut case ( #29253 )
...
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com >
2023-12-29 00:33:34 +08:00
99a1e066b5
[fix](group_commit) group_commit is not support on table with property light_schema_change=false ( #29244 )
2023-12-29 00:26:38 +08:00
9be0f04506
(improv)[group commit] refactor some group commit code ( #29180 )
2023-12-29 00:26:10 +08:00
9a277a6f11
[fix](move-memtable) don't abort in replica write layer unless all replica fails ( #29257 )
2023-12-29 00:03:28 +08:00
