Commit Graph

15672 Commits

Author SHA1 Message Date
afd55123ea [fix](partial update) report error directly if missing rowsets during doing alignment when flushing memtable due to compaction (#28677) 2023-12-20 11:49:11 +08:00
fe184e322a [code](pipelineX) refine some pipelineX code (#28570) 2023-12-20 11:45:06 +08:00
3e85797443 [fix](Nereids) should only do bind relation in view analyzer (#28637) 2023-12-20 10:47:51 +08:00
0ae87e705e [fix](planner) ctas update datev1 to datev2 should use equals (#28641) 2023-12-20 10:31:49 +08:00
23784787b1 [fix](planner)should bind expr using no slot to correct tuple (#28656) 2023-12-20 10:29:59 +08:00
c26c0c31a5 [refactor](runtimefilter) do not use QueryContext in runtime filter (#28559) 2023-12-20 10:28:55 +08:00
4c0080e237 [feat](Nereids) support outer join and aggregate bitmap rewrite by mv (#28596)
- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite
2023-12-20 10:23:30 +08:00
17268346d3 [Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction (#28062)" (#28674)
This reverts commit 485d7db5161efb011b43a10237df1ca83d5e5c3d.
2023-12-20 10:13:43 +08:00
xy
e59c707022 [fix](stream_load)fix bug for stream (#27752)
1. forbid thed  stream_load without content-length or chunked Transfer Encoding
2. forbid thed  stream_load both with content-length and chunked Transfer Encoding

Co-authored-by: xingying01 <xingying01@corp.netease.com>
2023-12-20 09:47:52 +08:00
044d7830c9 [improvement](transaction) reduce publish txn log (#28277) 2023-12-20 09:31:21 +08:00
81643d78e7 [fix](regression) fix test_alter_colocate_table due to force_olap_table_replication_num=3 (#28575) 2023-12-20 09:28:44 +08:00
0c535f93d7 [fix](regression) restore reserve num replicas (#28541) 2023-12-20 09:28:24 +08:00
eb50db1f3f [fix](regression) fix test_set_replica_status due to force_olap_table_replication_num=3 (#28573) 2023-12-20 09:27:18 +08:00
e33cf52dba [fix](test) fix ccr test cases (#28664) 2023-12-20 09:26:55 +08:00
9aa878ea1f [Fix](Job)Fixed job scheduling missing certain time window schedules (#28659)
Since scheduling itself consumes a certain amount of time, the start time of the time window should not be the current time, but the end time of the last schedule.
2023-12-20 09:21:15 +08:00
fb3b0afb85 [opt](task-assignment) use consistent hash as default task assigner and cache the consistent hash ring (#28522)
1. Use consistent hash algo as the default assigner for file query scan node
    A consistent assignment can better utilize the page cache of BE node.

2. Cache the consistent hash ring
    Init a consistent hash ring is time-consuming because there a thousands of virtual node need to be added.
    So cache it for better performance
2023-12-19 22:29:35 +08:00
aab859be56 [enhance](partition_id) check partition id before store meta (#28055) 2023-12-19 21:31:41 +08:00
b2d16856b4 [Fix](memtable) fix shrink_memtable_by_agg without duplicated keys (#28660)
remove duplicated logic:
```
vectorized::Block in_block = _input_mutable_block.to_block();
_put_into_output(in_block);
```
`_input_mutable_block.to_block()` will move `_input_mutable_block`, and lead to `flush` with empty block
2023-12-19 20:45:16 +08:00
d2275e73df [Fix](transactional-hive) Fix hive transactional table return empty result. (#28518) 2023-12-19 19:38:48 +08:00
7da86c37ec [fix](hive) add support for quoteChar and seperatorChar for hive (#28613)
add support for quoteChar and seperatorChar .
2023-12-19 19:35:03 +08:00
0883d47832 [Enhance](broker) add inputstream expire scheduled checker to avoid memory leak for broker scan (#28589)
This pr introduces 2 broker conf:

1. enable_input_stream_expire_check: which indicates whether enable inputStream expire check.
2. input_stream_expire_seconds: which indicates the timeout seconds for inputStream since last update.
2023-12-19 19:24:29 +08:00
9c9249e911 [feature](inverted index) add ignore_above property to prevent long s… (#28585)
When string is too long, clucene will throw an error. 
And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default.
We add an poperty `ignore_above` for user to customize.
2023-12-19 18:54:36 +08:00
73a3d84c5e [chore](user) Add user property parallel_fragment_exec_instance_num (#28447) 2023-12-19 18:33:01 +08:00
15e31d74e3 [feature](expr) Support kill query by query_id (#28530)
Issue Number: open #28517
2023-12-19 18:18:40 +08:00
71b7dcfb8f [feature][executor]support workload schedule policy (#28443) 2023-12-19 18:00:02 +08:00
c72191eb9e [refactor](profile&names) using dst_id in pipelinex profile to be same as non pipeline; rename some function names (#28626)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-12-19 17:44:29 +08:00
b99ac973d2 [fix](meta) update killed query state (#) (#25917) 2023-12-19 17:44:15 +08:00
a871409259 [feature](mtmv)after creating a materialized view, if other operations fail, roll back (#28621)
after create mtmv, if create job failed,need drop mtmv
2023-12-19 17:25:31 +08:00
111185407c [Improve](tvf)jni-avro support split file (#27933) 2023-12-19 16:37:34 +08:00
c41562a37b [Bug](cooldown) Fix problem that followers may never completely cooldown (#28561) 2023-12-19 16:03:59 +08:00
95073053bc [chore] Add bvar for meta operations of BE (#28374) 2023-12-19 15:54:19 +08:00
64ebdb2777 [fix](regression)Change analyze_timeout to global. (#28587)
Fix hive statistics regression case. analyze_timeout is a global session variable.
2023-12-19 15:52:38 +08:00
372a2e20f9 [fix](Nereids) stats estimation of lessThan #28444 2023-12-19 15:45:03 +08:00
a104914880 [feature](mtmv)create mtmv support refresh_partition_num (#28566)
- create/alter mtmv support refresh_partition_num
- mtmv task according to refresh_partition_num executes refresh tasks in batches
- `tasks` tvf add column `CompletedPartitions` and `progress`
- fix mtmv can not `show temp partition` and `drop temp partition`
- fix task can not get error msg when insert overwrite error
- fix when the partition field is capitalized, the verification of creating a mtmv does not pass
2023-12-19 15:09:19 +08:00
b142ade69e [refactor](renamefile) rename some files according to the class names (#28606) 2023-12-19 14:10:11 +08:00
1253ed006e [fix](memtable-limiter) do not block write if load mem usage is low (#28602)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2023-12-19 13:28:17 +08:00
9434ee5710 [fix](load) fix memtracking orphan too large (#28600) 2023-12-19 12:41:19 +08:00
d7dd7b775b enhance performance for broken tablet checking under multi-core scenario with a coarse-grained read lock (#28552) 2023-12-19 12:33:34 +08:00
e362bf674f [regression-test](memtable) test memtable flush is high priority for vtable writerV1 (#28502) 2023-12-19 12:33:12 +08:00
9c17f31767 [regression](memtable) add case for memtable flush error handle (#28285)
Co-authored-by: ziyang zhang <zhangziyang@stu.cdut.edu.cn>
2023-12-19 12:32:55 +08:00
1922638004 [Improvement](regression) change compound predicate regression case name to make it more clear (#28612) 2023-12-19 12:17:39 +08:00
87d36d0ffa [opt](Nereids)when both Nereids and old parsers report errors, prompt error messages for the Nereids (#28580) 2023-12-19 12:04:08 +08:00
85317122cf [Enhance](regression)enhance jdbc case to adapt to use case concurrency (#28565)
enhance jdbc case to adapt to use case concurrency
2023-12-19 11:59:00 +08:00
d24a1645d9 [improvement](http) add show_table_data http api (#28380)
In some cases, users need to get the data size of single replica of a table, and evaluate certain actions based on this, such as estimating the precise backup size.

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-12-19 11:44:33 +08:00
ddba98159e [feature](Nereids) support datev1 and datetimev1 (#28581) 2023-12-19 11:31:28 +08:00
8da9bb659d [feature](Nereids): eliminate semi join (#28588)
Eliminate Semi/Anti Join which is FALSE or TRUE.
2023-12-19 11:30:44 +08:00
7086956b4f [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch (#28597)
* [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch

return RuntimeError if copy_column_data_to_block nullable mismatch to avoid coredump in input_col_ptr->filter_by_selector(sel_rowid_idx, select_size, raw_res_ptr) .

The problem is reported by a doris user but I can not reproduce it, so there is no testcase added currently.

* clang format
2023-12-19 11:30:02 +08:00
5d3eea0ef0 [fix](stacktrace) ignore stacktrace for error code INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED (#27898) (#28598)
* ignore stacktrace for error INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED

* AndBlockColumnPredicate::evaluate
2023-12-19 11:14:41 +08:00
88f0bee722 [Performance](point query)Opimize partition prune for point query (#28150)
* [Performance](point query)Opimize partition prune for point query
2023-12-19 11:00:13 +08:00
d17ac99abe [feature](coverage): refresh the coverage file before exiting the program (#28354) 2023-12-19 10:54:57 +08:00