1. Change all required fields to optional
Although they all "required", but it not recommended to use `required`, because it is hard to modify in future.
2. Fix a missing field bug
1. rename PhysicalHeapSort to PhysicalQuickSort
2. add LogicalTopN and PhysicalTopN
3. add implementation rule for LogicalTopN
4. add a interface Sort for both logical and physical sort
5. add a interface TopN for both logical and physical top-n
6. add a AbstractPhysicalSort as super class of PhysicalQuickSort and PhysicalTopN
When we do write current plan with its child. Currently, nereids memo do nothing indeed.
This PR fix this situation by add a predicate in rewrite that only replaced target group and set newGroupExpressionGenerated to false when target is null or target is same with existed group expression's group.
here we need to handle one situation that original target is not the same with
existedGroupExpression.getOwnerGroup(). In this case, if we change target to
existedGroupExpression.getOwnerGroup(), we could not rewrite plan as we expected and the plan
will not be changed anymore.
Think below example:
We have a plan like this:
```
Original (Group 2 is root):
Group2: Project(outside)
Group1: |---Project(inside)
Group0: |---UnboundRelation
and we want to rewrite group 2 by Project(inside, GroupPlan(group 0))
After rewriting we should get (Group 2 is root):
Group2: Project(inside)
Group0: |---UnboundRelation
Group1: Project(inside)
```
After rewriting, Group 1's GroupExpression is not in GroupExpressionsMap anymore and Group 1 is unreachable.
Merge Group 1 into Group 2 is better, but in consideration of there is others way to let a Group take into
unreachable. There's no need to complicate to add a merge step. Instead, we need to have a clear step to
remove unreachable groups and GroupExpressions after rewrite.
1. add StatementContext, and PlannerContext is renamed to CascadsContext. CascadsContext belong to a StatementContext, and StatementContext belong to a ConnectionContext, and the lifecycle increases in turn. StatementContext can wrap some statement's lifecycle-related state, such as ExpressionId, TableLock. MemoTestUtil can simplify create a CascadesContext and Memo for test.
2. add PlanPreprocessor to process parsed logical plan before copy into memo. and add a PlanPostprocessor to process physical plan after copy out from memo.
3. utilize PlanPreprocessor to process SET_VAR hint, the class is EliminateLogicalSelectHint
4. pass the limit clause in regression test case, in set_var.groovy
This PR proposes to add a plan checker to facilitate plan checking in unit tests.
Usage of plan checker is like below:
```java
new PlanChecker()
.plan(myPlan)
.applyBottomUp(myRule)
.matches(expectedPattern);
```
I try to fix the bug in #10095. the error occurred when I first create a empty table and query it.
But I can't reproduced it again.
So I add a warn log here to observer
Before, if a table is unpartitioned, when executing following alter stmt:
```
alter table tbl1 set ("replication_num" = "1");
```
Only the tbl1 partition's replication_num is changed
(for unpartitioned table, it also has a single partition with same name as table's)
But the table's default replication_num is unchanged.
So when executing `show create table tbl1`, you will find that the replication_num is still the origin value.
This CL mainly changes:
1. For unpartitioned table, if user change it's replication num, both table's and partition's replication_num will be changed.
This pr mainly to supplement the syntax of the previous pr(#8861),
it supports users to manually inject statistics, including table, partition, and column statistics.
table/partition stats type:
- row_count
- data_size
column stats type:
- ndv
- avg_size
- max_size
- num_nulls
- min_value
- max_value
Modify table or partition statistics:
```
ALTER TABLE table_name
SET STATS ('k1' = 'v1', ...) [PARTITIONS(p_name1, p_name2...)]
```
Modify column statistics:
```
ALTER TABLE table_name MODIFY COLUMN columnName
SET STATS ('k1' = 'v1', ...) [PARTITIONS(p_name1, p_name2...)]
```
Some notes:
- Only support statistics injected into olap type tables.
- Statistics injected into temporary partitions are not supported.
- When injecting statistics, if it is a partitioned table, users need to specify a partition name.
- If multiple partitions are specified, the same stats will be injected on multiple partitions.
- The current code also has mock statistics @zhengshij
* [feature](planner): push limit to olapscan when meet sort.
* if olap_scan_node's sort_info is set, push sort_limit, read_orderby_key
and read_orderby_key_reverse for olap scanner
* There is a common query pattern to find latest time serials data.
eg. SELECT * from t_log WHERE t>t1 AND t<t2 ORDER BY t DESC LIMIT 100
If the ORDER BY columns is the prefix of the sort key of table, it can
be greatly optimized to read much fewer data instead of read all data
between t1 and t2.
By leveraging the same order of ORDER BY columns and sort key of table,
just read the LIMIT N rows for each related segment and merge N rows.
1. set read_orderby_key to true for read_params and _reader_context
if olap_scan_node's sort info is set.
2. set read_orderby_key_reverse to true for read_params and _reader_context
if is_asc_order is false.
3. rowset reader force merge read segments if read_orderby_key is true.
4. block reader and tablet reader force merge read rowsets if read_orderby_key is true.
5. for ORDER BY DESC, read and compare in reverse order
5.1 segment iterator read backward using a new BackwardBitmapRangeIterator and
reverse the result block before return to caller.
5.2 VCollectIterator::LevelIteratorComparator, VMergeIteratorContext return
opposite result for _is_reverse order in its compare function.
Co-authored-by: jackwener <jakevingoo@gmail.com>
1. Add InPredicate expression parser and translator
2. Add regression-test for In predicate (in nereids_syntax)
3. Support NOT EqualTo and NOT InPredicate in ExpressionTranslator#visitNot()
