Add a rule to optimize CASE WHEN expression.
Rewrite rule to convert CASE WHEN to IF.
For example:
CASE WHEN a > 1 THEN 1 ELSE 0 END -> IF(a > 1, 1, 0)
1. support struct data type
2. add array / map / struct literal syntax
3. fix array union / intersect / except type coercion
4. fix explict cast data type check for array
5. fix bound function type coercion
New structure for delete sub predicate.
Delete sub predicate uses a string type condition_str to stored temporarily now and fields will be extracted from it using std::regex, which may introduces stack overflow when matching a extremely large string(bug of libc).
Now we attempt to use a new PB structure to hold the delete sub predicate, to avoid that problem.
message DeleteSubPredicatePB {
optional int32 column_unique_id = 1;
optional string column_name = 2;
optional string op = 3;
optional string cond_value = 4;
}
Currently, 2 versions of sub predicate will both be filled. For query, we use the v2, and during compaction we still use v1. The old rowset meta with delete predicates which had sub predicate v1 will be attempted to convert to v2 when read from PB. Moreover, efforts will be made to rewrite these meta with the new delete sub predicate.
Make preparation to use column unique id to specify a column globally.
Using the column unique id rather than the column name to identify a column is vital for flexible schema change. The rewritten delete predicate will attach column unique id.
Support such sql grammar, Just for compatibility
```sql
select table_name
from information_schema.tables
where table_schema collate utf8_general_ci = 'information_schema'
and table_name collate utf8_general_ci = 'parameters';
```
The normalizedAgg rule can change the output of agg.
For example:
```
select c1 as c, c1 from t having c1 > 0
```
The normalizedAgg rule will make a plan with output c, which can cause the having filter error
Therefore, the output exprId should be unchanged after normalized
This is a follow up pr of #23438. It's not correct to forbid substitute all literal exprs, only need to prevent substitute literal expr in function's param list.
Add leading hint and ordered hint. Usage:
select /*+ ordered / * from a join b on xxx; which will limit join order to original order
select /+ leading ({b a}) */ from a join b on xxx; which will change join order to b join a.
#### `expr_depth_limit`
Default：3000
IsMutable：true
Limit on the depth of an expr tree. Exceed this limit may cause long analysis time while holding db read lock. Do not set this if you know what you are doing
#### `expr_children_limit`
Default：10000
IsMutable：true
Limit on the number of expr children of an expr tree. Exceed this limit may cause long analysis time while holding database read lock.
Fix bug that mistaken stats when analyzing table incrementally and partition number less than 512
Fix bug that cron expression lost during analyzing
Mark system job as running after registered to AnalysisManager to avoid submit same jobs if previous one take long time
For some certain bugs, jobs is stuck in FE by the table state. For example, There is a bug which causes table remains ROLLUP state after adding rollup job, then other alter jobs later will not succeed because the table state is always ROLLUP but not NORMAL.
This commit adds a statement which is used to set the state of the specified table.
1. do not split compress data file
Some data file in hive is compressed with gzip, deflate, etc.
These kinds of file can not be splitted.
2. Support lz4 block codec
for hive scan node, use lz4 block codec instead of lz4 frame codec
4. Support snappy block codec
For hadoop snappy
5. Optimize the `count(*)` query of csv file
For query like `select count(*) from tbl`, only need to split the line, no need to split the column.
Need to pick to branch-2.0 after this PR: #22304
```
CREATE ROW POLICY test_row_policy_1 ON test.table1
AS {RESTRICTIVE|PERMISSIVE} [TO user] [TO ROLE role] USING (id in (1, 2)); // add `to role`
DROP [ROW] POLICY [IF EXISTS] test_row_policy;//delete `for user` and `on table`
SHOW ROW POLICY [FOR user][FOR ROLE role] // add `for role`
```
Sometimes, the partitions of a hive table may on different storage, eg, some is on HDFS, others on object storage(cos, etc).
This PR mainly changes:
1. Fix the bug of accessing files via cosn.
2. Add a new field `fs_name` in TFileRangeDesc
This is because, when accessing a file, the BE will get a hdfs client from hdfs client cache, and different file in one query
request may have different fs name, eg, some of are `hdfs://`, some of are `cosn://`, so we need to specify fs name
for each file, otherwise, it may return error:
`reason: IllegalArgumentException: Wrong FS: cosn://doris-build-1308700295/xxxx, expected: hdfs://[172.xxxx:4007](http://172.xxxxx:4007/)`
When upgrading from 1.2 to 2.x(future version higher than 2.0), the default value of parameter broadcast_right_table_scale_factor may not be upgraded from old default value 10.0 to new default 0.0, which will cause the broadcast join behavior unexpected and may have a big performance impact. This pr will force to reset the value to new default value 0.0, to make sure the behavior correct.
Nereids doesn't support view based table value function, because tvf view doesn't contain the proper qualifier (catalog, db and table name). This pr is to support this function.
Also, fix nereids table value function explain output exprs incorrect bug.
The sql like
Select count(1) from view
would contain all the columns in old planner's execution plan, which is slow, because BE need to read all the column in data files. This pr is to improve the plan to only contain one column.
* Revert "[fix](testcase) fix test case failure of insert null value into not null column (#20963)"
This reverts commit 55a6649da962fb170ddb40fea8ef26bdc552a51a.
Mannual Revert "fix in strict mode, return error for insert if datatype convert fails (#20378)"
This mannual reverts commit 1b94b6368f5e871c9a0fe53dd7c64409079a4c9d
* fix case failure
