1. binding slot in order by that not show in project, such as:
SELECT c1 FROM t WHERE c2 > 0 ORDER BY c3
2. not check unbound when bind slot reference. Instead, do it in analysis check.
Introduce a SQL syntax for creating inverted index and related metadata changes.
```
-- create table with INVERTED index
CREATE TABLE httplogs (
ts datetime,
clientip varchar(20),
request string,
status smallint,
size int,
INDEX idx_size (size) USING INVERTED,
INDEX idx_status (status) USING INVERTED,
INDEX idx_clientip (clientip) USING INVERTED PROPERTIES("parser"="none")
)
DUPLICATE KEY(ts)
DISTRIBUTED BY RANDOM BUCKETS 10
-- add an INVERTED index to a table
CREATE INDEX idx_request ON httplogs(request) USING INVERTED PROPERTIES("parser"="english");
```
This PR implements the function of predicate inference
For example:
``` sql
select * from student left join score on student.id = score.sid where score.sid > 1
```
transformed logical plan tree:
left join
/ \
filter(sid >1) filter(id > 1) <---- inferred predicate
| |
scan scan
See `InferPredicatesTest` for more cases
The logic is as follows:
1. poll up bottom predicate then infer additional predicates
for example:
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id
1. poll up bottom predicate
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t.id = 1
2. infer
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t.id = 1 and t2.id = 1
finally transformed sql:
select * from (select * from t1 where t1.id = 1) t join t2 on t.id = t2.id and t2.id = 1
2. put these predicates into `otherJoinConjuncts` , these predicates are processed in the next
round of predicate push-down
Now only support infer `ComparisonPredicate`.
TODO: We should determine whether `expression` satisfies the condition for replacement
eg: Satisfy `expression` is non-deterministic
1. add back TPC-H regression test cases
2. fix decimal problem on aggregate function sum and agg introduced by #13764
3. fix memo merge group NPE introduced by #13900
Support Aliyun DLF
Support data on s3-compatible object storage, such as aliyun oss.
Refactor some interface of catalog, to make it more tidy.
Fix bug that the default text format field delimiter of hive should be \x01
Add a new class PooledHiveMetaStoreClient to wrap the IMetaStoreClient.
In GraphSimplifier, we can use simple cost to calculate the benefit.
And only when the best neighbor of the apply step is the processing edge, we need to update recursively.
1. add a feature that support statement having aggregate function in order by list. such as:
SELECT COUNT(*) FROM t GROUP BY c1 ORDER BY COUNT(*) DESC;
2. add clickbench analyze unit tests
This pr does three things：
1. Modified the framework of table-valued-function(tvf).
2. be support `fetch_table_schema` rpc.
3. Implemented `S3(path, AK, SK, format)` table-valued-function.
[What is DLF](https://www.alibabacloud.com/product/datalake-formation)
This PR is a preparation for support DLF, with some changes of multi catalog
1. Add RuntimeException for most of hive meta store or es client visit operation.
2. Add DLF related dependencies.
3. Move the checks of es catalog properties to the analysis phase of creating es catalog
TODO(in next PR):
1. Refactor the `getSplit` method to support not only hdfs, but s3-compatible object storage.
2. Finish the implementation of supporting DLF
Persist external catalog/db/table, including the columns of external tables.
After this change, external objects could have their own uniq ID through their lifetime,
this is required for the statistic information collection.
Add a rule to check the permission of a user who are executing a query. Forbid users who don't have SELECT_PRIV on some tables from executing queries on these tables.
