Nereids(new optimizer) code base
Nereids is new query planner for Doris. It include three main parts: parser, analyzer and optimizer.
The parser, generated by ANTLR4, transforms SQL into a logical plan with a tree structure. Analysis and optimization are performed on the logical plan of the tree structure. Each transformation is defined as a rule. The rule is applied to the logical plan using pattern matching. The implementation of the optimizer follows the approach in the Cascades paper.
Doris couldn't resolve the defaultFS of HDFS with HA configuration, so it could query hive table on HA HDFS.
This is because there's no way to send the HA configs to hive external table.
Describe the overview of changes.
Pass the ha configs to hive external table through create table properties.
Usage:
Example of creating hive table with ha configuration properties:
CREATE TABLE region (
r_regionkey integer NOT NULL,
r_name char(25) NOT NULL,
r_comment varchar(152)
) engine=hive properties
("database"="default",
"table"="region",
"hive.metastore.uris"="thrift://172.21.16.11:7004",
"dfs.nameservices"="hacluster",
"dfs.ha.namenodes.hacluster"="3,4",
"dfs.namenode.rpc-address.hacluster.3"="192.168.0.93:8020",
"dfs.namenode.rpc-address.hacluster.4"="172.21.16.11:8020", "dfs.client.failover.proxy.provider.hacluster"="org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
Support register suite plugin to add third-party function.
See
1. register in: ${DORIS_HOME}/regression-test/plugins/plugin_example.groovy
2. usage: ${DORIS_HOME}/regression-test/suites/demo/test_plugin.groovy
3. doc: ${DORIS_HOME}/docs/zh-CN/developer-guide/regression-testing.md
Add the http header size parameter to avoid failure due to too many fields when users import using stream load.
The normal default is 8192, and 10K is given here.
- Force change the existing olaptable's storage format from V1 to V2
- Forbidden to create new olap table with storage format == v1 OR do schema change that want to create new v1 format
1. Fix bug described in #9267
When report missing version replica, set last failed version to (replica version + 1)
2. Skip non-exist partition when handling transactions.
