binlog load Because txn exceeds the default value, resume is a failure,
and a friendly prompt message is given to the user, instead of prompting success now,
it still fails after a while, and the user will feel inexplicable
Issue Number: close#9468
Current fe check style check all files. But some rules should be only applied on production files.
Add suppressions to suppress some rules on test files.
Describe the overview of changes.
- rename GroupExpression
- use `HashSet<GroupExpression> groupExpressions` in `memo`
- add label of `Nereids` for CI
- remove `GroupExpr` from Plan
There are many places in FE where a group of BE nodes needs to be selected according to certain requirements. for example:
1. When creating replicas for a tablet.
2. When selecting a BE to execute Insert.
3. When Stream Load forwards http requests to BE nodes.
These operations all have the same logic. So this CL mainly changes:
1. Create a new `BeSelectionPolicy` class to describe the set of conditions for selecting BE.
2. The logic of selecting BE nodes in `SystemInfoService` has been refactored, and the following two methods are used uniformly:
1. `selectBackendIdsByPolicy`: Select the required number of BE nodes according to the `BeSelectionPolicy`.
2. `selectBackendIdsForReplicaCreation`: Select the BE node for the replica creation operation.
Note that there are some changes here:
For the replica creation operation, the round-robin method was used to select BE nodes before,
but now it is changed to `random` selection for the following reasons:
1. Although the previous logic is round-robin, it is actually random.
2. The final diff of the random algorithm will not be greater than 5%, so it can be considered that the random algorithm
can distribute the data evenly.
This patch supports utf8mb4 for mysql external table.
if someone needs a mysql external table with utf8mb4 charset, but only support charset utf8 right now.
When create mysql external table, it can add an optional propertiy "charset" which can set character fom mysql connection,
default value is "utf8". You can set "utf8mb4" instead of "utf8" when you need.
This pr provide a new pattern matching framework for Nereids optimizer.
The new pattern matching framework contains this concepts:
1. `Pattern`/`PatternDescriptor`: the tree node's multiple hierarchy shape, e.g. `logicalJoin(logicalJoin(), any()` pattern describe a plan that root is a `LogicalJoin` and the left child is `LogicalJoin` too.
2. `MatchedAction`: a callback function when the pattern matched, usually you can create new plan to replace the origin matched plan.
3. `MatchingContext`: the param pass through MatchedAction, contains the matched plan root and the PlannerContext.
4. `PatternMatcher`: contains PatternDescriptor and MatchedAction
5. `Rule`: a rewrite rule contains RuleType, PatternPromise, Pattern and transform function(equals to MatchedAction)
6. `RuleFactory`: the factory can help us build Rules easily. RuleFactory extends Patterns interface, and have some predefined pattern descriptors.
for example, Join commutative:
```java
public class JoinCommutative extends OneExplorationRuleFactory {
@Override
public Rule<Plan> build() {
return innerLogicalJoin().thenApply(ctx -> {
return new LogicalJoin(
JoinType.INNER_JOIN,
ctx.root.getOnClause(),
ctx.root.right(),
ctx.root.left()
);
}).toRule(RuleType.LOGICAL_JOIN_COMMUTATIVE);
}
}
```
the code above show the three step to create a Rule
1. 'innerLogicalJoin()' declare pattern is an inner logical join. 'innerLogicalJoin' is a predefined pattern.
2. invoke 'thenApply()' function to combine a MatchedAction, return a new LogicalJoin with exchange children.
3. invoke 'toRule()' function to convert to Rule
You can think the Rule contains three parts:
1. Pattern
2. transform function / MatchedAction
3. RuleType and RulePromise
So
1. `innerLogicalJoin()` create a `PatternDescriptor`, which contains a `Pattern`
2. `PatternDescriptor.then()` convert `PatternDescriptor` to `PatternMatcher,` witch contains Pattern and MatchedAction
3. `PatternMatcher.toRule()` convert `PatternMatcher` to a Rule
This three step inspired by the currying in function programing.
It should be noted, #9446 provide a generic type for TreeNode's children, so we can infer multiple hierarchy type in this pattern matching framework, so you can get the really tree node type without unsafely cast. like this:
```java
logicalJoin(logicalJoin(), any()).then(j -> {
// j can be inferred type to LogicalJoin<LogicalJoin<Plan, Plan>, Plan>
// so j.left() can be inferred type to LogicalJoin<Plan, Plan>,
// so you don't need to cast j.left() from 'Plan' to 'LogicalJoin'
var node = j.left().left();
})
```
Currently, we use `UtFrameUtils` to start a FE server in the FE unit test.
Each test class has to do some initialization and clean up stuff with the JUnit4
`@BeforeClass` and `@AfterClass` annotation. It's redundant and boring.
Besides, almost all the APIs in `UtFrameUtils` has a `ConnectContext` parameter, which is not easy to use.
This PR proposes to use an inherit-manner, i.e., wrap all the common logic in base class `TestWithFeService`,
leveraging the
JUnit5 `@BeforeAll` and `@AfterAll` annotation to narrow down the setup and cleanup lifecycle to each test class instance.
At the same time, the derived concrete test class could directly use utility methods inherited from the base class,
without calling a util class and passing a `ConnectContext` argument.
`UtFrameUtils` and `DorisAssert` are marked as deprecated. We could remove these two classes
if this refactor works well for a time.
Fix a bug.
The where condition cannot be pushed down because there is no derivation
eg:
select * from tb1 left join tb2 on tb1.id = tb2.id where tb2.id = 1;
The correct case is that the condition of "=1" needs to be deduced to tb1.id,
but the current implementation does not do the deduction
Nereids(new optimizer) code base
Nereids is new query planner for Doris. It include three main parts: parser, analyzer and optimizer.
The parser, generated by ANTLR4, transforms SQL into a logical plan with a tree structure. Analysis and optimization are performed on the logical plan of the tree structure. Each transformation is defined as a rule. The rule is applied to the logical plan using pattern matching. The implementation of the optimizer follows the approach in the Cascades paper.
Doris couldn't resolve the defaultFS of HDFS with HA configuration, so it could query hive table on HA HDFS.
This is because there's no way to send the HA configs to hive external table.
Describe the overview of changes.
Pass the ha configs to hive external table through create table properties.
Usage:
Example of creating hive table with ha configuration properties:
CREATE TABLE region (
r_regionkey integer NOT NULL,
r_name char(25) NOT NULL,
r_comment varchar(152)
) engine=hive properties
("database"="default",
"table"="region",
"hive.metastore.uris"="thrift://172.21.16.11:7004",
"dfs.nameservices"="hacluster",
"dfs.ha.namenodes.hacluster"="3,4",
"dfs.namenode.rpc-address.hacluster.3"="192.168.0.93:8020",
"dfs.namenode.rpc-address.hacluster.4"="172.21.16.11:8020", "dfs.client.failover.proxy.provider.hacluster"="org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
Add the http header size parameter to avoid failure due to too many fields when users import using stream load.
The normal default is 8192, and 10K is given here.
- Force change the existing olaptable's storage format from V1 to V2
- Forbidden to create new olap table with storage format == v1 OR do schema change that want to create new v1 format
1. Fix bug described in #9267
When report missing version replica, set last failed version to (replica version + 1)
2. Skip non-exist partition when handling transactions.
This pull request includes some implementations of the statistics(https://github.com/apache/incubator-doris/issues/6370), it will not affect any existing code and users will not be able to create statistics job.
After receiving the statistics statement and dividing the collection task, here we will start implementing the scheduling statistics task and updating the job information. Mainly include the following:
- Create a thread pool to schedule a certain number of tasks, and the number of concurrency is related to the configuration `cbo_concurrency_statistics_task_num`.
- After the task is completed, update the information of of the statistics Job.
* (Refactor)[Statistics] Fix lock risks in Statistics Job
1. Remove lock nesting between job and task
2. Solve the deadlock problem during job update
3. Avoid printing the log while holding the lock
* Add log
```
CREATE ROUTINE LOAD iaas.dws_nat ON dws_nat
WITH APPEND PROPERTIES (
"desired_concurrent_number"="2",
"max_batch_interval" = "20",
"max_batch_rows" = "400000",
"max_batch_size" = "314572800",
"format" = "json",
"max_error_number" = "0"
)
FROM KAFKA (
"kafka_broker_list" = "xxxx:xxxx",
"kafka_topic" = "nat_nsq",
"property.kafka_default_offsets" = "2022-04-19 13:20:00"
);
```
In the create statement example below, you can see
The user didn't specify the custom partitions.
So that 1. Fe will get all kafka partitions from server in routine load's scheduler.
The user set the default offset by datetime.
So that 2. Fe will get kafka offset by time from server in routine load's scheduler.
When 1 is success, meanwhile 2 is failed, the progress of this routine load may not contains any partitions and offsets.
Nevertheless, since newCurrentKafkaPartition which is get by kafka server may be always equal to currentKafkaPartitions,
the wrong progress will never be updated.
This pull request includes some implementations of the statistics(https://github.com/apache/incubator-doris/issues/6370), it will not affect any existing code and users will not be able to create statistics job.
After receiving the statistics collection statement, it generates a job. Here it implements the division of statistics collection jobs according to the following statistics categories:
table:
- `row_count`: table row count are critical in estimating cardinality and memory usage of scan nodes.
- `data_size`: table size, not applicable to CBO, mainly used to monitor and manage table size.
column:
- `num_distinct_value`: used to determine the selectivity of an equivalent expression.
- `min`: The minimum value.
- `max`: The maximum value.
- `num_nulls`: number of nulls.
- `avg_col_len`: the average length of a column, in bytes, is used for memory and network IO evaluation.
- `max_col_len`: the Max length of the column, in bytes, is used for memory and network IO evaluation.
After the job is divided, statistics tasks will be obtained.
use this stmt to show tablets storage format in be, if verbose is set,
will show detail message of tablet storage format.
e.g.
```
MySQL [(none)]> admin show tablet storage format;
+-----------+---------+---------+
| BackendId | V1Count | V2Count |
+-----------+---------+---------+
| 10002 | 0 | 2867 |
+-----------+---------+---------+
1 row in set (0.003 sec)
MySQL [test_query_qa]> admin show tablet storage format verbose;
+-----------+----------+---------------+
| BackendId | TabletId | StorageFormat |
+-----------+----------+---------------+
| 10002 | 39227 | V2 |
| 10002 | 39221 | V2 |
| 10002 | 39215 | V2 |
| 10002 | 39199 | V2 |
+-----------+----------+---------------+
4 rows in set (0.034 sec)
```
add storage format infomation to show full table statment.
```
MySQL [test_query_qa]> show full tables;
+-------------------------+------------+---------------+
| Tables_in_test_query_qa | Table_type | StorageFormat |
+-------------------------+------------+---------------+
| bigtable | BASE TABLE | V2 |
| test_dup | BASE TABLE | V2 |
| test | BASE TABLE | V2 |
| baseall | BASE TABLE | V2 |
| test_string | BASE TABLE | V2 |
+-------------------------+------------+---------------+
5 rows in set (0.002 sec)
```
