Generates partition names based on the granularity.
eg:
Year：prefix2020
Day: prefix20200325
Week: prefix2020_#, # is the week of year.
At the same time, for all granularity, align the partition range to 00:00:00.
#3153
implement subquery support for sub query in case when statement like
```
SELECT CASE
WHEN (
SELECT COUNT(*) / 2
FROM t
) > k4 THEN (
SELECT AVG(k4)
FROM t
)
ELSE (
SELECT SUM(k4)
FROM t
)
END AS kk4
FROM t;
```
this statement will be rewrite to
```
SELECT CASE
WHEN t1.a > k4 THEN t2.a
ELSE t3.a
END AS kk4
FROM t, (
SELECT COUNT(*) / 2 AS a
FROM t
) t1, (
SELECT AVG(k4) AS a
FROM t
) t2, (
SELECT SUM(k4) AS a
FROM t
) t3;
```
This commit support the non-correlated subquery in having clause.
For example:
select k1, sum(k2) from table group by k1 having sum(k2) > (select avg(k1) from table);
Also the non-scalar subquery is supportted in Doris.
For example:
select k1, sum(k2) from table group by k1 having sum(k2) > (select avg(k1) from table group by k2);
Doris will check the result row numbers of subquery in executing.
If more then one row returned by subquery, the query will thrown exception.
The implement method:
The entire outer query is regarded as inline view of new query.
The subquery in having clause is changed to the where predicate in this new query.
After this commit, tpc-ds 23,24,44 are supported.
This commit also support the subquery in ArithmeticExpr.
For example:
select k1 from table where k1=0.9*(select k1 from t);
to resolve the ISSUE: #3139
When user execute query by some client library such as python MysqlDb, if user execute like:
"select * from tbl1;" (with a comma at the end of statement)
The sql parser will produce 2 statements: `SelectStmt` and `EmptyStmt`.
Here we discard the `EmptyStmt` to make it act like one single statement.
This is for some compatibility. Because in python MysqlDb, if the first `SelectStmt` results in
some warnings, it will try to execute a `SHOW WARNINGS` statement right after the
SelectStmt, but before the execution of `EmptyStmt`. So there will be an exception:
`(2014, "Commands out of sync; you can't run this command now")`
I though it is a flaw of python MysqlDb.
However, in order to maintain the consistency of user use, here we remove all EmptyStmt
at the end to prevent errors.(Leave at least one statement)
But if user execute statements like:
`"select * from tbl1;;select 2"`
If first `select * from tbl1` has warnings, python MysqlDb will still throw exception.
fix a bug of const union query like `select null union select null`, this because the type of SlotDescriptor when clause is `select null` is null ,this will cause BE core dump, and FE find wrong cast function.
Related issue: #2663, #2828.
This CL support loading data into specified temporary partitions.
```
INSERT INTO tbl TEMPORARY PARTITIONS(tp1, tp2, ..) ....;
curl .... -H "temporary_partition: tp1, tp, .. " ....
LOAD LABEL db1.label1 (
DATA INFILE("xxxx")
INTO TABLE `tbl2`
TEMPORARY PARTITION(tp1, tp2, ...)
...
```
NOTICE: this CL change the FE meta version to 77.
There 3 major changes in this CL
## Syntax reorganization
Reorganized the syntax related to the `specify-partitions`. Removed some redundant syntax
definitions, and unified the syntax related to the `specify-partitions` under one syntax entry.
## Meta refactor
In order to be able to support specifying temporary partitions,
I made some changes to the way the partition information in the table is stored.
Partition information is now organized as follows:
The following two maps are reserved in OlapTable for storing formal partitions:
```
idToPartition
nameToPartition
```
Use the `TempPartitions` class for storing temporary partitions.
All the partition attributes of the formal partition and the temporary partition,
such as the range, the number of replicas, and the storage medium, are all stored
in the `partitionInfo` of the OlapTable.
In `partitionInfo`, we use two maps to store the range of formal partition
and temporary partition:
```
idToRange
idToTempRange
```
Use separate map is because the partition ranges of the formal partition and
the temporary partition may overlap. Separate map can more easily check the partition range.
All partition attributes except the partition range are stored using the same map,
and the partition id is used as the map key.
## Method to get partition
A table may contain both formal and temporary partitions.
There are several methods to get the partition of a table.
Typically divided into two categories:
1. Get partition by id
2. Get partition by name
According to different requirements, the caller may want to obtain
a formal partition or a temporary partition. These methods are
described below in order to obtain the partition by using the correct method.
1. Get by name
This type of request usually comes from a user with partition names. Such as
`select * from tbl partition(p1);`.
This type of request has clear information to indicate whether to obtain a
formal or temporary partition.
Therefore, we need to get the partition through this method:
`getPartition(String partitionName, boolean isTemp)`
To avoid modifying too much code, we leave the `getPartition(String
partitionName)`, which is same as:
`getPartition(partitionName, false)`
2. Get by id
This type of request usually means that the previous step has obtained
certain partition ids in some way,
so we only need to get the corresponding partition through this method:
`getPartition(long partitionId)`.
This method will try to get both formal partitions and temporary partitions.
3. Get all partition instances
Depending on the requirements, the caller may want to obtain all formal
partitions,
all temporary partitions, or all partitions. Therefore we provide 3 methods,
the caller chooses according to needs.
`getPartitions()`
`getTempPartitions()`
`getAllPartitions()`
Too much AlterJobsV2 may consume too much memory, which may cause FullGC. Clear some data for finished or cancelled alterJobs and remove them when expired.
Sometimes a replica is broken on BE, but FE does not notice that.
In this case, we have to manually delete that replica on BE.
If there are hundreds of replicas need to be handled, this is a disaster.
So I add a new stmt:
ADMIN SET REPLICA STATUS
which support setting tablet on specified BE as BAD or OK.
This CL solve the issue #3105
I add a new temporary table state WAITING_STABLE.
When an alter job is ready to start, it checks whether the table is stable. If it is not stable,
the table state is set to WAITING_STABLE. In this state, the tablet repair logic will continue to
repair the tablet until the table becomes stable.
After that, the table state will be reset to SCHEMA_CHANGE/ROLLUP and alter operations will begin.
This is just a temporary state, it does not need to be persistent, and only the master FE can see this state.
Backup job in BE only backup index which is visible, but the backup meta in FE contains the shadow index, after restore from this snapshot, the shadow index is visible to load process, and the tablets is not exist in BE, so load process would be cancelled. we could fix this bug by remove the useless shadow index at backup process.
2 Changes in this CL:
## Support multiple statements in one request like:
```
select 10; select 20; select 30;
```
ISSUE: #3049
For simple testing this CL, you can using mysql-client shell command tools:
```
mysql> delimiter //
mysql> select 1; select 2; //
+------+
| 1 |
+------+
| 1 |
+------+
1 row in set (0.01 sec)
+------+
| 2 |
+------+
| 2 |
+------+
1 row in set (0.02 sec)
Query OK, 0 rows affected (0.02 sec)
```
I add a new class called `OriginStatement.java`, to save the origin statement in string format with an index. This class is mainly for the following cases:
1. User send a multi-statement to the non-master FE:
`DDL1; DDL2; DDL3`
2. Currently we cannot separate the original string of a single statement from multiple statements. So we have to forward the entire statement to the Master FE. So I add an index in the forward request. `DDL1`'s index is 0, `DDL2`'s index is 1,...
3. When the Master FE handle the forwarded request, it will parse the entire statement, got 3 DDL statements, and using the `index` to get the specified the statement.
## Optimized the display of syntax errors
I have also optimized the display of syntax errors so that longer syntax errors can be fully displayed.
In a large scale cluster, we may rolling upgrade BEs, this patch add a
column named 'Version' for command 'show backends;', as well as website
'/system?path=//backends', to provide a method to check whether there
is any BE missing upgraded.
`DROP MATERIALIZE VIEW [ IF EXISTS ] <mv_name> ON [db_name].<table_name>`
Parameters:
IF EXISTS: Do not throw an error if the materialized view does not exist. A notice is issued in this case.
mv_name: The name of the materialized view to remove.
db_name: The name of db to which materialized view belongs.
table_name: The name of table to which materialized view belongs.
This bug occurred when BE make snapshot, the version required by fe had been merged into the cumulative version, so the snapshot task could not complete the task even if it retried. In order to solve this problem, the BackupJob could be set to CANCELLED, and the user could continue to retry the job.
Fix#3057
The index name in MaterializedViewMeta is still with `__doris_shadow` prefix
after schema change finished.
In this CL, I just remove the index name field in MaterializedViewMeta,
so that it would makes managing change of names less error-prone.
Firstly, add materialized index meta in olap table
The materialized index meta include index name, schema, schemahash, keystype etc.
The information itself scattered in each map is encapsulated into MaterializedIndexMeta.
Also the keys type of index meta maybe not same as keys type of base index after materialized view enabled.
Secondly, support the deduplicate mv.
If there is group by or aggregation function in create mv stmt, the keys type of mv is agg.
At the same time, the keys type of base table is duplicate.
For example
Duplicate table (k1, k2, v1)
MV (k1, k2) group by k1, k2
It should be aggregated during executing mv.
The default replication number of an olap table may not be set.
Every time we call `getReplicationNum()`, we have to check if it returns null,
which is inconvenience and may cause problem
So in this PR, I set a default value to table's replication number.
This bug is introduced by #2958
If there is no aggregated column in aggregate index, the index will be deduplicate table.
For example:
aggregate table (k1, k2, v1 sum)
mv index (k1, k2)
This kind of index is SPJG which same as `select k1, k2 from aggregate_table group by k1, k2`.
It also need to check the grouping column using following steps.
If there is no aggregated column in duplicate index, the index will be SPJ which passes the grouping verification directly.
Also after the supplement of index, the new candidate index should be checked the output columns also.
Format of some docs are incorrect for building the doc website.
* fix a bug that `gensrc` dir can not be built with -j.
* fix ut bug of CreateFunctionTest
This CL implements 3 new operations:
```
ALTER TABLE tbl ADD TEMPORARY PARTITION ...;
ALTER TABLE tbl DROP TEMPORARY PARTITION ...;
ALTER TABLE tbl REPLACE TEMPORARY PARTITION (p1, p2, ...);
```
User manual can be found in document:
`docs/documentation/cn/administrator-guide/alter-table/alter-table-temp-partition.md`
I did not update the grammar manual of `alter-table.md`.
This manual is too confusing and too big, I will reorganize this manual after.
This is the first part to implement the "overwrite load" feature mentioned in issue #2663.
I will implement the "load to temp partition" feature in next PR.
This CL also add GSON serialization method for the following classes (But not used):
```
Partition.java
MaterializedIndex.java
Tablet.java
Replica.java
```
The issue is #3011.
Reset the tablet and scan range info before compute it.
The old rollup selector has computed tablet and scan range info.
Then the new mv selector maybe compute tablet and scan range info again sometimes.
So, we need to reset those info in here.
Before this commit, the result is double when query is "select k1 ,k2 from aggregate_table "
This PR is to remove some unused log for unauthorized exception, some unauthorized access such as LVS probe request may cause connection exception which we should ignore.
There is a case where the META upload succeeded but the upload INFO failed, in which case the UPLOAD_INFO task will try again, but the META file has succeeded and filename.part has been renamed to `filename.md5sum`. The retry task will keep failing with rename and cannot complete the backup job. Therefore, the `file.md5sum` file needs to be deleted in advance
Fix#3001
This commit mainly implements the new materialized view selector which supports SPJ<->SPJG.
Two parameters are currently used to regulate this function.
1. test_materialized_view: When this parameter is set to true, the user can create a materialized view for the duplicate table by using 'CREATE MATERIALIZED VIEW' command.
At the same time, if the result of the new materialized views is different from the old version during the query, an error will be reported. This parameter is false by default, which means that the new version of the materialized view function cannot be enabled.
2. use_old_mv_selector: When this parameter is set to true, the result of the old version selector will be selected. If set to false, the result of the new version selector will be selected. This parameter is true by default, which means that the old selector is used.
If the default values of the above two parameters do not change, there will be no behavior changes in the current version.
The main steps for the new selector are as follows:
1. Predicates stage: This stage will mainly filter out all materialized views that do not meet the current query requirements.
2. Priorities stage: This stage will sort the results of the first stage and choose the best materialized view.
The predicates phase is divided into 6 steps:
1. Calculate the predicate gap between the current query and view.
2. Whether the columns in the view can meet the needs of the compensating predicates.
3. Determine whether the group by columns of view match the group by columns of query.
4. Determine whether the aggregate columns of view match the aggregate columns of query.
5. Determine whether the output columns of view match the output columns of query.
6. Add partial materialized views
The priorities phase is divided into two steps:
1. Find the materialized view that matches the best prefix index
2. Find the materialized view with the least amount of data
The biggest difference between the current materialized view selector and the previous one is that it supports SPJ <-> SPJG.
In this CL, the isAlive field in FsBroker class will be persisted in metadata, to solve the
problem describe in ISSUE: #2989
Notice: this CL update FeMetaVersion to 73
This CL modify 2 things:
1. When a routine load task submit failed, it will not be put back to the task queue.
2. The rpc timeout when executing a routine load task in BE is set to `query_timeout` of the task plan.
ISSUE: #2964
