After doris support aggregation materialized view on duplicate table,
desc stmt of metadata is confused in sometimes. The reason is that
there is no grouping information in desc stmt of metadata.
For example:
There are two materialized view as following.
1. create materialized view k1_k2 as select k1, k2 from table;
2. create materialzied view deduplicated_k1_k2 as select k1, k2 from table group by k1, k2;
Before this commit, the metatdata in desc stmt is the same.
```
+-----------------------+-------+----------+------+-------+---------+-------+
| IndexName | Field | Type | Null | Key | Default | Extra |
+-----------------------+-------+----------+------+-------+---------+-------+
| k1_k2 | k1 | TINYINT | Yes | true | N/A | |
| | k2 | SMALLINT | Yes | true | N/A | |
| deduplicated_k1_k2 | k1 | TINYINT | Yes | true | N/A | |
| | k2 | SMALLINT | Yes | true | N/A | |
+-----------------------+-------+----------+------+-------+---------+-------+
```
So, we need to show the KeysType of materialized view in desc stmt.
Now, the desc stmt of all mvs is changed as following:
```
+-----------------------+---------------+-------+----------+------+-------+---------+-------+
| IndexName | IndexKeysType | Field | Type | Null | Key | Default | Extra |
+-----------------------+---------------+-------+----------+------+-------+---------+-------+
| k1_k2 | DUP_KEYS | k1 | TINYINT | Yes | true | N/A | |
| | | k2 | SMALLINT | Yes | true | N/A | |
| deduplicated_k1_k2 | AGG_KEYS | k1 | TINYINT | Yes | true | N/A | |
| | | k2 | SMALLINT | Yes | true | N/A | |
+-----------------------+---------------+-------+----------+------+-------+---------+-------+
```
NOTICE: this modify the the column of `desc` stmt.
The bug is described in issue: #3200.
This CL solve the problem by:
1. Refactor the alter operation conflict checking logic by introducing new classes `AlterOperations` and `AlterOpType`.
2. Allow add/drop temporary partition when dynamic partition feature is enabled.
3. Allow modifying table's property when there is temporary partition in table.
4. Make the properties `dynamic_partition.enable` optional, and default is true.
Doris support choose medium when create table, and the cluster balance strategy is dependent
between different storage medium, and most use will not specify the storage medium when create table,
even they kown that they should choose a storage medium, they have no idea about the
cluster's storage medium, so, I think we should make storage_medium and storage_cooldown_time
configurable, and this should be the admin's responsibility.
For Example, if the cluster's storage medium is HDD, but we need to change part of machines to SSD,
if we change the machine, the tablets before change is stored in HDD and they can't find a dest path
to migrate, and user will create table as usual, it will make all tablets stored in old machines and
the new machines will only store a little tablets. Without this config the only way is admin need
to traverse all partitions in cluster and change the property of storage_medium, it will increase
operational and maintenance costs.
So I add a FE config default_storage_medium, so that user can set the default storage medium.
This PR is to reduce the time cost for waiting transactions to be completed in same db by filter the running transactions in table level.
NOTICE: Update FE meta version to 79
Currently, in the Spark-Doris-Connector, when Spark iteratively obtains each row of data,
it needs to synchronously convert the Arrow format data into the row format required by Spark.
In order to speed up the conversion process, we can add an asynchronous thread in the Connector,
which is responsible for obtaining the Arrow format data from BE and converting it into the row
format required by Spark calculation
In our test environment, Doris cluster used 1 fe and 7 be (32C+128G). When using Spark-Doris-Connector
to query a table containing 67 columns, the original query returned 69 million rows of data
took about 2.5min, but after improvement, it reduced to about 1.6min, which reduced the time by about 30%
The subquery in having clause should be rewritten too.
If not, ExprRewriteRule will not be apply in subquery.
For example:
select k1, sum (k2) from table group by k1 having sum(k2) > (select t1 from table2 where t2 between 1 and 2);
```t1 between 1 and 2``` should be rewritten to ```t1 >=1 and t1<=2```.
Fixed#3205. TPC-DS 14 will be passed after this commit.
The main optimization points:
1. Use std::unordered_set instead of std::set, and use RowsetId.hi as RowsetId's hash value.
2. Minimize the scope of SpinLock in UniqueRowsetIdGenerator.
Profile comparation:
* Run UniqueRowsetIdGeneratorTest.GenerateIdBenchmark 10 times
old version | new version
6s962ms | 3s647ms
6s139ms | 3s393ms
6s234ms | 3s686ms
6s060ms | 3s447ms
5s966ms | 4s127ms
5s786ms | 3s994ms
5s778ms | 4s072ms
6s193ms | 4s082ms
6s159ms | 3s560ms
5s591ms | 3s654ms
issue #2344
* Add install/unintall Plugin statement
* Add show plugin statement
* Support install plugin through two ways:
* Built-in Plugin: use PluginMgr's register method.
* Dynamic Plugin: install by SQL statement, and the process:
1. check Plugin has already install?
2. download Plugin file from remote source or copy from local source
3. extract Plugin's .zip
4. read Plugin's plugin.properties, and check Plugin's Value
5. dynamic load .jar and init Plugin's main Class
6. invoke Plugin's init method
7. register Plugin into PluginMgr.
8. update meta
* Support FE Plugin dynamic uninstall process
1. check Plugin has install?
2. invoke Plugin's close method
3. delete Plugin from PluginMgr
4. update meta
* Add audit plugin interface
* Add plugin enable flags in Config
* Add plugin install path in Config, default plugin will install in ${DORIS_FE_PATH}/plugins
* Add FE plugins project
* Add audit plugin demo
The usage:
```
// install plugin and show plugins;
mysql>
mysql> install plugin from "/home/users/seaven/auditplugin.zip";
Query OK, 0 rows affected (0.05 sec)
mysql>
mysql> show plugins;
+-------------------+-------+---------------+---------+-------------+------------------------+--------+---------------------------------------+
| Name | Type | Description | Version | JavaVersion | ClassName | SoName | Sources |
+-------------------+-------+---------------+---------+-------------+------------------------+--------+---------------------------------------+
| audit_plugin_demo | AUDIT | just for test | 0.11.0 | 1.8.31 | plugin.AuditPluginDemo | NULL | /home/users/hekai/auditplugindemo.zip |
+-------------------+-------+---------------+---------+-------------+------------------------+--------+---------------------------------------+
1 row in set (0.00 sec)
mysql> show plugins;
+-------------------+-------+---------------+---------+-------------+------------------------+--------+---------------------------------------+
| Name | Type | Description | Version | JavaVersion | ClassName | SoName | Sources |
+-------------------+-------+---------------+---------+-------------+------------------------+--------+---------------------------------------+
| audit_plugin_demo | AUDIT | just for test | 0.11.0 | 1.8.31 | plugin.AuditPluginDemo | NULL | /home/users/hekai/auditplugindemo.zip |
+-------------------+-------+---------------+---------+-------------+------------------------+--------+---------------------------------------+
1 row in set (0.00 sec)
mysql> uninstall plugin audit_plugin_demo;
Query OK, 0 rows affected (0.04 sec)
mysql> show plugins;
Empty set (0.00 sec)
```
TODO:
*Config.plugin_dir should be created if missing
Support BE plugin framework, include:
* update Plugin Manager, support Plugin find method
* support Builtin-Plugin register method
* plugin install/uninstall process
* PluginLoader:
* dynamic install and check Plugin .so file
* dynamic uninstall and check Plugin status
* PluginZip:
* support plugin remote/local .zip file download and extract
TODO:
* We should support a PluginContext to transmit necessary system variable when the plugin's init/close method invoke
* Add the entry which is BE dynamic Plugin install/uninstall process, include:
* The FE send install/uninstall Plugin statement (RPC way)
* The FE meta update request with Plugin list information
* The FE operation request(update/query) with Plugin (maybe don't need)
* Add the plugin status upload way
* Load already install Plugin when BE start
All of columns which belong to top of tupleIds in query should be considered in mv selector.
For example:
`select k1 from table group by k1 having sum(v1) >1;`
The candidate index should contain k1 and v1 columns instead of only k1.
The rollup which only has k1 column should not be selected.
The issue #3174 describe in detail.
Earlier we introduced `BlockManager` to separate data access logic from
underlying file read and write logic.
This CL further unifies all `SegmentV2` data access to the `BlockManager`,
removes the previous `FileManager` class, and move the file cache to the `FileBlockManager`.
There are no logical changes to this CL.
After this CL, all user table data is read through the `WritableBlock` and `ReadableBlock`
returned by the `BlockManager`, and no file operations are performed directly.
Generates partition names based on the granularity.
eg:
Year：prefix2020
Day: prefix20200325
Week: prefix2020_#, # is the week of year.
At the same time, for all granularity, align the partition range to 00:00:00.
#3153
implement subquery support for sub query in case when statement like
```
SELECT CASE
WHEN (
SELECT COUNT(*) / 2
FROM t
) > k4 THEN (
SELECT AVG(k4)
FROM t
)
ELSE (
SELECT SUM(k4)
FROM t
)
END AS kk4
FROM t;
```
this statement will be rewrite to
```
SELECT CASE
WHEN t1.a > k4 THEN t2.a
ELSE t3.a
END AS kk4
FROM t, (
SELECT COUNT(*) / 2 AS a
FROM t
) t1, (
SELECT AVG(k4) AS a
FROM t
) t2, (
SELECT SUM(k4) AS a
FROM t
) t3;
```
This commit support the non-correlated subquery in having clause.
For example:
select k1, sum(k2) from table group by k1 having sum(k2) > (select avg(k1) from table);
Also the non-scalar subquery is supportted in Doris.
For example:
select k1, sum(k2) from table group by k1 having sum(k2) > (select avg(k1) from table group by k2);
Doris will check the result row numbers of subquery in executing.
If more then one row returned by subquery, the query will thrown exception.
The implement method:
The entire outer query is regarded as inline view of new query.
The subquery in having clause is changed to the where predicate in this new query.
After this commit, tpc-ds 23,24,44 are supported.
This commit also support the subquery in ArithmeticExpr.
For example:
select k1 from table where k1=0.9*(select k1 from t);
to resolve the ISSUE: #3139
When user execute query by some client library such as python MysqlDb, if user execute like:
"select * from tbl1;" (with a comma at the end of statement)
The sql parser will produce 2 statements: `SelectStmt` and `EmptyStmt`.
Here we discard the `EmptyStmt` to make it act like one single statement.
This is for some compatibility. Because in python MysqlDb, if the first `SelectStmt` results in
some warnings, it will try to execute a `SHOW WARNINGS` statement right after the
SelectStmt, but before the execution of `EmptyStmt`. So there will be an exception:
`(2014, "Commands out of sync; you can't run this command now")`
I though it is a flaw of python MysqlDb.
However, in order to maintain the consistency of user use, here we remove all EmptyStmt
at the end to prevent errors.(Leave at least one statement)
But if user execute statements like:
`"select * from tbl1;;select 2"`
If first `select * from tbl1` has warnings, python MysqlDb will still throw exception.
1. BlockManager has been added into StorageEngine.
So StorageEngine should be initialized when starting BetaRowset unit test.
2. Cache should not use the same buf to store value, otherwise the address
will be freed twice and crash.
improve performent of hash join when build table has to many duplicated rows, this will cause hash table collisions and slow down the probe performence.
In this pr when join type is semi join or anti join, we will build a hash table without duplicated rows.
benchmark:
dataset: tpcds dataset `store_sales` and `catalog_sales`
```
mysql> select count(*) from catalog_sales;
+----------+
| count(*) |
+----------+
| 14401261 |
+----------+
1 row in set (0.44 sec)
mysql> select count(distinct cs_bill_cdemo_sk) from catalog_sales;
+------------------------------------+
| count(DISTINCT `cs_bill_cdemo_sk`) |
+------------------------------------+
| 1085080 |
+------------------------------------+
1 row in set (2.46 sec)
mysql> select count(*) from store_sales;
+----------+
| count(*) |
+----------+
| 28800991 |
+----------+
1 row in set (0.84 sec)
mysql> select count(distinct ss_addr_sk) from store_sales;
+------------------------------+
| count(DISTINCT `ss_addr_sk`) |
+------------------------------+
| 249978 |
+------------------------------+
1 row in set (2.57 sec)
```
test querys:
query1: `select count(*) from (select store_sales.ss_addr_sk from store_sales left semi join catalog_sales on catalog_sales.cs_bill_cdemo_sk = store_sales.ss_addr_sk) a;`
query2: `select count(*) from (select catalog_sales.cs_bill_cdemo_sk from catalog_sales left semi join store_sales on catalog_sales.cs_bill_cdemo_sk = store_sales.ss_addr_sk) a;`
benchmark result:
||query1|query2|
|:--:|:--:|:--:|
|before|14.76 sec|3 min 16.52 sec|
|after|12.64 sec|10.34 sec|
fix a bug of const union query like `select null union select null`, this because the type of SlotDescriptor when clause is `select null` is null ,this will cause BE core dump, and FE find wrong cast function.
Related issue: #2663, #2828.
This CL support loading data into specified temporary partitions.
```
INSERT INTO tbl TEMPORARY PARTITIONS(tp1, tp2, ..) ....;
curl .... -H "temporary_partition: tp1, tp, .. " ....
LOAD LABEL db1.label1 (
DATA INFILE("xxxx")
INTO TABLE `tbl2`
TEMPORARY PARTITION(tp1, tp2, ...)
...
```
NOTICE: this CL change the FE meta version to 77.
There 3 major changes in this CL
## Syntax reorganization
Reorganized the syntax related to the `specify-partitions`. Removed some redundant syntax
definitions, and unified the syntax related to the `specify-partitions` under one syntax entry.
## Meta refactor
In order to be able to support specifying temporary partitions,
I made some changes to the way the partition information in the table is stored.
Partition information is now organized as follows:
The following two maps are reserved in OlapTable for storing formal partitions:
```
idToPartition
nameToPartition
```
Use the `TempPartitions` class for storing temporary partitions.
All the partition attributes of the formal partition and the temporary partition,
such as the range, the number of replicas, and the storage medium, are all stored
in the `partitionInfo` of the OlapTable.
In `partitionInfo`, we use two maps to store the range of formal partition
and temporary partition:
```
idToRange
idToTempRange
```
Use separate map is because the partition ranges of the formal partition and
the temporary partition may overlap. Separate map can more easily check the partition range.
All partition attributes except the partition range are stored using the same map,
and the partition id is used as the map key.
## Method to get partition
A table may contain both formal and temporary partitions.
There are several methods to get the partition of a table.
Typically divided into two categories:
1. Get partition by id
2. Get partition by name
According to different requirements, the caller may want to obtain
a formal partition or a temporary partition. These methods are
described below in order to obtain the partition by using the correct method.
1. Get by name
This type of request usually comes from a user with partition names. Such as
`select * from tbl partition(p1);`.
This type of request has clear information to indicate whether to obtain a
formal or temporary partition.
Therefore, we need to get the partition through this method:
`getPartition(String partitionName, boolean isTemp)`
To avoid modifying too much code, we leave the `getPartition(String
partitionName)`, which is same as:
`getPartition(partitionName, false)`
2. Get by id
This type of request usually means that the previous step has obtained
certain partition ids in some way,
so we only need to get the corresponding partition through this method:
`getPartition(long partitionId)`.
This method will try to get both formal partitions and temporary partitions.
3. Get all partition instances
Depending on the requirements, the caller may want to obtain all formal
partitions,
all temporary partitions, or all partitions. Therefore we provide 3 methods,
the caller chooses according to needs.
`getPartitions()`
`getTempPartitions()`
`getAllPartitions()`
Too much AlterJobsV2 may consume too much memory, which may cause FullGC. Clear some data for finished or cancelled alterJobs and remove them when expired.
