Commit Graph

918 Commits

Author SHA1 Message Date
fa8e4ec2f0 [fix] Disable cast operation of object type (#8882)
Disable cast between string and object type(bitmap, hll, quantile_state)
2022-04-08 09:13:56 +08:00
e3daa9580a [Fix](Lateral View) The Error expr type when exploding a function result of inline view (#8851)
Fixed #8850

The column in inline view maybe a function instead of slotRef.
So when this column is used as the input of explode function,
it can't be converted to slotRef.

The correct way is to treat it as an Expr and extract the required slotRef for materialization.
For example:
```
with d as (select k1+k1 as k1_plus from table)
select k1_plus from d explode_split(k1_plus, ",")
```
FnExp: SlorRef<k1_plus>
SubstituteFnExpr: functionCallExpr<k1+k1>
originSlotRefList: SlotRef<k1>
2022-04-08 09:08:55 +08:00
318feb01f3 [improvement](account) support to account management sql (#8849)
Add [IF EXISTS] support to following statements:
- CREATE [IF NOT EXISTS] USER
- CREATE [IF NOT EXISTS] ROLE
- DROP [IF EXISTS] USER
- DROP [IF EXISTS] ROLE
2022-04-08 09:08:08 +08:00
32bba15e34 [refactor][fix] remove useless import in Config.java (#8878) 2022-04-07 11:40:05 +08:00
64d18364db [improvement](restore) set table property 'dynamic_partition.enable' to false after restore (#8852)
when restore table with dynamic partition properties, 'dynamic_partition.enable' is set to the backup time value.
but Doris could not turn on dynamic partition automatically when restore.
So we cloud see table never do dynamic partition with dynamic_partition.enable is set to 'true'.
2022-04-07 11:34:01 +08:00
ce50c4d826 [feature](diagnose) support "ADMIN DIAGNOSE TABLET" stmt (#8839)
`ADMIN DIAGNOSE TABLET tablet_id`

This statement makes it easier to quickly diagnose the status of a tablet.
See "ADMIN-DIAGNOSE-TABLET.md" for details

```
mysql> admin diagnose tablet 10196;
+----------------------------------+------------------------------+------------+
| Item                             | Info                         | Suggestion |
+----------------------------------+------------------------------+------------+
| TabletExist                      | Yes                          |            |
| TabletId                         | 10196                        |            |
| Database                         | default_cluster:db1: 10192   |            |
| Table                            | tbl1: 10194                  |            |
| Partition                        | tbl1: 10193                  |            |
| MaterializedIndex                | tbl1: 10195                  |            |
| Replicas(ReplicaId -> BackendId) | {"10197":10002}              |            |
| ReplicasNum                      | OK                           |            |
| ReplicaBackendStatus             | Backend 10002 is not alive.  |            |
| ReplicaVersionStatus             | OK                           |            |
| ReplicaStatus                    | OK                           |            |
| ReplicaCompactionStatus          | OK                           |            |
+----------------------------------+------------------------------+------------+
```
2022-04-07 11:30:03 +08:00
e72ccfd80c [Refactor][httpv2]remove http v1 code (#8848)
http v2 has been actually tested in production, and it is completely replaceable to have http code. In order to simplify code maintenance, remove the previous http part of the code
2022-04-07 08:38:29 +08:00
98cab78320 [refactor](schema_hash) remove schema_hash since every tablet id in be is unique (#8574) 2022-04-07 08:37:45 +08:00
319f1f634a [fix](ut) fix fe run CreateTableAsSelectStmtTest ,UserPropertyTest, ProjectPlannerFunctionTest and AggregateTest failed (#8838) 2022-04-06 15:23:49 +08:00
33736e45fa [fix](table-function) Fixed unreasonable nullable conversion (#8818) 2022-04-03 11:02:35 +08:00
a8417e6c8b [fix](restore) fix restore issue when meta version is too low (#8816)
When restore snapshot from 0.13 to master, the restore job is pending for long time.
However, we get error "Could not set meta version to 93 since it is lower than minimum required version 100" in log.
We should cancel restore job once get that error.
2022-04-03 10:56:23 +08:00
0e3b15f2d7 [fix](colocate) Fix the error colocate plan when query is (rollup + instance >1) (#8779)
The Repeat Node will change the fragment data partition.

So the output partition of child fragment is different from the data partition of current fragment.
When judging whether colocate can be enabled,
the current data partition of fragment should be used directly instead of the child's output partition.

Before this PR fix, queries with '''rollup + concurrency greater than 1''' may have incorrect results.
For example: 
```
select t1.tc1,t1.tc2,sum(t1.tc3) as total from t1 join[shuffle] t1 t2 on t1.tc1=t2.tc1
group by rollup(tc1,tc2) order by t1.tc1,t1.tc2,total;
```

Fixed #8778
2022-04-03 10:19:39 +08:00
a75e4a1469 Window funnel (#8485)
Add new feature window funnel
2022-04-02 22:08:50 +08:00
4d516bece8 [feature-wip](array-type)Add element_at and subscript functions (#8597)
Describe the overview of changes.
1. add function element_at;
2. support element_subscript([]) to get element of array, col_array[N] <==> element_at(col_array, N);
3. return error message instead of BE crash while array function execute failed;

element_at(array, index) desc:
>   Returns element of array at given **(1-based)** index. 
  If **index < 0**, accesses elements from the last to the first. 
  Returns NULL if the index exceeds the length of the array or the array is NULL.

Usage example:
1. create table with ARRAY type column and insert some data:
```
+------+------+--------+
| k1   | k2   | k3     |
+------+------+--------+
|    1 |    2 | [1, 2] |
|    2 |    3 | NULL   |
|    4 | NULL | []     |
|    3 | NULL | NULL   |
+------+------+--------+
```
2. enable vectorized:
```
set enable_vectorized_engine=true;
```
3. element_subscript([]) usage example:
```
> select k1,k3,k3[1] from array_test;
+------+--------+----------------------------+
| k1   | k3     | %element_extract%(`k3`, 1) |
+------+--------+----------------------------+
|    3 | NULL   |                       NULL |
|    1 | [1, 2] |                          1 |
|    2 | NULL   |                       NULL |
|    4 | []     |                       NULL |
+------+--------+----------------------------+
```
4. element_at function usage example:
```
> select k1,k3 from array_test where element_at(k3, -1) = 2;
+------+--------+
| k1   | k3     |
+------+--------+
|    1 | [1, 2] |
+------+--------+
```
2022-04-02 12:03:56 +08:00
6c5bbc6e4c fix agg functions check failed from empty table (#8785)
fix agg functions check failed from empty table
2022-04-02 10:44:55 +08:00
9f80f6cf5e [Improvement](Planner)Enable hash join project (#8618) 2022-04-01 15:42:25 +08:00
6729d41c93 [improvement] add switch of quantile_state column (#8706)
Add switch for quantile_state column, default false.
2022-03-31 22:59:27 +08:00
d17ee7b476 [fix](sql-block-rule)Fix sql block rule bug (#8738)
1. Check properties' effectiveness of sql_block_rule, can't set limitations of sql_block_rule to be negative.
2. Optimize the judgment conditions when a query hits a sql_block_rule
3. Check if sql_block_rule has already exist when exec set property for "user" "sql_block_rule" = "xxx"
4. Add UT ad SqlBlockRuleMgrTest.java
2022-03-31 13:51:13 +08:00
b98da02611 [chore][fix](httpv2) Use mariadb-java-client for http query api (#8716)
In #8319, I remove mysql-connector-java dependency because of license incompatibility.
But we need a mysql compatible driver for http query api. So I choose mariadb-java-client,
which is under LGPL.
2022-03-30 09:59:45 +08:00
3f5bc5206d [Improvement] broker load with hdfs support wildcard (#8718)
broker load with hdfs support wildcard
2022-03-29 18:21:41 +08:00
66a3c574df [Vectorized][Bug] fix percentile_approx function to return always nullable (#8572) 2022-03-29 14:47:39 +08:00
f3659c87c1 [fix][chore](repository)(fe) check reponame when creating repository and modify build.sh (#8671)
1. We need to check repo name when creating repository
2. modify build.sh to not install spark-dpp when spark-dpp is not compiled
2022-03-29 11:32:52 +08:00
d82c138a60 [fix](user-property) Fix bug that can not set exec_mem_limit at user level (#8710) 2022-03-29 10:03:33 +08:00
7cfce63a13 [fix](mini-load) Remove mini load in LOADING and PENDING state (#8649)
1. Remove some unused code.
2. handle mini load with wrong state
    1. For some historical reasons, some mini load jobs in LOADING state have not been cleared.
        As a result, new load jobs cannot be committed.
    2. If a mini load job is created right before FE restart, the mini load job will be in PENDING state forever.
        But it should be removed finally.
2022-03-28 10:22:17 +08:00
HB
39717a85a2 [fix](load) Fix null column bug in load's mapping column setting (#8625) 2022-03-28 10:08:00 +08:00
f96bc62573 [feature](balance) Support balance between disks on a single BE (#8553)
Current situation of Doris is that the cluster is balanced, but the disks of a backend may be unbalanced.
for example, backend A have two disks: disk1 and disk2, disk1's usage is 98%, but disk2's usage is only 40%.
disk1 is unable to take more data, therefore only one disk of backend A can take new data,
the available write throughput of backend A is only half of its ability, and we can not resolve this through load or 
partition rebalance now.

So we introduce disk rebalancer, disk rebalancer is different from other rebalancer(load or partition)
which take care of cluster-wide data balancing. it takes care about backend-wide data balancing.

[For more details see #8550](https://github.com/apache/incubator-doris/issues/8550)
2022-03-28 10:03:21 +08:00
cfb57be731 [api-change] add soft limit of String type length (#8567)
1. add a config string_type_soft_limit to soft limit max length of string type
2. disable using String type in Key column, partition column and
   distribution column
3. remove String type alias BLOB for futrue use
2022-03-25 09:28:41 +08:00
5f606c9d57 [fix] Fix coredump of stddev function (#8543)
This is only a temporary fix its performance is not ideal. Finally,
we need to reconstruct the functions of `stddev` and delete the interface of `insert_to_null_default ()`.
2022-03-24 11:39:29 +08:00
a58e56f0b4 [fix](load) fix another bug that BE may crash when calling mark_as_failed (#8607)
Same as #8501
2022-03-24 09:13:54 +08:00
bea9a7ba4f [feature] Support pre-aggregation for quantile type (#8234)
Add a new column-type to speed up the approximation of quantiles.
1. The  new column-type is named `quantile_state` with fixed aggregation function `quantile_union`, which stores the intermediate results of pre-aggregated approximation calculations for quantiles.
2. support pre-aggregation of new column-type and quantile_state related functions.
2022-03-24 09:11:34 +08:00
72dfdb9a6c [fix] Fix Check_time return wrong value when exec show table status (#8578) 2022-03-23 10:34:23 +08:00
b89e4c7bba [feature-wip](java-udf) support java UDF with fixed-length input and output (#8516)
This feature is propsoed in [DSIP-1](https://cwiki.apache.org/confluence/display/DORIS/DSIP-001%3A+Java+UDF). 
This PR support fixed-length input and output Java UDF. Phase I in DIP-1 is done after this PR.

To support Java UDF effeciently, I use no data copy in JNI call and all compute operations are off-heap in Java.
To achieve that, I use a UdfExecutor instead. 

For users, a UDF class must have a public evaluate method.
2022-03-23 10:32:50 +08:00
71ce3c4a6e [feature-wip](array-type) Add codes and UT for array_contains and array_position functions (#8401) (#8589)
array_contains function Usage example:
1. create table with ARRAY column, and insert some data:
```
> select * from array_test;
+------+------+--------+
| k1   | k2   | k3     |
+------+------+--------+
|    1 |    2 | [1, 2] |
|    2 |    3 | NULL   |
|    4 | NULL | []     |
|    3 | NULL | NULL   |
+------+------+--------+
```
2. enable vectorized:
```
> set enable_vectorized_engine=true;
```
3. select with array_contains:
```
> select k1,array_contains(k3,1) from array_test;
+------+-------------------------+
| k1   | array_contains(`k3`, 1) |
+------+-------------------------+
|    3 |                    NULL |
|    1 |                       1 |
|    2 |                    NULL |
|    4 |                       0 |
+------+-------------------------+
```
4. also we can use array_contains in where condition
```
> select * from array_test where array_contains(k3,1);
+------+------+--------+
| k1   | k2   | k3     |
+------+------+--------+
|    1 |    2 | [1, 2] |
+------+------+--------+
```
5. array_position usage example
```
> select k1,k3,array_position(k3,2) from array_test;
+------+--------+-------------------------+
| k1   | k3     | array_position(`k3`, 2) |
+------+--------+-------------------------+
|    3 | NULL   |                    NULL |
|    1 | [1, 2] |                       2 |
|    2 | NULL   |                    NULL |
|    4 | []     |                       0 |
+------+--------+-------------------------+
```
2022-03-22 15:42:40 +08:00
b638c07533 [feature-wip](array-type) Support nested array insertion. (#8305) (#8586)
Please refer to #8304 .
2022-03-22 15:28:26 +08:00
e44038caf3 [feature-wip](array-type) Array data can be loaded in stream load. (#8368) (#8585)
Please refer to #8367 .
2022-03-22 15:25:40 +08:00
38ec3cbbdf [feature-wip](array-type) Support ArrayLiteral in SQL. (#8089) (#8582)
Please refer to #8074
2022-03-22 15:07:06 +08:00
cf0a9fd177 [feature-wip](array-type) Create table with nested array type. (#8003) (#8575)
```
create table array_type_table(k1 INT, k2 Array<Array<int>>) duplicate key (k1)
distributed by hash(k1) buckets 1 properties('replication_num' = '1');
```
2022-03-22 15:03:32 +08:00
106d7c2e41 [fix] Wrong conf be used for Filesytem in S3Storage (#8568)
wrong conf for Filesytem in S3Storage to disable cache.
it will lead to wrong behavior when use it to list objects in object store
2022-03-22 11:42:38 +08:00
9a0a1c693e [fix] fix NPE in thrift when forwarding stmt to master FE 2022-03-22 11:41:13 +08:00
Pxl
be3d203289 [feature][vectorized] support table function explode_numbers() (#8509) 2022-03-22 11:38:00 +08:00
f06780249a fix some fe ut failed (#8547) 2022-03-21 10:36:06 +08:00
eeae516e37 [Feature](Memory) Hook TCMalloc new/delete automatically counts to MemTracker (#8476)
Early Design Documentation: https://shimo.im/docs/DT6JXDRkdTvdyV3G

Implement a new way of memory statistics based on TCMalloc New/Delete Hook,
MemTracker and TLS, and it is expected that all memory new/delete/malloc/free
of the BE process can be counted.
2022-03-20 23:06:54 +08:00
8470455e0a [fix](tablet-report) Fix bug that tabletReport function of ReportHandler in fe may throw NullPointerException due to transaction check logic (#8481) 2022-03-18 09:31:51 +08:00
30d8089b2f [fix](partition_cache) Fix Partition Cache NullPointerException bug (#8454)
Filter the partitions in predicate but not in OlapTable.
2022-03-17 10:04:49 +08:00
2252ff81d7 [fix](dynamic-partition) fix bug that can not set dynamic_partition.replication_allocation property (#8471) 2022-03-15 11:45:18 +08:00
30eff9d6e9 [improvement] Update ShowExecutor.java (#8462)
we have some engines like mysql,olap,es,hive and so on , we should add more details for show engines
2022-03-15 11:44:36 +08:00
7d1d45d6dc [feature-wip](udf) support java udf in FE (#8437)
First step to support Java UDF in Doris. After this PR, we can create Java UDF in doris.

For example, we create Java UDF function by code below.
```
CREATE FUNCTION test_udf(int) RETURNS int 
PROPERTIES (
"file"="file:///root/hive-udf-1.0-SNAPSHOT.jar",
"symbol"="udf.Main", 
"type"="JAVA_UDF"
)
```
1. `file` indicate where user file is.
2. `symbol` for java udf means udf class in this jar.
3. `type` indicate this function is a java udf.
2022-03-15 11:42:39 +08:00
571f0b688d [improvment] show export support label like (#8202)
using `show export where label like 'xxx%'` to list more results.
2022-03-15 11:41:59 +08:00
a4b710cb2d [chore](dependency) fix build thirdparty errors (#8456)
1. the patch for aws-c-cal-0.4.5 does not need anymore
2. remove duplicate bit_length document
3. add some debug log for routine load
2022-03-13 22:11:24 +08:00
2c63fc1d6c [improvement](vectorized) Support BetweenPredicate enable fold const expr (#8450) 2022-03-13 09:36:24 +08:00