Commit Graph

5755 Commits

Author SHA1 Message Date
34d5ab8e07 [BUG] Ensure that the correct lead/lag function is selected (#4732)
* fix

* fix
2020-10-18 21:38:10 +08:00
d371d363a4 Fix bug CTE statement with nested select (#4731)
Co-authored-by: lixueyan07 <lixueyan07@meituan.com>
2020-10-17 13:36:14 +08:00
fa28542da8 Fix ut (#4752)
Change-Id: I7e3eb8bea00518ad71fd0b63d6fd9c440a536993
2020-10-16 15:35:15 +08:00
21e7428999 [Feature] Support cancel load jobs in batch (#4515)
Support statement like:
`cancel load where label like 'xxx';`
2020-10-15 22:49:39 +08:00
5a6e926c7b fix join error when the table has enbale batch delete (#4734) 2020-10-14 09:32:57 +08:00
453aeb7a19 [UT] Fixes PartitionCacheTest unit test randomly failed (#4729)
Fixes PartitionCacheTest unit test randomly failed. #4728
2020-10-14 09:31:15 +08:00
bf8f68f751 [Bug] Add regular column when materialized slot is empty in tuple (#4719)
If the materialized slot is empty in inline view, the scanner will throw exception 'no materialized slot!'.
When the outer query does not need any real columns of the inner query at all, such as the outer query only contains constant calculations,
all slots in the inner query will not be materialized.
For example:
```
select c1 from (select 'xx' c1, k1 from test) t1;
```

But this does not mean that the inner query (select 'xx' c1, k1 from test) does not need to be executed.
Because the number of rows in the outer result needs to be determined by the number of rows in the inner query.
In this case, the inner query scan node needs to add a column to ensure the correct result.
At the same time it can avoid the scan node reporting errors.
2020-10-13 10:12:03 +08:00
f431d8d94c [Enhance][Log] Make RPC error log more clear (#4702)
At present, when some rpc errors occur, the client cannot obtain the error information well.

And this CL change the RPC error returned to client like this:

```
ERROR 1064 (HY000): errCode = 2, detailMessage = there is no scanNode Backend. [10002: in black list(A error 
occurred: errorCode=2001 errorMessage:Channel inactive error!)]

ERROR 1064 (HY000): failed to send brpc batch, error=The server is overcrowded, error_text=[E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R1][E1011]The server is overcrowded @xx.xx.xx.xx:8060 [R2][E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R3][E1011]The server is overcrowded @xx.xx.xx.xx:8060, client: yy.yy.yy.yy
```
2020-10-13 10:08:43 +08:00
c934cf93e1 [UI Part 5] Enable HTTP Server 2 by FE config (#4684)
This is the last PR of proposal #4308

1. Add a new FE config `enable_http_server_v2` to enable new HTTP Server implementation. The default value is false.
2. Add a new FE config `http_api_extra_base_path` so that we can set base path for Frontend UI.
3. Refactor the new HTTP API response body. The return http status code is always 200, and using internal code in response body to indicate the certain error.
2020-10-13 10:07:50 +08:00
c69fe23c42 [Bug] Fix hard cardinality check which makes queries fail (#4678)
fix hard `Preconditions.checkState()` by setting `cardinality` to `-1` when computation is invalid.
2020-10-13 10:07:22 +08:00
3f0cb04376 [Feature] Support REPLACE TABLE operation. (#4669)
Implement REPLACE TABLE feature.
Details can be found in document `alter-table-replace-table.md`
2020-10-13 10:06:57 +08:00
d73d205de7 [ODBC/MySQL] Support Limit Clause Push Down For ODBC Table And MySQL Table(#4706) (#4707)
1. Support limit clause push down both odbc table and mysql table.
2. Code refactor of ODBC Scan Node, change `build_connect_string` and `query_string` from BE to FE to make it easily to modify
2020-10-11 21:11:04 +08:00
04f26e4b7f [SQL] Support Bucket Shuffle Join (#4677)
Support Bucket Shuffle Join
issue:#4394
2020-10-11 15:37:32 +08:00
28f4e922a7 [CREATE TABLE]Support new syntax CREATE TABLE LIKE to clone an existe… (#4705)
Support new synatx CREATE TABLE [IF NOT EXISTS] [db_name].table_name AS [db_name2].table_name2;
to create a new table from existed table with same table schema.
ISSUE: #4355
2020-10-10 21:16:53 +08:00
94a81e52c7 Fix typo in our code (#4696)
* fix typo in our code

* fix

* fix
2020-10-10 21:16:25 +08:00
6ab858bb10 [Bug] Fix duplicate columns in case when statement (#4693)
Fix #4692
The reason of this bug is case-when statement may produce implicit CastExpr<SlotRef>
as SelectListItem in analyze step.
And these CastExpr<SlotRef> in SelectList will not be re-anlyze after rewrite step,
which will result in the incorrect number of self-incrementing SlotDescriptor ID in resultExprs .

So we need to reset the analysis state of CastExpr<SlotRef> in rewrite step.
2020-10-10 21:16:00 +08:00
ed09ea9cf7 [Bug] Fix bug that tablet report always out of date (#4695)
This is because the report version is not set correctly in the tablet report request.

Also modify the download url of libevent Fix #4689 Fix #4691
2020-10-09 15:54:31 +08:00
725ee59824 Fix mysqlslap hang under high concurrent (#4680) 2020-10-07 12:30:18 +08:00
f0bb012dd2 [Bug] Fix Windows function lag()/lead() function throw AnalysisException. (#4666)
Add lag() and lead() char and varchar check in `isCastMatchAllowed`.
2020-10-04 10:24:36 +08:00
97dd634cdc Fix hidden cloumn may disappeared (#4686)
* xx

* xx

* xx
2020-09-30 10:01:49 +08:00
0475aa9b93 [Bug]Fix delete on clause may not work in routineLoad (#4683)
fix delete on may not work in some cases, this is describe in #4682
2020-09-30 09:56:19 +08:00
9f001efc16 [Bug]Fix information_schema.columns table column_comment does not show (#4676) 2020-09-28 13:40:33 +08:00
a202bf878e [Bug] Fix that the partitions of a dynamic-partitioned table has not been created at the time of load or insert (#4658)
when a dynamic partitioned table is created, it will take some time to create the partitions.
A exception needs to be thrown when users try to load data into this table,
otherwise the load will stuck in loading phase all the time.
2020-09-28 13:37:36 +08:00
4e3b576fd3 [NewFeature] Support ExternalCatalogResource to simplify external table manage operation. (#4559)
1. Add new Resource ExternalCatalogResource

```
create external resource "odbc"
properties
(
   "type" = "external_catalog", (required)
   "user" = "test",(required)
   "password" = "", (required)
   "host" = "192.168.0.1", (required)
   "port" = "8086", (required)
   "type" = "oracle" , (optinal,only odbc exteranl table use)
   "driver" = "Oracle 19 ODBC driver" (optional,only odbc exteranl table use)
)
```

2.After create ExternalCatalogResource, can create external table like:

```
CREATE TABLE `test_mysql` (
  `k1` tinyint(4) NOT NULL,
  `k2` smallint(6) NOT NULL,
  `k3` int(11) NOT NULL,
  `k4` bigint(20) NOT NULL,
  `k5` decimal(9,3) NOT NULL,
  `k6` char(5) NOT NULL,
  `k10` date DEFAULT NULL,
  `k11` datetime DEFAULT NULL,
  `k7` varchar(20) NOT NULL,
  `k8` double NOT NULL,
  `k9` float NOT NULL
) ENGINE=MYSQL
PROPERTIES (
"external_catalog_resource" = "odbc",
"database" = "test",
"table" = "test"
);
```
2020-09-25 10:20:33 +08:00
f87e294caa [License] Add license header (#4664) 2020-09-24 18:59:04 +08:00
f0576beb7a 【bug】Fix unit test bug (#4653) 2020-09-24 16:22:39 +08:00
e5f8c7cfc7 [BUG] Fix transaction not be cleared after BE down. (#4661)
After BE down, transactions related should be cleared.
The hostname is used to retrive the transactions.
But hostname is recorded as hostname + port, the port
should be removed.
2020-09-24 16:00:20 +08:00
724f70a405 Fix all queries failed when one BE network or disk has issue (#4651) 2020-09-23 15:25:08 +08:00
e7d9438fd9 [Bug] Fix analysis error when there are different types in case-when-then-else with group by clause (#4646) 2020-09-23 11:15:28 +08:00
1821d1baea [Compile] Add pluginRepository for java-cup-plugins (#4636)
the cup-maven-plugin of net.sourceforge.czt.dev is missing in maven central repo.
It has been moved to https://repository.cloudera.com/content/groups/public/

Co-authored-by: morningman <chenmingyu@baidu.com>
2020-09-21 12:38:28 +08:00
01894c82fe [Bug] Fix bug that wrong usage for HashBasedTable iterator may cause ConcurrentModificationException (#4631)
Fix bug that wrong usage for HashBasedTable iterator may cause ConcurrentModificationException
2020-09-20 20:58:08 +08:00
cf91c7062a [SQL][Bug]Fix join Predicate transitivity bug (#4629) 2020-09-20 20:57:50 +08:00
531cf11e6e [BUG] Some function in odbc external table different in Doris (#4625) (#4626)
Now Doris do not push down filter contains function in OdbcScanNode
Except Mysql external table.
2020-09-20 20:56:36 +08:00
3ae64884f8 [Syntax] Support CTE statement without use db (#4622) 2020-09-20 20:56:11 +08:00
ea45f4107e [Syntax]Change force order in Drop db, Drop table, Drop partition stmt (#4615) 2020-09-20 20:55:43 +08:00
ab059928c8 [Bug] catch IllegalArgumentException in DynamicPartitionScheduler (#4613) 2020-09-20 20:55:20 +08:00
0f811941fe [Bug] Fix the bug of #4608, support order by, where, limit in show alter table rollup (#4611)
support order by, where, limit in `show alter table rollup`
2020-09-20 20:54:43 +08:00
3e8b0618fd [Bug] Fix bug that forwarding show tablet stmt return error (#4605)
When forwarding show tablet stmt to Master FE. If the result contains null, FE will throw exception.
Fix it by using `\N` instead of "null" in result, to make thrift rpc happy.
2020-09-20 20:52:23 +08:00
065b979f35 [Bug] behavior of function str_to_date() and date_format() on BE and FE is inconsistent (#4612)
1. add date range check in `DateLiteral` for `FEFunctions`
2. `select str_to_date(202009,'%Y%m')` and `select str_to_date(str,'%Y%m') from tb where tb.str = '202009'` will return same output `2020-09-00`.
3. add support of zero-date to function `str_to_date()`,`date_format()` 
4. fix FE can calculate negative value bug, eg: `select str_to_date('-2020', '%Y')` will return `NULL` instead of date value.

current behavior is same as MySQL **without** sql_mode `NO_ZERO_IN_DATE` and `NO_ZERO_DATE`.

**current behavior**
```
mysql> select siteid,str_to_date(siteid,'%Y%m%d') from table2  order by siteid;
+------------+---------------------------------+
| siteid     | str_to_date(`siteid`, '%Y%m%d') |
+------------+---------------------------------+
|          1 | 2001-00-00                      |
|          2 | 2002-00-00                      |
|          2 | 2002-00-00                      |
|          3 | 2003-00-00                      |
|          4 | 2004-00-00                      |
|          5 | 2005-00-00                      |
|         20 | 2020-00-00                      |
|        202 | 0202-00-00                      |
|       2020 | 2020-00-00                      |
|      20209 | 2020-09-00                      |
|     202008 | 2020-08-00                      |
|     202009 | 2020-09-00                      |
|    2020009 | 2020-00-09                      |
|   20200009 | 2020-00-09                      |
|   20201309 | NULL                            |
| 2020090909 | 2020-09-09                      |
+------------+---------------------------------+

mysql> select str_to_date('2','%Y%m%d'),str_to_date('20','%Y%m%d'),str_to_date('202','%Y%m%d'),str_to_date('2020','%Y%m%d'),str_to_date('20209','%Y%m%d'),str_to_date('202009','%Y%m%d'),str_to_date('2020099','%Y%m%d'),str_to_date('20200909','%Y%m%d'),str_to_date('2020090909','%Y%m%d'),str_to_date('2020009','%Y%m%d'),str_to_date('20200009','%Y%m%d'),str_to_date('20201309','%Y%m%d');
+----------------------------+-----------------------------+------------------------------+-------------------------------+--------------------------------+---------------------------------+----------------------------------+-----------------------------------+-------------------------------------+----------------------------------+-----------------------------------+-----------------------------------+
| str_to_date('2', '%Y%m%d') | str_to_date('20', '%Y%m%d') | str_to_date('202', '%Y%m%d') | str_to_date('2020', '%Y%m%d') | str_to_date('20209', '%Y%m%d') | str_to_date('202009', '%Y%m%d') | str_to_date('2020099', '%Y%m%d') | str_to_date('20200909', '%Y%m%d') | str_to_date('2020090909', '%Y%m%d') | str_to_date('2020009', '%Y%m%d') | str_to_date('20200009', '%Y%m%d') | str_to_date('20201309', '%Y%m%d') |
+----------------------------+-----------------------------+------------------------------+-------------------------------+--------------------------------+---------------------------------+----------------------------------+-----------------------------------+-------------------------------------+----------------------------------+-----------------------------------+-----------------------------------+
| 2002-00-00                 | 2020-00-00                  | 0202-00-00                   | 2020-00-00                    | 2020-09-00                     | 2020-09-00                      | 2020-09-09                       | 2020-09-09                        | 2020-09-09                          | 2020-00-09                       | 2020-00-09                        | NULL                              |
+----------------------------+-----------------------------+------------------------------+-------------------------------+--------------------------------+---------------------------------+----------------------------------+-----------------------------------+-------------------------------------+----------------------------------+-----------------------------------+-----------------------------------+
```
2020-09-17 10:10:19 +08:00
a1f52ec2ab [SQL] Support where, limit, order clause in show resourcestmt. (#4502)
* [SQL] Support where, limit, order clause in show resourcestmt.

Grammar

    SHOW RESOURCES
    [
        WHERE
        [NAME [ = "your_resource_name" | LIKE "name_matcher"]]
        [RESOURCETYPE = ["SPARK"]]
    ]
    [ORDER BY ...]
    [LIMIT limit][OFFSET offset];

issue #4501
2020-09-16 17:57:48 +08:00
17a8b57018 [UI Part 3] New implemented HTTP RESTful API of Frontend (#4596)
Use spring mvc rest to replace the original netty http rest
Created a new package `org/apache/doris/httpv2`,
and the origin implementations under `org/apache/doris/http` remain unchanged.

This part of the code will not be used at present, so it will not affect existing functions.

API document can be found in #4584 

Proposal #4308
2020-09-16 15:02:59 +08:00
95111f9228 [Feature] Support alter table syntax for sequence column (#4582)
* enable sequence col

Co-authored-by: yangwenbo6 <yangwenbo3@jd.com>
2020-09-15 10:19:38 +08:00
4f4c22b459 [SQL][Planner] Support int datekey format implicit cast to date (#4591) 2020-09-13 19:15:53 +08:00
8c4f9d9a73 [Bug] Set BoolLiteral's selectivity (#4590)
`BoolLiteral`'s `selectivity`should be 1 when it is true else 0.
2020-09-13 19:15:26 +08:00
2f0d725a25 [Batch Delete] Add a session variable to show or hide hidden columns (#4579)
Sometimes we need to show hidden columns for debug.
So we need to add a session variable to show or hide hidden columns
2020-09-13 19:14:31 +08:00
b9829003fb [SQL][Planner] Fix the the parallesim of fragment which has 3 or more childern #4569 (#4570)
fix the the parallelism of fragment which has 3 or more childern
2020-09-13 19:14:00 +08:00
3e36505aea [Bug]Fix bug that tablet info with wrong capacity may cause fe oom (#4567)
* [Bug]Fix bug that tablet info with wrong capacity may cause fe oom

* remove unused code

* fix

* remove empty line

Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
2020-09-13 19:13:28 +08:00
c38593702f [Config] Add default configuration of max_fiter_ratio #4541# (#4553)
Co-authored-by: shqmh <shqmh@126.com>
2020-09-13 19:12:45 +08:00
2c24fe80fa [SparkDpp] Support complete types (#4524)
For[Spark Load]
1 support decimal andl largeint
2 add validate logic for char/varchar/decimal
3 check data load from hive with strict mode
4 support decimal/date/datetime aggregator
2020-09-13 11:57:33 +08:00
7afc66f8fe [Bug] catch plugin init error for Reinstall plugin (#4561) 2020-09-12 11:17:46 +08:00