Commit Graph

1198 Commits

Author SHA1 Message Date
f431d8d94c [Enhance][Log] Make RPC error log more clear (#4702)
At present, when some rpc errors occur, the client cannot obtain the error information well.

And this CL change the RPC error returned to client like this:

```
ERROR 1064 (HY000): errCode = 2, detailMessage = there is no scanNode Backend. [10002: in black list(A error 
occurred: errorCode=2001 errorMessage:Channel inactive error!)]

ERROR 1064 (HY000): failed to send brpc batch, error=The server is overcrowded, error_text=[E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R1][E1011]The server is overcrowded @xx.xx.xx.xx:8060 [R2][E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R3][E1011]The server is overcrowded @xx.xx.xx.xx:8060, client: yy.yy.yy.yy
```
2020-10-13 10:08:43 +08:00
c934cf93e1 [UI Part 5] Enable HTTP Server 2 by FE config (#4684)
This is the last PR of proposal #4308

1. Add a new FE config `enable_http_server_v2` to enable new HTTP Server implementation. The default value is false.
2. Add a new FE config `http_api_extra_base_path` so that we can set base path for Frontend UI.
3. Refactor the new HTTP API response body. The return http status code is always 200, and using internal code in response body to indicate the certain error.
2020-10-13 10:07:50 +08:00
c69fe23c42 [Bug] Fix hard cardinality check which makes queries fail (#4678)
fix hard `Preconditions.checkState()` by setting `cardinality` to `-1` when computation is invalid.
2020-10-13 10:07:22 +08:00
3f0cb04376 [Feature] Support REPLACE TABLE operation. (#4669)
Implement REPLACE TABLE feature.
Details can be found in document `alter-table-replace-table.md`
2020-10-13 10:06:57 +08:00
d73d205de7 [ODBC/MySQL] Support Limit Clause Push Down For ODBC Table And MySQL Table(#4706) (#4707)
1. Support limit clause push down both odbc table and mysql table.
2. Code refactor of ODBC Scan Node, change `build_connect_string` and `query_string` from BE to FE to make it easily to modify
2020-10-11 21:11:04 +08:00
04f26e4b7f [SQL] Support Bucket Shuffle Join (#4677)
Support Bucket Shuffle Join
issue:#4394
2020-10-11 15:37:32 +08:00
28f4e922a7 [CREATE TABLE]Support new syntax CREATE TABLE LIKE to clone an existe… (#4705)
Support new synatx CREATE TABLE [IF NOT EXISTS] [db_name].table_name AS [db_name2].table_name2;
to create a new table from existed table with same table schema.
ISSUE: #4355
2020-10-10 21:16:53 +08:00
94a81e52c7 Fix typo in our code (#4696)
* fix typo in our code

* fix

* fix
2020-10-10 21:16:25 +08:00
6ab858bb10 [Bug] Fix duplicate columns in case when statement (#4693)
Fix #4692
The reason of this bug is case-when statement may produce implicit CastExpr<SlotRef>
as SelectListItem in analyze step.
And these CastExpr<SlotRef> in SelectList will not be re-anlyze after rewrite step,
which will result in the incorrect number of self-incrementing SlotDescriptor ID in resultExprs .

So we need to reset the analysis state of CastExpr<SlotRef> in rewrite step.
2020-10-10 21:16:00 +08:00
ed09ea9cf7 [Bug] Fix bug that tablet report always out of date (#4695)
This is because the report version is not set correctly in the tablet report request.

Also modify the download url of libevent Fix #4689 Fix #4691
2020-10-09 15:54:31 +08:00
725ee59824 Fix mysqlslap hang under high concurrent (#4680) 2020-10-07 12:30:18 +08:00
f0bb012dd2 [Bug] Fix Windows function lag()/lead() function throw AnalysisException. (#4666)
Add lag() and lead() char and varchar check in `isCastMatchAllowed`.
2020-10-04 10:24:36 +08:00
97dd634cdc Fix hidden cloumn may disappeared (#4686)
* xx

* xx

* xx
2020-09-30 10:01:49 +08:00
0475aa9b93 [Bug]Fix delete on clause may not work in routineLoad (#4683)
fix delete on may not work in some cases, this is describe in #4682
2020-09-30 09:56:19 +08:00
9f001efc16 [Bug]Fix information_schema.columns table column_comment does not show (#4676) 2020-09-28 13:40:33 +08:00
a202bf878e [Bug] Fix that the partitions of a dynamic-partitioned table has not been created at the time of load or insert (#4658)
when a dynamic partitioned table is created, it will take some time to create the partitions.
A exception needs to be thrown when users try to load data into this table,
otherwise the load will stuck in loading phase all the time.
2020-09-28 13:37:36 +08:00
4e3b576fd3 [NewFeature] Support ExternalCatalogResource to simplify external table manage operation. (#4559)
1. Add new Resource ExternalCatalogResource

```
create external resource "odbc"
properties
(
   "type" = "external_catalog", (required)
   "user" = "test",(required)
   "password" = "", (required)
   "host" = "192.168.0.1", (required)
   "port" = "8086", (required)
   "type" = "oracle" , (optinal,only odbc exteranl table use)
   "driver" = "Oracle 19 ODBC driver" (optional,only odbc exteranl table use)
)
```

2.After create ExternalCatalogResource, can create external table like:

```
CREATE TABLE `test_mysql` (
  `k1` tinyint(4) NOT NULL,
  `k2` smallint(6) NOT NULL,
  `k3` int(11) NOT NULL,
  `k4` bigint(20) NOT NULL,
  `k5` decimal(9,3) NOT NULL,
  `k6` char(5) NOT NULL,
  `k10` date DEFAULT NULL,
  `k11` datetime DEFAULT NULL,
  `k7` varchar(20) NOT NULL,
  `k8` double NOT NULL,
  `k9` float NOT NULL
) ENGINE=MYSQL
PROPERTIES (
"external_catalog_resource" = "odbc",
"database" = "test",
"table" = "test"
);
```
2020-09-25 10:20:33 +08:00
f87e294caa [License] Add license header (#4664) 2020-09-24 18:59:04 +08:00
f0576beb7a 【bug】Fix unit test bug (#4653) 2020-09-24 16:22:39 +08:00
e5f8c7cfc7 [BUG] Fix transaction not be cleared after BE down. (#4661)
After BE down, transactions related should be cleared.
The hostname is used to retrive the transactions.
But hostname is recorded as hostname + port, the port
should be removed.
2020-09-24 16:00:20 +08:00
724f70a405 Fix all queries failed when one BE network or disk has issue (#4651) 2020-09-23 15:25:08 +08:00
e7d9438fd9 [Bug] Fix analysis error when there are different types in case-when-then-else with group by clause (#4646) 2020-09-23 11:15:28 +08:00
1821d1baea [Compile] Add pluginRepository for java-cup-plugins (#4636)
the cup-maven-plugin of net.sourceforge.czt.dev is missing in maven central repo.
It has been moved to https://repository.cloudera.com/content/groups/public/

Co-authored-by: morningman <chenmingyu@baidu.com>
2020-09-21 12:38:28 +08:00
01894c82fe [Bug] Fix bug that wrong usage for HashBasedTable iterator may cause ConcurrentModificationException (#4631)
Fix bug that wrong usage for HashBasedTable iterator may cause ConcurrentModificationException
2020-09-20 20:58:08 +08:00
cf91c7062a [SQL][Bug]Fix join Predicate transitivity bug (#4629) 2020-09-20 20:57:50 +08:00
531cf11e6e [BUG] Some function in odbc external table different in Doris (#4625) (#4626)
Now Doris do not push down filter contains function in OdbcScanNode
Except Mysql external table.
2020-09-20 20:56:36 +08:00
3ae64884f8 [Syntax] Support CTE statement without use db (#4622) 2020-09-20 20:56:11 +08:00
ea45f4107e [Syntax]Change force order in Drop db, Drop table, Drop partition stmt (#4615) 2020-09-20 20:55:43 +08:00
ab059928c8 [Bug] catch IllegalArgumentException in DynamicPartitionScheduler (#4613) 2020-09-20 20:55:20 +08:00
0f811941fe [Bug] Fix the bug of #4608, support order by, where, limit in show alter table rollup (#4611)
support order by, where, limit in `show alter table rollup`
2020-09-20 20:54:43 +08:00
3e8b0618fd [Bug] Fix bug that forwarding show tablet stmt return error (#4605)
When forwarding show tablet stmt to Master FE. If the result contains null, FE will throw exception.
Fix it by using `\N` instead of "null" in result, to make thrift rpc happy.
2020-09-20 20:52:23 +08:00
065b979f35 [Bug] behavior of function str_to_date() and date_format() on BE and FE is inconsistent (#4612)
1. add date range check in `DateLiteral` for `FEFunctions`
2. `select str_to_date(202009,'%Y%m')` and `select str_to_date(str,'%Y%m') from tb where tb.str = '202009'` will return same output `2020-09-00`.
3. add support of zero-date to function `str_to_date()`,`date_format()` 
4. fix FE can calculate negative value bug, eg: `select str_to_date('-2020', '%Y')` will return `NULL` instead of date value.

current behavior is same as MySQL **without** sql_mode `NO_ZERO_IN_DATE` and `NO_ZERO_DATE`.

**current behavior**
```
mysql> select siteid,str_to_date(siteid,'%Y%m%d') from table2  order by siteid;
+------------+---------------------------------+
| siteid     | str_to_date(`siteid`, '%Y%m%d') |
+------------+---------------------------------+
|          1 | 2001-00-00                      |
|          2 | 2002-00-00                      |
|          2 | 2002-00-00                      |
|          3 | 2003-00-00                      |
|          4 | 2004-00-00                      |
|          5 | 2005-00-00                      |
|         20 | 2020-00-00                      |
|        202 | 0202-00-00                      |
|       2020 | 2020-00-00                      |
|      20209 | 2020-09-00                      |
|     202008 | 2020-08-00                      |
|     202009 | 2020-09-00                      |
|    2020009 | 2020-00-09                      |
|   20200009 | 2020-00-09                      |
|   20201309 | NULL                            |
| 2020090909 | 2020-09-09                      |
+------------+---------------------------------+

mysql> select str_to_date('2','%Y%m%d'),str_to_date('20','%Y%m%d'),str_to_date('202','%Y%m%d'),str_to_date('2020','%Y%m%d'),str_to_date('20209','%Y%m%d'),str_to_date('202009','%Y%m%d'),str_to_date('2020099','%Y%m%d'),str_to_date('20200909','%Y%m%d'),str_to_date('2020090909','%Y%m%d'),str_to_date('2020009','%Y%m%d'),str_to_date('20200009','%Y%m%d'),str_to_date('20201309','%Y%m%d');
+----------------------------+-----------------------------+------------------------------+-------------------------------+--------------------------------+---------------------------------+----------------------------------+-----------------------------------+-------------------------------------+----------------------------------+-----------------------------------+-----------------------------------+
| str_to_date('2', '%Y%m%d') | str_to_date('20', '%Y%m%d') | str_to_date('202', '%Y%m%d') | str_to_date('2020', '%Y%m%d') | str_to_date('20209', '%Y%m%d') | str_to_date('202009', '%Y%m%d') | str_to_date('2020099', '%Y%m%d') | str_to_date('20200909', '%Y%m%d') | str_to_date('2020090909', '%Y%m%d') | str_to_date('2020009', '%Y%m%d') | str_to_date('20200009', '%Y%m%d') | str_to_date('20201309', '%Y%m%d') |
+----------------------------+-----------------------------+------------------------------+-------------------------------+--------------------------------+---------------------------------+----------------------------------+-----------------------------------+-------------------------------------+----------------------------------+-----------------------------------+-----------------------------------+
| 2002-00-00                 | 2020-00-00                  | 0202-00-00                   | 2020-00-00                    | 2020-09-00                     | 2020-09-00                      | 2020-09-09                       | 2020-09-09                        | 2020-09-09                          | 2020-00-09                       | 2020-00-09                        | NULL                              |
+----------------------------+-----------------------------+------------------------------+-------------------------------+--------------------------------+---------------------------------+----------------------------------+-----------------------------------+-------------------------------------+----------------------------------+-----------------------------------+-----------------------------------+
```
2020-09-17 10:10:19 +08:00
a1f52ec2ab [SQL] Support where, limit, order clause in show resourcestmt. (#4502)
* [SQL] Support where, limit, order clause in show resourcestmt.

Grammar

    SHOW RESOURCES
    [
        WHERE
        [NAME [ = "your_resource_name" | LIKE "name_matcher"]]
        [RESOURCETYPE = ["SPARK"]]
    ]
    [ORDER BY ...]
    [LIMIT limit][OFFSET offset];

issue #4501
2020-09-16 17:57:48 +08:00
17a8b57018 [UI Part 3] New implemented HTTP RESTful API of Frontend (#4596)
Use spring mvc rest to replace the original netty http rest
Created a new package `org/apache/doris/httpv2`,
and the origin implementations under `org/apache/doris/http` remain unchanged.

This part of the code will not be used at present, so it will not affect existing functions.

API document can be found in #4584 

Proposal #4308
2020-09-16 15:02:59 +08:00
95111f9228 [Feature] Support alter table syntax for sequence column (#4582)
* enable sequence col

Co-authored-by: yangwenbo6 <yangwenbo3@jd.com>
2020-09-15 10:19:38 +08:00
4f4c22b459 [SQL][Planner] Support int datekey format implicit cast to date (#4591) 2020-09-13 19:15:53 +08:00
8c4f9d9a73 [Bug] Set BoolLiteral's selectivity (#4590)
`BoolLiteral`'s `selectivity`should be 1 when it is true else 0.
2020-09-13 19:15:26 +08:00
2f0d725a25 [Batch Delete] Add a session variable to show or hide hidden columns (#4579)
Sometimes we need to show hidden columns for debug.
So we need to add a session variable to show or hide hidden columns
2020-09-13 19:14:31 +08:00
b9829003fb [SQL][Planner] Fix the the parallesim of fragment which has 3 or more childern #4569 (#4570)
fix the the parallelism of fragment which has 3 or more childern
2020-09-13 19:14:00 +08:00
3e36505aea [Bug]Fix bug that tablet info with wrong capacity may cause fe oom (#4567)
* [Bug]Fix bug that tablet info with wrong capacity may cause fe oom

* remove unused code

* fix

* remove empty line

Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
2020-09-13 19:13:28 +08:00
c38593702f [Config] Add default configuration of max_fiter_ratio #4541# (#4553)
Co-authored-by: shqmh <shqmh@126.com>
2020-09-13 19:12:45 +08:00
2c24fe80fa [SparkDpp] Support complete types (#4524)
For[Spark Load]
1 support decimal andl largeint
2 add validate logic for char/varchar/decimal
3 check data load from hive with strict mode
4 support decimal/date/datetime aggregator
2020-09-13 11:57:33 +08:00
7afc66f8fe [Bug] catch plugin init error for Reinstall plugin (#4561) 2020-09-12 11:17:46 +08:00
31e451b8c7 [Feature] Doris can set default configuration of database data quota and replica quota #4540# (#4550)
When Doris create a database, the database of data quota and replica quota is initialized by configuration(Config.default_db_data_quota_bytes and Config.default_db_replica_quota_bytes).
In other words, I convert FeConstants.default_db_data_quota_bytes to Config.default_db_data_quota_bytes
Config.default_db_data_quota_bytes can be change in configuration file.
2020-09-12 11:17:27 +08:00
fe0f3ffbab [Bug] Fix bug that colocate join can not perceive parameter parallel_fragment_exec_instance_num (#4546) (#4547)
The instance of colocate join should be limit by the parameter parallel_fragment_exec_instance_num.
2020-09-12 11:16:31 +08:00
6e9d2074fb [UI Part 1] Update pom.xml of Frontend (#4583)
Add spring related dependencies
2020-09-11 17:48:42 +08:00
356bfcf065 fix (#4577)
Fix batch delete not work when using broker load, becuase of merger type not initialized
2020-09-11 09:45:22 +08:00
81784d6471 Revert "Add a session variable to show or hide hidden columns (#4510)" (#4576)
This reverts commit fe0260e54f8dfa37260423cffcf42096de19ed1f.
2020-09-10 15:18:36 +08:00
fe0260e54f Add a session variable to show or hide hidden columns (#4510)
* add session variable to show hidden columns
2020-09-10 13:07:43 +08:00
c4feb82229 [Bug] Fix null pointer when replay RollupJob (#4572)
When fe replay rollup job(v1) with deleted table, it will throw null pointer exception and exit.
This commit ignore this error and print a warning log to avoid fe exit.

Fixed #4571

Change-Id: I302b554a94d42aee645db6b224cd989e00cd3ca6
2020-09-10 11:55:14 +08:00