Commit Graph

2409 Commits

Author SHA1 Message Date
c00a5cb543 [Bug] Fix the core problem of function split_part and add the UT of core case (#4721)
issue:#4720
2020-10-13 10:09:39 +08:00
f431d8d94c [Enhance][Log] Make RPC error log more clear (#4702)
At present, when some rpc errors occur, the client cannot obtain the error information well.

And this CL change the RPC error returned to client like this:

```
ERROR 1064 (HY000): errCode = 2, detailMessage = there is no scanNode Backend. [10002: in black list(A error 
occurred: errorCode=2001 errorMessage:Channel inactive error!)]

ERROR 1064 (HY000): failed to send brpc batch, error=The server is overcrowded, error_text=[E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R1][E1011]The server is overcrowded @xx.xx.xx.xx:8060 [R2][E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R3][E1011]The server is overcrowded @xx.xx.xx.xx:8060, client: yy.yy.yy.yy
```
2020-10-13 10:08:43 +08:00
83f6f46c34 [Config] Limit the version number of tablet (#4687)
Add a BE config `max_tablet_version_num` to limit the version number of a single tablet.
To avoid too many versions
2020-10-13 10:08:16 +08:00
c934cf93e1 [UI Part 5] Enable HTTP Server 2 by FE config (#4684)
This is the last PR of proposal #4308

1. Add a new FE config `enable_http_server_v2` to enable new HTTP Server implementation. The default value is false.
2. Add a new FE config `http_api_extra_base_path` so that we can set base path for Frontend UI.
3. Refactor the new HTTP API response body. The return http status code is always 200, and using internal code in response body to indicate the certain error.
2020-10-13 10:07:50 +08:00
c69fe23c42 [Bug] Fix hard cardinality check which makes queries fail (#4678)
fix hard `Preconditions.checkState()` by setting `cardinality` to `-1` when computation is invalid.
2020-10-13 10:07:22 +08:00
3f0cb04376 [Feature] Support REPLACE TABLE operation. (#4669)
Implement REPLACE TABLE feature.
Details can be found in document `alter-table-replace-table.md`
2020-10-13 10:06:57 +08:00
75e0ba32a1 Fixes some be typo (#4714) 2020-10-13 09:37:15 +08:00
d73d205de7 [ODBC/MySQL] Support Limit Clause Push Down For ODBC Table And MySQL Table(#4706) (#4707)
1. Support limit clause push down both odbc table and mysql table.
2. Code refactor of ODBC Scan Node, change `build_connect_string` and `query_string` from BE to FE to make it easily to modify
2020-10-11 21:11:04 +08:00
fd5e3011da [Docs] Added some missing documents (#4710)
1. Add ODBC resource in resource manage doc and add english of resource manage doc
2. Add miss part in sidebar
2020-10-11 15:40:53 +08:00
98e71a8b9f [Bug][Function] Fix rand() function return same value (#4709)
fix rand function return same value when no parameter
2020-10-11 15:40:38 +08:00
04f26e4b7f [SQL] Support Bucket Shuffle Join (#4677)
Support Bucket Shuffle Join
issue:#4394
2020-10-11 15:37:32 +08:00
eba595583e [Optimize] Optimize the execution model of compaction to limit memory consumption (#4670)
Currently, there are M threads to do base compaction and N threads to do cumulative compaction for each disk. 
Too many compaction tasks may run out of memory, so the max concurrency of running compaction tasks
is limited by semaphore.
If the running threads cost too much memory, we can't defense it. In addition, reducing concurrency to avoid OOM
will lead to some compaction tasks can't be executed in time and we may encounter more heavy compaction. 
Therefore, concurrency limitation is not enough.

The strategy proposed in #3624  may be effective to solve the OOM. 

A CompactionPermitLimiter is used for compaction limitation, and use single-producer/multi-consumer model.
Producer will try to generate compaction tasks and acquire `permits` for each task. 
The compaction task which can hold `permits` will be executed in thread pool and each finished task will
release its `permits`.

`permits` should be applied for before a compaction task can execute. When the sum of `permits`
held by executing compaction tasks reaches a threshold, subsequent compaction task will be no longer allowed,
until some `permits` are released. Tablet compaction score is used as `permits` of compaction task here.

To some extent, memory consumption can be limited by setting appropriate `permits` threshold.
2020-10-11 11:39:25 +08:00
28f4e922a7 [CREATE TABLE]Support new syntax CREATE TABLE LIKE to clone an existe… (#4705)
Support new synatx CREATE TABLE [IF NOT EXISTS] [db_name].table_name AS [db_name2].table_name2;
to create a new table from existed table with same table schema.
ISSUE: #4355
2020-10-10 21:16:53 +08:00
94a81e52c7 Fix typo in our code (#4696)
* fix typo in our code

* fix

* fix
2020-10-10 21:16:25 +08:00
6ab858bb10 [Bug] Fix duplicate columns in case when statement (#4693)
Fix #4692
The reason of this bug is case-when statement may produce implicit CastExpr<SlotRef>
as SelectListItem in analyze step.
And these CastExpr<SlotRef> in SelectList will not be re-anlyze after rewrite step,
which will result in the incorrect number of self-incrementing SlotDescriptor ID in resultExprs .

So we need to reset the analysis state of CastExpr<SlotRef> in rewrite step.
2020-10-10 21:16:00 +08:00
f3cdf167d1 [Feature] Add time_round builtin functions (#4640)
#4619 
Add time_round functions that provides `time_floor` & `time_ceil` at each time unit.

Fix two related bugs.
- #4618 
- Fix `struct TimeInterval` to use `int64_t` instead of `int32_t`, in case when the second diff overflow
2020-10-09 16:05:51 +08:00
ed09ea9cf7 [Bug] Fix bug that tablet report always out of date (#4695)
This is because the report version is not set correctly in the tablet report request.

Also modify the download url of libevent Fix #4689 Fix #4691
2020-10-09 15:54:31 +08:00
725ee59824 Fix mysqlslap hang under high concurrent (#4680) 2020-10-07 12:30:18 +08:00
f0bb012dd2 [Bug] Fix Windows function lag()/lead() function throw AnalysisException. (#4666)
Add lag() and lead() char and varchar check in `isCastMatchAllowed`.
2020-10-04 10:24:36 +08:00
97dd634cdc Fix hidden cloumn may disappeared (#4686)
* xx

* xx

* xx
2020-09-30 10:01:49 +08:00
0475aa9b93 [Bug]Fix delete on clause may not work in routineLoad (#4683)
fix delete on may not work in some cases, this is describe in #4682
2020-09-30 09:56:19 +08:00
364396355a [Dockerfile] Update Dockerfile (#4474)
Update Dockerfile, add nodejs, and change some URL
2020-09-29 10:44:40 +08:00
9f001efc16 [Bug]Fix information_schema.columns table column_comment does not show (#4676) 2020-09-28 13:40:33 +08:00
b1853caeed [UDF] Improve performance of function money_format (#4672)
Use static local variable instead of create it every calls.
Time cost of the new added unit benchmark test could reduce
from about 60 seconds to 10 seconds.
2020-09-28 13:39:41 +08:00
0eb54007be [Bug]External engines(e.g. ES) don't have segments, ignore those tables (#4671)
It will crash when there are External engine tables in doris after executing "python show_segment_status.py"
The main reason is external engine tables don't have any index and partitions. So it should be ignored.
2020-09-28 13:39:05 +08:00
05512d7b47 [Doc] Fix typo 2020-09-28 13:38:39 +08:00
a202bf878e [Bug] Fix that the partitions of a dynamic-partitioned table has not been created at the time of load or insert (#4658)
when a dynamic partitioned table is created, it will take some time to create the partitions.
A exception needs to be thrown when users try to load data into this table,
otherwise the load will stuck in loading phase all the time.
2020-09-28 13:37:36 +08:00
50b576a626 [Docs] Change incorrect bos endpoint in broker docs (#4642) 2020-09-28 13:37:05 +08:00
e611ab96e7 [Docs] Supply BE config docs of setting and examples (#4641) 2020-09-28 13:36:12 +08:00
5199a17a4b [cache][be]Fix the bug of cross-border access cache (#4639)
* When the different partition of the table is updated frequently, the partition key list of the cache is discontinuous,
and the partition key in the request cannot hit the key list in the cache, resulting in the access overrun,the BE will crash.

* Add some unit test case,add test cases that fail to hit the boundary value of cache
2020-09-28 13:35:52 +08:00
1dacadb015 [BUG] Fix DATA_TYPE in information_schema.columns is not compatible to mysql meta (#4648)
Describe the bug

DATA_TYPE in information_schema.columns is not compatible to mysql meta

To Reproduce
Steps to reproduce the behavior:
select * from information_schema.columns

Expected behavior
the result of data_type is (int, decimal, char, varchar, ...),but doris data_type is (int(11), varchar(20), ...)
Excess number will affect some BI systems or upper system can't get right type
2020-09-25 13:38:09 +08:00
e072002edc [Doc] Update data-model-rollup.md (#4659)
fix typo
2020-09-25 10:26:05 +08:00
4e3b576fd3 [NewFeature] Support ExternalCatalogResource to simplify external table manage operation. (#4559)
1. Add new Resource ExternalCatalogResource

```
create external resource "odbc"
properties
(
   "type" = "external_catalog", (required)
   "user" = "test",(required)
   "password" = "", (required)
   "host" = "192.168.0.1", (required)
   "port" = "8086", (required)
   "type" = "oracle" , (optinal,only odbc exteranl table use)
   "driver" = "Oracle 19 ODBC driver" (optional,only odbc exteranl table use)
)
```

2.After create ExternalCatalogResource, can create external table like:

```
CREATE TABLE `test_mysql` (
  `k1` tinyint(4) NOT NULL,
  `k2` smallint(6) NOT NULL,
  `k3` int(11) NOT NULL,
  `k4` bigint(20) NOT NULL,
  `k5` decimal(9,3) NOT NULL,
  `k6` char(5) NOT NULL,
  `k10` date DEFAULT NULL,
  `k11` datetime DEFAULT NULL,
  `k7` varchar(20) NOT NULL,
  `k8` double NOT NULL,
  `k9` float NOT NULL
) ENGINE=MYSQL
PROPERTIES (
"external_catalog_resource" = "odbc",
"database" = "test",
"table" = "test"
);
```
2020-09-25 10:20:33 +08:00
a61d0de173 [ODBC SCAN NODE] 4/4 Add ODBC_SCAN_NODE and Odbc_Scanner in BE and add ODBC_SCAN_NODE docs (#4438) 2020-09-25 10:19:50 +08:00
04e2ce2acb [Compile] Fix md5 of thirdparty(#4663)
Fix compile error when compile Doris from source and not using docker
2020-09-24 19:00:58 +08:00
f87e294caa [License] Add license header (#4664) 2020-09-24 18:59:04 +08:00
f0576beb7a 【bug】Fix unit test bug (#4653) 2020-09-24 16:22:39 +08:00
e5f8c7cfc7 [BUG] Fix transaction not be cleared after BE down. (#4661)
After BE down, transactions related should be cleared.
The hostname is used to retrive the transactions.
But hostname is recorded as hostname + port, the port
should be removed.
2020-09-24 16:00:20 +08:00
724f70a405 Fix all queries failed when one BE network or disk has issue (#4651) 2020-09-23 15:25:08 +08:00
e7d9438fd9 [Bug] Fix analysis error when there are different types in case-when-then-else with group by clause (#4646) 2020-09-23 11:15:28 +08:00
fd37c4f352 [Document] Fix some typo in alter table document
Fix some typos in document that confusing users.
2020-09-22 16:23:23 +08:00
1821d1baea [Compile] Add pluginRepository for java-cup-plugins (#4636)
the cup-maven-plugin of net.sourceforge.czt.dev is missing in maven central repo.
It has been moved to https://repository.cloudera.com/content/groups/public/

Co-authored-by: morningman <chenmingyu@baidu.com>
2020-09-21 12:38:28 +08:00
01894c82fe [Bug] Fix bug that wrong usage for HashBasedTable iterator may cause ConcurrentModificationException (#4631)
Fix bug that wrong usage for HashBasedTable iterator may cause ConcurrentModificationException
2020-09-20 20:58:08 +08:00
cf91c7062a [SQL][Bug]Fix join Predicate transitivity bug (#4629) 2020-09-20 20:57:50 +08:00
2a637f848d [Refactor] Remove meaningless return value of RowBlock::init (#4627)
Simplify some code, mainly remove meaningless return value of RowBlock::init.
2020-09-20 20:57:00 +08:00
531cf11e6e [BUG] Some function in odbc external table different in Doris (#4625) (#4626)
Now Doris do not push down filter contains function in OdbcScanNode
Except Mysql external table.
2020-09-20 20:56:36 +08:00
3ae64884f8 [Syntax] Support CTE statement without use db (#4622) 2020-09-20 20:56:11 +08:00
ea45f4107e [Syntax]Change force order in Drop db, Drop table, Drop partition stmt (#4615) 2020-09-20 20:55:43 +08:00
ab059928c8 [Bug] catch IllegalArgumentException in DynamicPartitionScheduler (#4613) 2020-09-20 20:55:20 +08:00
0f811941fe [Bug] Fix the bug of #4608, support order by, where, limit in show alter table rollup (#4611)
support order by, where, limit in `show alter table rollup`
2020-09-20 20:54:43 +08:00