Commit Graph

2446 Commits

Author SHA1 Message Date
c8d496830b [Compile] Remove unused import (#4794) 2020-10-26 22:54:27 +08:00
a61eea3b2e Make some debug log settings configurable and change some log level from info to debug to avoid performance bottlenecks (#4766) 2020-10-26 16:10:19 +08:00
f89b660c94 [Feature] Support Alter Table Clause For External Table (#4699)
* Support Alter Table Clause For External Table
5 alter operation be supported:

* RENAME
* ADD COLUMN
* DROP COLUMN
* MODIFY COLUMN
* REORDER COLUMN
2020-10-25 17:17:56 +08:00
2fa3ffda7b change code style (#4776) 2020-10-24 17:36:06 +08:00
0213e93ea1 [Feature][Config] Support persistence of configuration items modified at runtime (#4704)
Support persistence of configuration items modified at runtime via HTTP API.

```
FE:
GET /api/_set_config?key=value&persist=true

BE
POST /api/update_config?key=value&persist=true
```

The modified config will be saved in `fe_custom.conf` or `be_custom.conf`.
And when process starts, it will load `fe.conf/be.conf` first, then `fe_custom.conf/be_custom.conf`.
2020-10-22 21:39:02 +08:00
16f5d223e8 [Compile] Update Repository for java-cup and cup-maven-plugin (#4769) 2020-10-22 21:38:19 +08:00
6cbefd5621 [LRUCache] Expose LRU Cache status to metrics (#4688)
Expose LRU Cache status to metrics would be helpful to diagnose
problems like high usage, low hit rate.
2020-10-22 21:37:02 +08:00
fae5c624ea Fix delete_sign predicate assigned to join node (#4770) 2020-10-22 14:29:16 +08:00
0c37b42bc7 [Bug] Because of thread-safety problem , replace connectionMap with concurrentMap from hashMap. (#4774)
Thread [Connect-Scheduler-Check-Timer] fail to iterate the connectionMap , because other threads will modify the connectionMap at the same time.
Fe's connection which is timed out can't be released because of thread [Connect-Scheduler-Check-Timer] not working.
2020-10-22 11:15:21 +08:00
7b2762b1b1 [Doris On ES][Bug-Fix] Can not pushdown limit when some predicate can not processed by ES (#4768)
Can not pushdown limit when some predicate not processed by ES, fixed: #4761
2020-10-21 12:10:55 +08:00
77835dd9c4 [Bug][Compaction] Fix bug that compaction may be blocked (#4750)
the logic of compaction producer thread may failed to produce compaction task due to
invalid order of modifying task map.
2020-10-21 10:12:37 +08:00
349cc9ef17 [Bug] Do not push down limit operation when ODBC table do not push all conjunct as filter. (#4764) 2020-10-21 10:12:12 +08:00
588e5bee47 [Bug] Fix bug of cumulative compaction and deletion of stale version (#4593)
When selecting candidate rowsets to do the cumulative compaction,
some rowsets may not be selected because the protection time has not expired.
Therefore, we need to find the current longest continuous version path in the candidate rowsets.
2020-10-21 10:03:55 +08:00
a95ce69c0d [Doc] Bug fix that help commend not work (#4760)
There are 2 docs with same name "bitmap", which cause error
when building help system.
2020-10-20 09:47:51 +08:00
09f97f8a05 [Refactor] Fixes some be typo part 2 (#4747) 2020-10-20 09:28:57 +08:00
34d5ab8e07 [BUG] Ensure that the correct lead/lag function is selected (#4732)
* fix

* fix
2020-10-18 21:38:10 +08:00
45fa67aa71 [Refactor] Remove objects which are only used for unit test (#4751)
We create some objects which are only used for unit tests, it's not necessary,
and it may cause create duplicate instances for some classes.
This patch remove unnecessary instance of class BlockManager and StoragePageCache.
2020-10-18 21:37:12 +08:00
e31b4a4561 [Bug] fix illegal defer in Tablet::rowset_with_max_version() (#4737) 2020-10-17 13:44:15 +08:00
0199055be7 [Document] Fix some errors in the insert document (#4749) 2020-10-17 13:40:40 +08:00
d371d363a4 Fix bug CTE statement with nested select (#4731)
Co-authored-by: lixueyan07 <lixueyan07@meituan.com>
2020-10-17 13:36:14 +08:00
fa28542da8 Fix ut (#4752)
Change-Id: I7e3eb8bea00518ad71fd0b63d6fd9c440a536993
2020-10-16 15:35:15 +08:00
21e7428999 [Feature] Support cancel load jobs in batch (#4515)
Support statement like:
`cancel load where label like 'xxx';`
2020-10-15 22:49:39 +08:00
6b0d456b41 [Doc]Fixed a word error in outfile.md document syntax (#4743) 2020-10-15 22:29:43 +08:00
ce270f0139 [License] Modify the license of thirdparties (#4740)
About MySQL and LZO
2020-10-15 22:28:49 +08:00
5909677cb3 [Build][Doc]Fix doc build crash (#4748)
Vuepress build failed caused by the latest `vue-router`
refer to https://github.com/vuejs/vuepress/issues/2656
2020-10-15 20:29:35 +08:00
3438a746ac [Typo] Fix typo in metrics macros (#4739)
Just fix typo.
Rename DEFINE_GAUGE_METRIC_PROTOTYPE_5ARG(name, unit) to DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(name, unit)
Rename DEFINE_GAUGE_METRIC_PROTOTYPE_2ARG(name, unit) witch define core metrics to DEFINE_GAUGE_CORE_METRIC_PROTOTYPE_2ARG(name, unit)
2020-10-15 19:56:43 +08:00
e7220c9cd5 [Doc] Fix wrong title of function document (#4744)
workofyear to weekofyear
2020-10-15 17:29:20 +08:00
a605b3160f [Docs] update data types doc and fix some typo (#4712)
* update data types doc and fix some typo

* update data types doc and fix some typo

Co-authored-by: lixueyan07 <lixueyan07@meituan.com>
2020-10-14 09:34:58 +08:00
96aeeac5b1 fix docs typo (#4724) 2020-10-14 09:33:11 +08:00
5a6e926c7b fix join error when the table has enbale batch delete (#4734) 2020-10-14 09:32:57 +08:00
453aeb7a19 [UT] Fixes PartitionCacheTest unit test randomly failed (#4729)
Fixes PartitionCacheTest unit test randomly failed. #4728
2020-10-14 09:31:15 +08:00
2959a6b24f fix docs typo (#4727) 2020-10-14 09:28:25 +08:00
1f3a430b40 fix docs typo (#4726) 2020-10-14 09:28:07 +08:00
751aa05cc0 fix docs typo (#4725) 2020-10-14 09:27:50 +08:00
dec91a3d43 fix docs typo (#4723) 2020-10-14 09:27:31 +08:00
3f55c1425c fix docs typo (#4722) 2020-10-14 09:27:12 +08:00
bf8f68f751 [Bug] Add regular column when materialized slot is empty in tuple (#4719)
If the materialized slot is empty in inline view, the scanner will throw exception 'no materialized slot!'.
When the outer query does not need any real columns of the inner query at all, such as the outer query only contains constant calculations,
all slots in the inner query will not be materialized.
For example:
```
select c1 from (select 'xx' c1, k1 from test) t1;
```

But this does not mean that the inner query (select 'xx' c1, k1 from test) does not need to be executed.
Because the number of rows in the outer result needs to be determined by the number of rows in the inner query.
In this case, the inner query scan node needs to add a column to ensure the correct result.
At the same time it can avoid the scan node reporting errors.
2020-10-13 10:12:03 +08:00
c00a5cb543 [Bug] Fix the core problem of function split_part and add the UT of core case (#4721)
issue:#4720
2020-10-13 10:09:39 +08:00
f431d8d94c [Enhance][Log] Make RPC error log more clear (#4702)
At present, when some rpc errors occur, the client cannot obtain the error information well.

And this CL change the RPC error returned to client like this:

```
ERROR 1064 (HY000): errCode = 2, detailMessage = there is no scanNode Backend. [10002: in black list(A error 
occurred: errorCode=2001 errorMessage:Channel inactive error!)]

ERROR 1064 (HY000): failed to send brpc batch, error=The server is overcrowded, error_text=[E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R1][E1011]The server is overcrowded @xx.xx.xx.xx:8060 [R2][E1011]The server is 
overcrowded @xx.xx.xx.xx:8060 [R3][E1011]The server is overcrowded @xx.xx.xx.xx:8060, client: yy.yy.yy.yy
```
2020-10-13 10:08:43 +08:00
83f6f46c34 [Config] Limit the version number of tablet (#4687)
Add a BE config `max_tablet_version_num` to limit the version number of a single tablet.
To avoid too many versions
2020-10-13 10:08:16 +08:00
c934cf93e1 [UI Part 5] Enable HTTP Server 2 by FE config (#4684)
This is the last PR of proposal #4308

1. Add a new FE config `enable_http_server_v2` to enable new HTTP Server implementation. The default value is false.
2. Add a new FE config `http_api_extra_base_path` so that we can set base path for Frontend UI.
3. Refactor the new HTTP API response body. The return http status code is always 200, and using internal code in response body to indicate the certain error.
2020-10-13 10:07:50 +08:00
c69fe23c42 [Bug] Fix hard cardinality check which makes queries fail (#4678)
fix hard `Preconditions.checkState()` by setting `cardinality` to `-1` when computation is invalid.
2020-10-13 10:07:22 +08:00
3f0cb04376 [Feature] Support REPLACE TABLE operation. (#4669)
Implement REPLACE TABLE feature.
Details can be found in document `alter-table-replace-table.md`
2020-10-13 10:06:57 +08:00
75e0ba32a1 Fixes some be typo (#4714) 2020-10-13 09:37:15 +08:00
d73d205de7 [ODBC/MySQL] Support Limit Clause Push Down For ODBC Table And MySQL Table(#4706) (#4707)
1. Support limit clause push down both odbc table and mysql table.
2. Code refactor of ODBC Scan Node, change `build_connect_string` and `query_string` from BE to FE to make it easily to modify
2020-10-11 21:11:04 +08:00
fd5e3011da [Docs] Added some missing documents (#4710)
1. Add ODBC resource in resource manage doc and add english of resource manage doc
2. Add miss part in sidebar
2020-10-11 15:40:53 +08:00
98e71a8b9f [Bug][Function] Fix rand() function return same value (#4709)
fix rand function return same value when no parameter
2020-10-11 15:40:38 +08:00
04f26e4b7f [SQL] Support Bucket Shuffle Join (#4677)
Support Bucket Shuffle Join
issue:#4394
2020-10-11 15:37:32 +08:00
eba595583e [Optimize] Optimize the execution model of compaction to limit memory consumption (#4670)
Currently, there are M threads to do base compaction and N threads to do cumulative compaction for each disk. 
Too many compaction tasks may run out of memory, so the max concurrency of running compaction tasks
is limited by semaphore.
If the running threads cost too much memory, we can't defense it. In addition, reducing concurrency to avoid OOM
will lead to some compaction tasks can't be executed in time and we may encounter more heavy compaction. 
Therefore, concurrency limitation is not enough.

The strategy proposed in #3624  may be effective to solve the OOM. 

A CompactionPermitLimiter is used for compaction limitation, and use single-producer/multi-consumer model.
Producer will try to generate compaction tasks and acquire `permits` for each task. 
The compaction task which can hold `permits` will be executed in thread pool and each finished task will
release its `permits`.

`permits` should be applied for before a compaction task can execute. When the sum of `permits`
held by executing compaction tasks reaches a threshold, subsequent compaction task will be no longer allowed,
until some `permits` are released. Tablet compaction score is used as `permits` of compaction task here.

To some extent, memory consumption can be limited by setting appropriate `permits` threshold.
2020-10-11 11:39:25 +08:00
28f4e922a7 [CREATE TABLE]Support new syntax CREATE TABLE LIKE to clone an existe… (#4705)
Support new synatx CREATE TABLE [IF NOT EXISTS] [db_name].table_name AS [db_name2].table_name2;
to create a new table from existed table with same table schema.
ISSUE: #4355
2020-10-10 21:16:53 +08:00