Commit Graph

3269 Commits

Author SHA1 Message Date
e795c7d2cc [Community] Add new template for issues (#6534)
* [Community] Add new template for issues

Inpired by Apache Skywalking

https://github.com/apache/skywalking/issues/new?assignees=&labels=bug&template=bug-report.yml&title=%5BBug%5D+
2021-09-01 09:59:44 +08:00
a949dcd9f6 [Feature] Create table like clause support copy rollup (#6475)
for issue #6474

```sql
create table test.table1 like test.table with rollup r1,r2 -- copy some rollup

create table test.table1 like test.table with rollup all -- copy all rollup

create table test.table1 like test.table  -- only copy base table
```
2021-08-31 20:33:26 +08:00
138e7e896d Fix min(string) Unable to find symbol (#6531) 2021-08-31 11:19:13 +08:00
d5d8316ff3 [Optimize][Clone] Take version count into consideration when choosing src replica for clone task (#6513)
Fix #6512 

If there is missing replica for a tablet, clone task will be executed to restore missing replica from a healthy replica. Src replica selector will randomly choose a healthy replica as src replica.

It's better to choose the health replica with min version count as src replica so that it could avoid repetitive compaction task. In addition, replica with less version count is good for query performance.
2021-08-30 18:52:41 +08:00
7324f4b0ae [Bug] Regularly clean up old DeleteInfos in the DeleteHandler (#6448)
fix #6447
1. FE master regularly triggers the remove operation
2. After the master completes the removal of deleteInfo, it is synchronized to the Follower through editlog for remove
3. When the DeleteInfo creation time is longer than the current time, it will be cleaned up, which is determined by the `delete_info_keep_max_second` configuration
2021-08-30 18:52:18 +08:00
abbc9202af support routine load isolation_level read_committed (#6191)
Co-authored-by: Geoffrey <gaofeng01@rd.netease.com>
2021-08-30 17:22:08 +08:00
0393c9b3b9 [Optimize] Support send batch parallelism for olap table sink (#6397)
* Support send batch parallelism for olap table sink

Co-authored-by: caiconghui <caiconghui@xiaomi.com>
2021-08-30 11:03:09 +08:00
Pxl
5eed1f897a [Document] update docker env version to 1.3.1 (#6517)
* update docker env version
2021-08-30 11:01:39 +08:00
a2a13dadba [Optimize] Make light schema change complete more faster under concurrent conditions (#6292)
* [Optimize] Make schema change complete more faster under concurrent conditions

Co-authored-by: caiconghui <caiconghui@xiaomi.com>
2021-08-29 09:41:56 +08:00
dedb57f87e [Enhancement] Modify the method of calculating compaction score (#6252)
* optimize calculation method of compaction score to lower the priority of rowset with 0 segments

Co-authored-by: weizuo <weizuo@xiaomi.com>
2021-08-27 11:10:41 +08:00
ace21ebf83 Doris-spark connector examples (#6485)
* doris spark connector examples

* add usage documentation and license

Co-authored-by: shengy <whyMy2017>
2021-08-27 10:57:11 +08:00
3f2fdd236f Add scan thread token (#6443) 2021-08-27 10:56:17 +08:00
4cfebc35a7 Flink reads multiple data sources to doris (#6490)
* Flink reads multiple data sources to doris

Co-authored-by: caol <caol@shuhaisc.com>
2021-08-27 10:55:53 +08:00
a7b8d110a0 Spark 2.x and 3.x version compilation instructions (#6503)
Spark 2.x and 3.x version compilation instructions
2021-08-27 10:55:29 +08:00
7235d86331 [Bug] Support show load for insert 0 row (#6510)
* support show load for insert 0 row

* support show load for insert 0 row

Co-authored-by: weizuo <weizuo@xiaomi.com>
2021-08-27 10:55:02 +08:00
bfb2252175 [RuntimeFilter] provide no simd block bloom filter implement to support arm (#6511) 2021-08-27 10:22:36 +08:00
ca3eb6490e push down conditions on unique table value columns to base rowset (#6457) 2021-08-26 09:14:49 +08:00
acc5fd2f21 [BUG] Fix string type cast bug and runtime filter may core when not support avx2 (#6495)
* fix string type cast bug and runtime filter instructions may not support

* add arm support
2021-08-26 09:14:31 +08:00
5419d74abf [Doc]Update hit-the-rollup.md (#6430) 2021-08-25 22:35:05 +08:00
92e50504e5 [Feature] Supports case-insensitive table names. (#6403)
Implement the lower_case_table_names variable of mysql. The value meaning is as follows:
0: the table names are case-sensitive.
1: table names are stored in lowercase and comparisons are not case sensitive.
2: table names are stored as given but compared case-insensitively.
2021-08-25 22:34:45 +08:00
96013decd3 [BUG] Fixed the materialized number of resultExprs/constExprs and output slot of Union Node is inconsistent (#6380) 2021-08-25 22:33:49 +08:00
fa290383dc [Doc] Modify README to add some statistical indicators (#6486)
1. Add license/total line/release badegs.
2. Add monthly active contributor and contributor growth graph
3. fix a pom.xml bug
4. Modify some routine load log on BE side
2021-08-25 09:36:26 +08:00
7e30b28f3a [Optimize] Speed up converting the data of other types to string in mysql_result_writer (#6384)
Co-authored-by: caiconghui <caiconghui@xiaomi.com>
2021-08-24 22:30:58 +08:00
146060dfc0 [Bug]Fix result_writer may coredump (#6482)
fix result_writer may coredump, let BufferControlBlock owns the memory
2021-08-22 22:04:00 +08:00
4ff6eb55d0 [FlinkConnector] Make flink datastream source parameterized (#6473)
make flink datastream source parameterized as List<?> instead of Object.
2021-08-22 22:03:32 +08:00
c71f58fef9 [Doc] Add sidebar for percentile doc (#6470) 2021-08-22 22:03:07 +08:00
0cf2bc6644 [Doc] Refactor all grammar help documents (#6337)
See #6336 for details
2021-08-22 22:02:51 +08:00
6c23f8d413 [Bug] Fix bug that check point load image failed in some circumstance (#6465)
Fix bug that check point load image failed in some circumstance
2021-08-19 14:17:57 +08:00
52f39e3fde [Bug][SparkLoad]: bitmap value in or operator in spark load should be deep copied (#6453)
fix multi rollup hold the same Ref of bitmapvalue which may be updated repeatedly.
fix #6452
2021-08-19 14:17:31 +08:00
fa382f8602 [Bug][MemLimit] Modify the memory limit of storage page cache (#6451)
This CL mainly changes:

1. the `storage_page_cache_limit` is based on config `mem_limit`

    the default is 20% of `mem_limit`. 

2. the `buffer_pool_limit` is based on config `mem_limit`

    the default is 20% of `mem_limit`. 

3. the `buffer_pool_clean_pages_limit` is based on config `buffer_pool_limit`

    the default is 50% of `buffer_pool_limit`

4. Fix some show bugs of lru cache hit ratio and usage ratio
5. Fix a create view bug that `notEvalNondeterministicFunction` should be reset after analyze.
2021-08-19 14:16:53 +08:00
c65ec3136b [Improvement] spark load without agg and de/serialization (#6270)
fix #6269 

The outline of our changes is to improve our memory in case of OOM in BE and to speed up the calculation.
1. We do not need to do Aggregation in load, which has already been done in the ETL spark job.
2. Based on 1, we do not need to serialize/deserialize bitmap/HLL objects.
2021-08-19 14:15:01 +08:00
4ea2fcefbc [Improve]The connector supports spark 3.0, flink 1.13 (#6449)
Modify the flink/spark compilation documentation
2021-08-18 15:57:50 +08:00
66a7a4b294 [Feature] Support exact percentile aggregate function (#6410)
Support to calculate the exact percentile value array of numeric column `col` at the given percentage(s).
2021-08-18 15:56:06 +08:00
9148bcb673 [Build] Reduce the parallel of build (#6469) 2021-08-18 15:24:19 +08:00
Pxl
999eaeb276 fix Wrong use on SCOPED_RAW_TIMER (#6459) 2021-08-18 09:06:18 +08:00
0c5c3f7d87 Fixed the problem that there may be redundant retries when the query result export fails (#6436) 2021-08-18 09:06:02 +08:00
8738ce380b Add long text type STRING, with a maximum length of 2GB. Usage is similar to varchar, and there is no guarantee for the performance of storing extremely long data (#6391) 2021-08-18 09:05:40 +08:00
2f90aaab8e [Doc] flink/spark connector: add sources/javadoc plugins (#6435)
spark-doris-connector/flink-doris-connect add plugins to generate javadoc and sources jar,
so can be easy to distribute and debug.
2021-08-16 22:41:24 +08:00
b13e512a65 [Feature] Support spark connector sink data to Doris (#6256)
support spark conector write dataframe to doris
2021-08-16 22:40:43 +08:00
63a0d9d23a Add statistics struct and Support manually inject statistics (#6420)
* Add statistics struct and Support manually inject statistics

This PR mainly developed the data structure used by statistical information
and the function of manually modifying the statistical information.
We use a statistics package alone to store statistical information,
and use the 'statistics manager' as a unified entry for statistical information.
For detailed data structure and explanation, please refer to the comments on the class.

Manually modify statistics include: Manually modify table statistics and column statistics.
The syntax is explained in the issue #6370.

* Show table and column statistics

'SHOW TABLE STATS' used to show the statistics of table.
'SHOW COLUMN STATS' used to show the statistics of columns.

Currently, only the tables and columns for setting statistics
will be displayed in the results.
2021-08-16 17:20:05 +08:00
4be06a470f fix typo: dynamic_partitoin -> dynamic_partition (#6445) 2021-08-16 09:17:57 +08:00
285d44cd48 [BUG] Fix potential overflow exception when do money format for double (#6408)
* [BUG] Fix potential overflow bug when do money format for double

Co-authored-by: caiconghui <caiconghui@xiaomi.com>
2021-08-15 18:40:26 +08:00
2030c44dba [Log] Modify some log level on BE side (#6381) 2021-08-14 10:25:45 +08:00
42fedc0a56 [Docs] Support json file format in routine load doc (#6439) 2021-08-14 10:25:06 +08:00
34af66bf1d [BUG][Memory] fix memory tracker DCHECK fail in debug mode and Fix Process Memory limit fail (#6438) 2021-08-14 10:24:33 +08:00
d9cc235d6d Fix typo pdqosrt (#6441) 2021-08-14 10:24:12 +08:00
6f6d50a484 fix typo: '分许'->'分离' (#6440) 2021-08-14 10:22:28 +08:00
5e6f1b89da [Feature] Support sql block rule (#6192)
Support grammar:
- SHOW SQL_BLOCK_RULE [FOR NAME]
- CREATE SQL_BLOCK_RULE test_rule PROPERTIES ("user"="default", "sql"="select .* from .* join .*", "enable": "true");
- ALTER SQL_BLOCK_RULE test_rule PROPERTIES ("user"="test_user", "enable": "false");
- DROP SQL_BLOCK_RULE test_rule1,test_rule2;
2021-08-13 21:56:34 +08:00
240dd9b110 fix typo: '一下' -> '以下' (#6434) 2021-08-13 12:18:42 +08:00
671d8f6af8 [Bug] Return error if user failed to pause/resume a certain routine load. (#6426)
When operating a single job, maintain the same behavior as before
This problem is introduced by #6394.
2021-08-12 11:50:57 +08:00