Commit Graph

740 Commits

Author SHA1 Message Date
4232f787ad [Doc] datax doriswriter use case (#6612)
datax doriswriter use case
2021-10-10 23:03:12 +08:00
237a8ae948 [Feature] support spark connector sink data using sql (#6796)
Co-authored-by: wei.zhao <wei.zhao@aispeech.com>
2021-10-09 15:47:36 +08:00
7a20d6d4c2 [Doc] Modify document of resource tag (#6778)
Fix typo
2021-10-03 11:37:45 +08:00
e7707c8180 [FOLLOWUP] create table like clause support copy rollup (#6580)
* Remove `ALL` key word to make grammar more clear.

Co-authored-by: qzsee <shizhiqiang03@meituan.com>
2021-09-30 18:26:21 +08:00
ad3c9390a2 [Bug] Fix bdbje getDatabaseNames() bug and scan node close bug (#6769)
1. This bug is introduced from #6582
2. Optimize the error log of Address used used error msg.
3. Add some document about compilation.
    1. Add a custom thirdparty download url.
    2. Add a custom com.alibaba maven jar package for DataX.
4. Fix bug that BE crash when closing scan node, introduced from #6622.
2021-09-29 11:11:28 +08:00
8d471007a6 [Feature] support spark connector sink stream data to doris (#6761)
* [Feature] support spark connector sink stream data to doris

* [Doc] Add spark-connector batch/stream writing instructions

* add license and remove meaningless blanks code

Co-authored-by: wei.zhao <wei.zhao@aispeech.com>
2021-09-28 17:46:19 +08:00
df5ba6b5a2 [Fix] Flink connector support json import and use httpclient to streamlaod (#6740)
* [Bug]:fix when data null , throw NullPointerException

* [Bug]:Distinguish between null and empty string

* [Feature]:flink-connector supports streamload parameters

* [Fix]:code style

* [Fix]: support json format import and use httpclient to streamload

* [Fix]:remove System out

* [Fix]:upgrade httpclient  version

* [Doc]: add json format import doc

Co-authored-by: wudi <wud3@shuhaisc.com>
2021-09-28 17:37:03 +08:00
cdf9f9e980 [Dynamic Partition] reserve specific history periods by dynamic partition. (#6554)
Add RESERVED_HISTORY_STARTS and RESERVED_HISTORY_ENDS.
Fixes #6514
2021-09-28 11:39:35 +08:00
982b76c3c0 [Bug] Fix resource tag bug, add documents and some other bug fix (#6708)
1. Fix bug of UNKNOWN Operation Type 91
2. Support using resource_tag property of user to limit the usage of BE
3. Add new FE config `disable_tablet_scheduler` to disable tablet scheduler.
4. Add documents for resource tag.
5. Modify the default value of FE config `default_db_data_quota_bytes` to 1PB.
6. Add a new BE config `disable_compaction_trace_log` to disable the trace log of compaction time cost.
7. Modify the default value of BE config `remote_storage_read_buffer_mb` to 16MB
8. Fix `show backends` results error
9. Add new BE config `external_table_connect_timeout_sec` to set the timeout when connecting to odbc and mysql table.
10. Modify issue template to enable blank issue, for release note or other specific usage.
11. Fix a bug in alpha_row_set split_range() function.
2021-09-28 10:37:42 +08:00
a121124fb2 [Doc] Update doris-on-es.md (#6734)
Typo
2021-09-25 12:28:03 +08:00
f3d4c475b1 [DOC] Add connection reset exception solution (#6733)
Add solution for connection reset exception when doing stream load.
2021-09-25 12:27:35 +08:00
ec777aa122 [DOCS] improve docs (#6718) 2021-09-25 12:26:41 +08:00
e5a4172b27 [Bug][Docs]Fix outfile docs for parquet (#6709)
Update outfile documents for parquet.
2021-09-25 12:24:52 +08:00
36d6788bc3 [Optimize] Use compact mode to send query plan thrift data structure. (#6702)
In some cases, the query plan thrift structure of a query may be very large
(for example, when there are many columns in SQL), resulting in a large number
of "send fragment timeout" errors.

This PR adds an FE config to control whether to transmit the query plan in a compressed format.

Using compressed format transmission can reduce the size by ~50%. But it may reduce
the concurrency by ~10%. Therefore, in the high concurrency small query scenario,
you can choose to turn off compaction.
2021-09-25 12:13:29 +08:00
56031cbbe1 [Doc] Change CN/EN sql-functions single quote in markdown (#6698) 2021-09-24 21:42:52 +08:00
f73af475ce [HTTP API] Add aggregation type information in table schema api (#6686)
```
{
	"msg": "success",
	"code": 0,
	"data": {
		"properties": [{
			"type": "INT",
			"name": "k1",
			"comment": "",
			"aggregation_type":""
		}, {
			"type": "INT",
			"name": "k2",
			"comment": "",
			"aggregation_type":"MAX"
		}],
		"status": 200
	},
	"count": 0
}
```
2021-09-24 21:42:24 +08:00
e03b74ebc1 [Doc] Add the error code document of returned by the OLAP function on the BE side (#6666) 2021-09-24 21:40:20 +08:00
bdc8c98008 [Outfile] Support hdfs in select outfile clause (#6644)
Support hdfs in select outfile clause without broker.
This PR implement a HDFS writer in BE which is used to write HDFS file directly without using broker.
Also the hdfs outfile clause syntax check has been added in FE.
The syntax:
```
select * from xx into outfile "hdfs://user/outfile_" format as csv
properties ("hdfs.fs.dafultFS" = "xxx", "hdfs.hdfs_user" = "xxx");
```
Note that all hdfs configurations need to carry a prefix `hdfs.`.
2021-09-24 10:07:11 +08:00
840a7ef3a8 Fix a typo (#6688)
Fix a typo
2021-09-23 09:44:46 +08:00
521fb15a9b [Bug] Fix some memory bugs (#6699)
1. Fix a memory leak in `collect_iterator.cpp` (Fix #6700)
2. Add a new BE config `max_segment_num_per_rowset` to limit the num of segment in new rowset.(Fix #6701)
3. Make the error msg of stream load more friendly.
2021-09-22 12:30:14 +08:00
085942b30f [Doc] Download hashes and signatures use "downloads.apache.org" (#6677)
The latest release should use https://www.apache.org/dyn/closer.lua <https://www.apache.org/dyn/closer.lua>
The latest hashes and signatures should use https://downloads.apache.org/
The old release should use http://archive.apache.org/dist
2021-09-16 18:09:08 +08:00
7ee39743de [Doc] Fix tabletScore expression in be_config.md (#6638)
Co-authored-by: Geoffrey <gaofeng01@rd.netease.com>
2021-09-16 10:24:46 +08:00
225bdb1fda [Bug] fix replace function bug (#6605)
* fix replace function bug

* fix replace docs
2021-09-14 09:59:13 +08:00
5d3c7fbd80 add doc for storage_root_path (#6593)
* add doc for storage_root_path

* Maintain consistency in both Chinese and English documents

Co-authored-by: Geoffrey <gaofeng01@rd.netease.com>
2021-09-10 09:52:58 +08:00
b3f02955d3 [Doc] modify irregular documents (like/ not like/ regexp.md) (#6572) 2021-09-09 14:11:37 +08:00
9469b2ce1a [Outfile] Support concurrent export of query results (#6539)
This pr mainly supports
1. Export query result sets concurrently
2. Query result set export supports s3 protocol

Among them, there are several preconditions for concurrently exporting query result sets
1. Enable concurrent export variables
2. The query itself can be exported concurrently
    (some queries containing sort nodes at the top level cannot be exported concurrently)
3. Export the s3 protocol used instead of the broker

After exporting the result set concurrently,
the file prefix is changed to outfile_{query_instance_id}_filenumber.{file_format}
2021-09-07 11:53:32 +08:00
79fd117d60 Update load-json-format.md (#6546)
change stripe_outer_array to strip_outer_array
2021-09-02 16:08:09 +08:00
7a15e583a7 [Feature]Support functions of json_array, json_object, json_quote (#6504) 2021-09-02 09:59:02 +08:00
Pxl
4dd610c28d [Feature] Support for storage layer benchmark (#6506)
* add benchmark tool
2021-09-02 09:57:19 +08:00
e01a845a4a [Doc] Update stream-load-manual.md (#6524)
Origin stream load column order transformation is unclear , a user is struggling for a long time in this part ,so i modified some expressions to make it clearer.
2021-09-01 13:28:25 +08:00
a949dcd9f6 [Feature] Create table like clause support copy rollup (#6475)
for issue #6474

```sql
create table test.table1 like test.table with rollup r1,r2 -- copy some rollup

create table test.table1 like test.table with rollup all -- copy all rollup

create table test.table1 like test.table  -- only copy base table
```
2021-08-31 20:33:26 +08:00
7324f4b0ae [Bug] Regularly clean up old DeleteInfos in the DeleteHandler (#6448)
fix #6447
1. FE master regularly triggers the remove operation
2. After the master completes the removal of deleteInfo, it is synchronized to the Follower through editlog for remove
3. When the DeleteInfo creation time is longer than the current time, it will be cleaned up, which is determined by the `delete_info_keep_max_second` configuration
2021-08-30 18:52:18 +08:00
0393c9b3b9 [Optimize] Support send batch parallelism for olap table sink (#6397)
* Support send batch parallelism for olap table sink

Co-authored-by: caiconghui <caiconghui@xiaomi.com>
2021-08-30 11:03:09 +08:00
Pxl
5eed1f897a [Document] update docker env version to 1.3.1 (#6517)
* update docker env version
2021-08-30 11:01:39 +08:00
3f2fdd236f Add scan thread token (#6443) 2021-08-27 10:56:17 +08:00
5419d74abf [Doc]Update hit-the-rollup.md (#6430) 2021-08-25 22:35:05 +08:00
92e50504e5 [Feature] Supports case-insensitive table names. (#6403)
Implement the lower_case_table_names variable of mysql. The value meaning is as follows:
0: the table names are case-sensitive.
1: table names are stored in lowercase and comparisons are not case sensitive.
2: table names are stored as given but compared case-insensitively.
2021-08-25 22:34:45 +08:00
c71f58fef9 [Doc] Add sidebar for percentile doc (#6470) 2021-08-22 22:03:07 +08:00
0cf2bc6644 [Doc] Refactor all grammar help documents (#6337)
See #6336 for details
2021-08-22 22:02:51 +08:00
4ea2fcefbc [Improve]The connector supports spark 3.0, flink 1.13 (#6449)
Modify the flink/spark compilation documentation
2021-08-18 15:57:50 +08:00
66a7a4b294 [Feature] Support exact percentile aggregate function (#6410)
Support to calculate the exact percentile value array of numeric column `col` at the given percentage(s).
2021-08-18 15:56:06 +08:00
8738ce380b Add long text type STRING, with a maximum length of 2GB. Usage is similar to varchar, and there is no guarantee for the performance of storing extremely long data (#6391) 2021-08-18 09:05:40 +08:00
4be06a470f fix typo: dynamic_partitoin -> dynamic_partition (#6445) 2021-08-16 09:17:57 +08:00
42fedc0a56 [Docs] Support json file format in routine load doc (#6439) 2021-08-14 10:25:06 +08:00
6f6d50a484 fix typo: '分许'->'分离' (#6440) 2021-08-14 10:22:28 +08:00
5e6f1b89da [Feature] Support sql block rule (#6192)
Support grammar:
- SHOW SQL_BLOCK_RULE [FOR NAME]
- CREATE SQL_BLOCK_RULE test_rule PROPERTIES ("user"="default", "sql"="select .* from .* join .*", "enable": "true");
- ALTER SQL_BLOCK_RULE test_rule PROPERTIES ("user"="test_user", "enable": "false");
- DROP SQL_BLOCK_RULE test_rule1,test_rule2;
2021-08-13 21:56:34 +08:00
240dd9b110 fix typo: '一下' -> '以下' (#6434) 2021-08-13 12:18:42 +08:00
Pxl
8a267f1ac5 [Feature] Support for cleaning the trash actively (#6323) 2021-08-12 10:07:51 +08:00
708b6c529e [RoutineLoad] Support pause or resume all routine load jobs (#6394)
1. PAUSE ALL ROUTINE LOAD;
2. RESUME ALL ROUTINE LOAD;
2021-08-11 16:38:06 +08:00
7e93405df3 [Alter] Support alter table and column's comment (#6387)
1. alter table tbl1 modify comment "new comment";
2. alter table tbl1 modify column k1 comment "k1", modify column v1 comment "v1";
2021-08-11 16:37:42 +08:00