Commit Graph

400 Commits

Author SHA1 Message Date
a0c19df18c [Website] Redesign the home page of document website (master) (#4069) 2020-07-16 11:36:24 +08:00
78a1dea19d Support using B/K/KB/M/MB/G/GB/T/TB/P/PB as unit in session variable exec_mem_limit (#4063)
Support using B/K/KB/M/MB/G/GB/T/TB/P/PB as unit in  session variable exec_mem_limit
2020-07-13 20:54:14 +08:00
d7893f0fa7 [Bug]Fix some schema change not work right (#4009)
[Bug]Fix some schema change not work right
This CL mainly fix some schema change to varchar type not work right
because forget to logic check && Add ConvertTypeResolver to add
supported convert type in order to avoid forget logic check
2020-07-11 10:18:29 +08:00
d2ab38a5e0 [Feature] Batch update partition's property in one command (#3981)
Support following command.
```
alter table tbl_name modify partition (p1, p2, p3) set ("replication_num" = "3");
```
2020-07-09 21:48:43 +08:00
5a27981e49 [Config] Add thrift_client_retry_interval_ms config in be for thrift client to avoid avalanche disaster in fe thrift server (#4022)
This PR is mainly to add  `thrift_client_retry_interval_ms` config in be for thrift client
to avoid avalanche disaster in fe thrift server and fix some typo and some rpc
setting problems at the same time.
2020-07-08 21:07:00 +08:00
7715a84d4d [Config] Enable some features by default (#4031)
Its time to enable some features by default.

1. Enable FE plugins by setting `plugin_enable=true`
2. Enable dynamic partition by setting `dynamic_partition_enable=true`
3. Enable nio mysql server by setting `mysql_service_nio_enabled=true`

Also modify installation doc, add download link of MySQL client.
2020-07-08 09:59:10 +08:00
b7051d0971 [Config]Make it easier for users to find configuration items needed (#3957)
This PR is to make config items ordered by key and support like predicate for admin show config stmt
2020-07-07 23:12:21 +08:00
c3d9feed75 [Load][Json] Refactor json load logic to make it more reasonable (#4020)
This CL mainly changes:

1. Reorganized the code logic to limit the supported json format to two, and the import behavior is more consistent.
2. Modified the statistical behavior of the number of error rows when loading in json format, so that the error rows can be counted correctly.
3. See `load-json-format.md` to get details of loading json format.
2020-07-07 23:07:28 +08:00
d396408861 Correct typos (#4024) 2020-07-07 13:33:46 +08:00
913b2caac4 [Dynamic Partition]Support set replication number (#3965)
This CL mainly support set replication_num property in dynamic partition
table if dynamic_partition.replication_num is not set, the value is the
same as table's default replication_num.
2020-07-05 16:28:38 +08:00
7351f7c237 [Config]Allower use to config different thrift server model (#3986)
Doris only support TThreadPoolServer model in thrift server, but the
server model is not effective in some high concurrency scenario, so this
PR introduced new config to allow user to choose different server model
by their scenario.
Add new FE config: `thrift_server_type`
2020-07-05 16:24:29 +08:00
6a7583bb08 [Doc] Add doc for setting dev env of FE in Eclipse (#3952)
Also fix some doc bugs
2020-07-02 13:54:36 +08:00
210ee9664f [SparkLoad]add user doc for build global dict (#3938)
describe global dict and how to use it in spark load
2020-06-30 19:12:35 +08:00
48398232e7 [Bug] Fix bug that default_rowset_type have a session variable (#3953)
This  PR is mainly for fixing bug that  `default_rowset_type` have a session variable
2020-06-29 19:16:42 +08:00
2c96d27fdc [Enhance] Add MetaUrl and CompactionUrl for "show tablet" stmt (#3962)
* [Enhance] Add MetaUrl and CompactionUrl for "show tablet" stmt

Add MetaUrl and CompactionUrl in result of following stmt:

`show tablet 10010`;

* fix ut

* add doc

Co-authored-by: chenmingyu <chenmingyu@baidu.com>
2020-06-29 19:15:38 +08:00
af1beb6ce4 [Enhance] Add prepare phase for some timestamp functions (#3947)
Fix: #3946 

CL:
1. Add prepare phase for `from_unixtime()`, `date_format()` and `convert_tz()` functions, to handle the format string once for all.
2. Find the cctz timezone when init `runtime state`, so that don't need to find timezone for each rows.
3. Add constant rewrite rule for `utc_timestamp()`
4. Add doc for `to_date()`
5. Comment out the `push_handler_test`, it can not run in DEBUG mode, will be fixed later.
6. Remove `timezone_db.h/cpp` and add `timezone_utils.h/cpp`

The performance shows bellow:

11,000,000 rows

SQL1: `select count(from_unixtime(k1)) from tbl1;`
Before: 8.85s
After: 2.85s

SQL2: `select count(from_unixtime(k1, '%Y-%m-%d %H:%i:%s')) from tbl1 limit 1;`
Before: 10.73s
After: 4.85s

The date string format seems still slow, we may need a further enhancement about it.
2020-06-29 19:15:09 +08:00
4003ed07b5 add doc 2020-06-28 14:07:11 +08:00
b2b9e22b24 [CreateTable] Check backend disk has available capacity by storage medium before create table (#3519)
Currently we choose BE random without check disk is available, 
the create table will failed until create tablet task is sent to BE
and BE will check is there has available capacity to create tablet.
So check backend disk available by storage medium will reduce unnecessary RPC call.
2020-06-28 09:36:31 +08:00
a894b1edc5 [Doris On ES] Split /_cluster/state to [indexName/_mappings, indexName/_search_shards] (#3454)
1. Split /_cluster/state into /_mapping and /_search_shards requests to reduce permissions and make the logic clearer
2. Rename part es related objects to make their representation more accurate
3. Simply support docValue and Fields in alias mode, and take the first one by default

#3311
2020-06-26 17:46:43 +08:00
b956bd5c8e [Doc] Add document for setting up IntelliJ IDEA (#3939) 2020-06-26 14:35:02 +08:00
6c768f5303 [Doc] Add doc of enable_materialized_view (#3940) 2020-06-25 16:26:18 +08:00
46c64f0861 [Bug] Enable to get TCP metrics for linux kernel 2.x (#3921)
Fix #3920 

CL:
1. Parse the TCP metrics header in `/proc/net/snmp` to get the right position of the metrics.
2. Add 2 new metrics: `tcp_in_segs` and `tcp_out_segs`
2020-06-24 21:29:07 +08:00
feec4ee5bf [UDF] Support external users to contribute udf (#3760) 2020-06-23 13:43:08 +08:00
e5da108110 [Doris On ES][Docs] update document for best practices (#3924)
Add best practices for #3559 and update feature for #3901
2020-06-23 13:39:56 +08:00
4c3ccfb906 [FE] Prohibit pointing helper to itself when starting FE (#3850)
When starting FE with `start_fe.sh --helper xxx` command, do not allow to
point helper to FE itself. Because this is meaningless and may cause some
confusing problemes.
2020-06-22 09:21:08 +08:00
66a8383ac0 [Running_Profile] Fix all counter in DataStreamRecv and change the image path in docs (#3858) 2020-06-22 09:20:22 +08:00
35d07d8012 [Doc] Fix audit-plugin doc error (#3922)
Fix audit-plugin doc error
Droris -> Doris
2020-06-22 09:09:56 +08:00
03fa1fefa9 [Doc] Fix doc-bug (#3914) 2020-06-21 16:39:27 +08:00
5d40218ae6 [Config] Support max_stream_load_timeout_second config in fe (#3902)
This configuration is specifically used to limit timeout setting for stream load.
It is to prevent that failed stream load transactions cannot be canceled within
a short time because of the user's large timeout setting.
2020-06-19 17:09:27 +08:00
5a253bc2c6 [BE][Tool] Add segment v2 footer meta viewer (#3822)
Add segment v2 footer meta viewer tool
2020-06-19 09:32:11 +08:00
2f99f632e8 Modify docs format (#3896) 2020-06-18 09:43:28 +08:00
e9f7576b9d [Enhancement] make metrics api more clear (#3891) 2020-06-17 12:17:54 +08:00
c6f2b5ef0d [Doris On ES][Docs] refator documentation for doe (#3867) 2020-06-17 10:54:28 +08:00
6c4d7c60dd [Feature] Add QueryDetail to store query statistics. (#3744)
1. Store the query statistics in memory.
2. Supporting RESTFUL interface to get the statistics.
2020-06-15 18:16:54 +08:00
2211cb0ee0 [Metrics] Add metrics document and 2 new metrics of TCP (#3835) 2020-06-15 09:48:09 +08:00
3c09e1e1d8 [trace] Adapt trace util to compaction module (#3814)
Trace util is helpful for diagnosing compaction performance problems,
we can get trace log for base compaction like:
```
W0610 11:26:33.804431 56452 storage_engine.cpp:552] Trace:
0610 11:23:03.727535 (+     0us) storage_engine.cpp:554] start to perform base compaction
0610 11:23:03.728961 (+  1426us) storage_engine.cpp:560] found best tablet 546859
0610 11:23:03.728963 (+     2us) base_compaction.cpp:40] got base compaction lock
0610 11:23:03.729029 (+    66us) base_compaction.cpp:44] rowsets picked
0610 11:24:51.784439 (+108055410us) compaction.cpp:46] got concurrency lock and start to do compaction
0610 11:24:51.784818 (+   379us) compaction.cpp:74] prepare finished
0610 11:26:33.359265 (+101574447us) compaction.cpp:87] merge rowsets finished
0610 11:26:33.484481 (+125216us) compaction.cpp:102] output rowset built
0610 11:26:33.484482 (+     1us) compaction.cpp:106] check correctness finished
0610 11:26:33.513197 (+ 28715us) compaction.cpp:110] modify rowsets finished
0610 11:26:33.513300 (+   103us) base_compaction.cpp:49] compaction finished
0610 11:26:33.513441 (+   141us) base_compaction.cpp:56] unused rowsets have been moved to GC queue
Metrics: {"filtered_rows":0,"input_row_num":3346807,"input_rowsets_count":42,"input_rowsets_data_size":1256413170,"input_segments_num":44,"merge_rowsets_latency_us":101574444,"merged_rows":0,"output_row_num":3346807,"output_rowset_data_size":1228439659,"output_segments_num":6}
```
for cumulative compaction like:
```
W0610 11:14:18.714366 56468 storage_engine.cpp:518] Trace:
0610 11:14:08.068484 (+     0us) storage_engine.cpp:520] start to perform cumulative compaction
0610 11:14:08.069844 (+  1360us) storage_engine.cpp:526] found best tablet 547083
0610 11:14:08.069846 (+     2us) cumulative_compaction.cpp:42] got cumulative compaction lock
0610 11:14:08.069947 (+   101us) cumulative_compaction.cpp:46] calculated cumulative point
0610 11:14:08.070141 (+   194us) cumulative_compaction.cpp:50] rowsets picked
0610 11:14:08.070143 (+     2us) compaction.cpp:46] got concurrency lock and start to do compaction
0610 11:14:08.070518 (+   375us) compaction.cpp:74] prepare finished
0610 11:14:15.389893 (+7319375us) compaction.cpp:87] merge rowsets finished
0610 11:14:15.390916 (+  1023us) compaction.cpp:102] output rowset built
0610 11:14:15.390917 (+     1us) compaction.cpp:106] check correctness finished
0610 11:14:15.409460 (+ 18543us) compaction.cpp:110] modify rowsets finished
0610 11:14:15.409496 (+    36us) cumulative_compaction.cpp:55] compaction finished
0610 11:14:15.410138 (+   642us) cumulative_compaction.cpp:65] unused rowsets have been moved to GC queue
Metrics: {"filtered_rows":0,"input_row_num":136707,"input_rowsets_count":302,"input_rowsets_data_size":76617836,"input_segments_num":302,"merge_rowsets_latency_us":7319372,"merged_rows":0,"output_row_num":136707,"output_rowset_data_size":53893280,"output_segments_num":1}
```
2020-06-13 19:31:51 +08:00
b3811f910f [Spark load][Fe 4/6] Add hive external table and update hive table syntax in loadstmt (#3819)
* Add hive external table and update hive table syntax in loadstmt

* Move check hive table from SelectStmt to FromClause and update doc

* Update hive external table en sql reference
2020-06-13 16:28:24 +08:00
414a0a35e5 [Dynamic Partition] Use ZonedDateTime to support set timezone (#3799)
This CL mainly support timezone in dynamic partition:
1. use new Java Time API to replace Calendar.
2. support set time zone in dynamic partition parameters.
2020-06-13 16:27:09 +08:00
b8ee84a120 [Doc] Add docs to OLAP_SCAN_NODE query profile (#3808) 2020-06-13 16:25:40 +08:00
wyb
44dbdf4986 Update hive external table en sql reference 2020-06-12 21:38:05 +08:00
wyb
7f7ee63723 Move check hive table from SelectStmt to FromClause and update doc 2020-06-11 16:53:41 +08:00
8d11ad3a16 [Doc] Fix website doc error (#3823) 2020-06-11 10:01:54 +08:00
86d235a76a [Extension] Logstash Doris output plugin (#3800)
This plugin is used to output data to Doris for logstash
Use the HTTP protocol to interact with the Doris FE Http interface
Load data through Doris's stream load
2020-06-11 08:54:51 +08:00
4adc9d45c2 [Doc] Update ALTER TABLE.md 2020-06-10 22:58:29 +08:00
de91037d8c [Doc]Add some routine load docs (#3796)
Add some documentation about using routine load in the cloud environment
2020-06-10 22:57:00 +08:00
4cb5f7a535 [Config]Remove max_user_connections from config (#3805)
Update max_user_connections by user property:

```
set property `user` max_user_connections=100;
```
2020-06-10 22:56:05 +08:00
wyb
4c2e73a5fe Add hive external table and update hive table syntax in loadstmt 2020-06-10 16:32:32 +08:00
a7bf006b51 Use BackendStatus to show BE's infomation in show backends; (#3713)
The infomation is displayed in JSON format.For example:
{"lastTabletReportTime":"2020-05-28 15:29:01"}
2020-06-06 11:37:48 +08:00
ed9022a908 Ignore broken disk when BE starts up (#3741) 2020-06-05 10:26:07 +08:00
73719f263d Fix document (#3773) 2020-06-05 10:19:17 +08:00