Commit Graph

1322 Commits

Author SHA1 Message Date
caa7af3d1f [Metric] Standardise histogram metric output for prometheus (#5671)
Update histogram metric's output to prometheus standard, the output
like following:

test_registry_task_duration{quantile="0.50"} 50
test_registry_task_duration{quantile="0.75"} 75
test_registry_task_duration{quantile="0.90"} 95.8333
test_registry_task_duration{quantile="0.95"} 100
test_registry_task_duration{quantile="0.99"} 100
test_registry_task_duration_sum 5050
test_registry_task_duration_count 100
2021-04-20 09:14:28 +08:00
b4a4c29651 (#5638) stale rowset can't be access after clone finish (#5639)
* (#5638) stale rowset can't be access after clone finish

* clear stale rowset after clone
2021-04-19 09:27:41 +08:00
6f000c2ea4 (#5621) using KeyMayExist instead of Get when visit rocksdb (#5622) 2021-04-19 09:25:59 +08:00
892fbf6ded Update s3_reader_test.cpp (#5658) 2021-04-15 10:59:30 +08:00
c4cc681d14 remove boost_foreach, using c++ foreach instead (#5611) 2021-04-15 10:52:29 +08:00
50ffae44b1 [BUG] Fix bug that Unique/AGG key will read all key columns when there are two rowsets (#5632) 2021-04-14 00:12:05 +08:00
70b0d113e1 fix license header (#5630) 2021-04-13 11:01:29 +08:00
75db273b93 [Doris On ES][WIP] Support external ES table with SSL secured and configurable node sniffing (#5325)
Support external ES  table with `SSL` secured and configurable node sniffing
2021-04-12 11:23:49 +08:00
9c7d8d2e98 [Bug] Fix bug that isPreAggregation is incorrectly set (#5608)
1. The MaterializedViewSelector should be reset for each scan node
2. On the BE side, columns with delete conditions must be added to the return column.
2021-04-09 14:13:06 +08:00
40f53ac71f fix bitmap unit test failed (#5610) 2021-04-08 10:25:59 +08:00
2b5e4dc951 Add fmt library to speed up mysql text result serialization (#5554)
* Add fmt library to speed up mysql text result serialization

* use BUILD_SYSTEM instead of make

Co-authored-by: gaodayue <gaodayue@bytedance.com>
2021-04-08 09:16:58 +08:00
b423274f17 [Enhance] Make MemTracker more accurate (#5515) (#5516)
* [Enhance] Make MemTracker more accurate (#5515)
 This PR main about:
 1. Improve the readability of MemTrackers' name
 2. Add the MemTracker of:
    * Load
    * Compaction
    * SchemaChange
    * StoragePageCache
    * TabletManager
 3. Change SchemaChange to a Singleon

* revise some code for Code Review

* change the name of mem_tracker

* keep reader_context have the same lifetime of rowset_reader in schema change.

* change vlog notice to log(warning) in schema change
2021-04-08 09:14:55 +08:00
5a0a039026 [refacor] Remove minizip source code (#5571) 2021-04-08 09:12:22 +08:00
d641a26490 [Refactor] Remove boost filesystem (#5579)
* use std::filesystem instead of boost
Co-authored-by: Mingyu Chen <morningman.cmy@gmail.com>
2021-04-08 09:11:59 +08:00
1e8c4584ab [Function] Add BE udf bitmap_min (#2538) (#5581)
this function will return the min result of the input bitmap .
2021-04-08 09:11:32 +08:00
79544d39cb [Metrics][LOG] Update metrics of 'max_compaction_score' and log for compaction (#5592)
* optimize compaction metrics and log

Co-authored-by: weizuo <weizuo@xiaomi.com>
2021-04-08 09:10:40 +08:00
bef2904f28 [Refactor] refactor the destructor in olap_meta.cpp (#5563) 2021-04-06 22:28:04 +08:00
e2c702c959 remove some unused file for compatible to older kernels (#5568) 2021-03-29 11:32:59 +08:00
1ddaac7832 [Compaction] Optimize cumulative compaction point calculate when after restart be (#5561)
This pr is to reduce a large number of useless cumulative compactions caused by
cumulative compaction point calculation problems after be restarts.
2021-03-27 18:59:50 +08:00
a56e7e2192 [Refactor] make uint24_t,OLAPIndexFixedHeader as a POD type (#5559) 2021-03-27 18:59:23 +08:00
86af8c76a3 [DOC] Add docs of load and export using S3 protocol (#5551)
Add docs of load and export using S3 protocol
2021-03-27 18:58:29 +08:00
ad67dd34a0 update gcc to gcc 10 and support c++17 (#5394)
* update gcc to gcc 10 and support c++17
    update brpc to 0.9.7
    update boost to 1.73
    remove third-party boost 1.54 for mysql

* update cmake version

* ignore jdk version

* remove unused patch

* avoid use SYS_getrandom call
2021-03-25 09:30:38 +08:00
bfeb717abe [Refactor] fix some warning in gcc higher than 7 make decimal12_t as a POD type (#5547) 2021-03-23 09:37:10 +08:00
cef3cbc53a [Bug] Fix bug that the last column may be null when using multibytes separator (#5534) 2021-03-23 09:35:30 +08:00
3a96b6dfbd [Bug] Fix the bug of rowset file being deleted by mistake (#5541)
* [Bug] Fix the bug of rowset file being deleted by mistake
2021-03-23 09:33:36 +08:00
a91888a68b [BUG] fix memory limit failure and optimize memory usage in join stage (#5514)
This patch works well on tpcds-1T query-24
2021-03-21 11:32:51 +08:00
8343abaad1 [Feature] Local Exechange (#5470)
Avoid network transmission when the data stream sender node and destination exchange node
are in same BE, to improve performance and save CPU.
2021-03-21 11:25:33 +08:00
19b3a950de [ODBC] change SQL_DRIVER_COMPLETE_REQUIRED to SQL_DRIVER_NOPROMPT make mysql connect err clear (#5538) 2021-03-21 11:20:25 +08:00
5fa56e162b [Bug] Fix compaction failed when doing incremental clone (#5540) 2021-03-21 11:18:15 +08:00
087fd8159b [Bug] Fix bug that the stale rowset file will not be deleted (#5527)
1. If cumulative compaction compact only one rowset, the old rowset will not be put into `stale_rowset_meta_map`

2. Show rowset id in `/api/compaction/show`

Co-authored-by: xxiao2018 <benghua3_1@sina.com>
2021-03-17 22:31:05 +08:00
a1bce25677 [BUG] Fix Memory Leak in SchemaChange And Fix some DCHECK error (#5491) 2021-03-17 09:27:05 +08:00
105a86d1cd Use O_SYNC instead of O_DIRECT to be fs agnostic. (#5518)
Some fs might not support O_DIRECT and O_SYNC is semantically the same to be used for disk checking.
2021-03-15 10:07:51 +08:00
1100a0f3a0 [Profile] Add more timer for scan thread (#5511)
1.
Add timer to count the time the transfer thread waits for the scaner thread to return rowbatch.
2.
Add timer to count the time that the scanner thread waits for the available worker threads in the thread pool.

Co-authored-by: chenmingyu <chenmingyu@baidu.com>
2021-03-15 10:07:11 +08:00
4b316e4c3f [Outfile] Support exporting query result to local disk (#5489)
1.
User can export query result to local disk like:

`select * from tbl into outfile ("file:///disk1/result_");`

And modify the return result to show the details of export:

```
mysql> select * from tbl1 limit 10 into outfile "file:///home/work/path/result_";
+------------+-----------+----------+--------------+
| FileNumber | TotalRows | FileSize | URL          |
+------------+-----------+----------+--------------+
|          1 |         2 |        8 | 192.168.1.10 |
+------------+-----------+----------+--------------+
```

2.
Support create a mark file after export successfully finished.

Co-authored-by: chenmingyu <chenmingyu@baidu.com>
2021-03-14 15:39:46 +08:00
e9a73ee278 [Bug] Fix the memory expand 10~1000x of compression algorithm (#5504)
Fix the memory expand 10~1000x of compression algorithm in load and compaction
2021-03-12 23:04:07 +08:00
c9a25aa29e [UT] fix memory tracker ut (#5501)
* [UT] fix memory tracker ut

* Update mem_limit_test.cpp
2021-03-12 13:45:04 +08:00
8ead0aaad8 [Enhance] Sort directories by available space when do trash sweep (#5498)
* [Enhance] Sort directories by available space when do trash sweep

In the case when one disk is about to be full, we want to sweep trash
data on this disk as quickly as possible. The currently trash sweep
function is to remove trashed files order by path's name, however, disk
data directories may have some large different available space because
of the load balance algorithm, this patch improve it to remove files by
directories' available space.

* add log
2021-03-12 13:43:27 +08:00
9254e78e57 Fix compatibility of glibc (#5502)
* fix compatibility

* remove eventfd.c because eventfd in differene glibc has different declarition
2021-03-12 11:44:43 +08:00
689602e686 [Enhancement] Support Pallralel Merge In Exchange Node (#5468)
Support Parallel Merge In Exchange Node
2021-03-11 22:34:18 +08:00
0131c33966 [Enhance] Improve the readability of memtrackers' name (#5455)
Improve the readability of memtrackers' name, then you will be happy to read website be_ip:port/mem_tracker
2021-03-11 22:33:31 +08:00
e5c7a6dd9f [Bug] hll serialize 160 items cause backend crash(#5424) (#5425)
Co-authored-by: lanhuajian <lanhuajian@sankuai.com>
2021-03-11 22:24:01 +08:00
7a8fbe5db8 [internal] [doris-1084] support compressed csv file in stream load (#5463) 2021-03-11 10:53:05 +08:00
36002bec48 Fix bug that sql with limit query statistics are wrong (#5347)
Co-authored-by: weixiang <weixiang06@meituan.com>

When querying sql is with limit, the query statistics will be enlarged by merging the query statistics of each batch.

Fix #5340
2021-03-10 10:24:12 +08:00
e023ef5404 [Load] Support multi bytes LineDelimiter and ColumnSeparator (#5462)
* [Internal][Support Multibytes Separator] doris-1079
support multi bytes LineDelimiter and ColumnSeparator
2021-03-09 09:35:39 +08:00
43dd583cfc Fix dlopen faild by upgrade cmake (#5481)
* fix dlopen faild

* remove useless code
2021-03-08 09:02:53 +08:00
db2120a7f2 [Build][BE] Fix GLIBC_COMPATIBILITY can not compile in centos6 (#5472)
Add option to disable glibc_compatibility
2021-03-07 20:47:13 +08:00
35f5cb8e0c [Bug] Fix bug that BE failed to start when validating conf from be_custom.conf (#5465) 2021-03-07 17:37:14 +08:00
d6ac8f4e35 Masking glibc symbols for better portability (#4180)
* Masking glibc symbols for better portability

* Remove redundant files
2021-03-05 13:15:55 +08:00
805f98e0f9 [Bug] Set dest tuple to null when src_tuple is NULL. (#5431) 2021-03-04 22:26:05 +08:00
4e1b6b3eef [ODBC] Let the type conversion of the fail in query in ODBC of MySQL table to prompt the information of the column (#5422)
Let the type conversion of the fail in query in ODBC of MySQL table to prompt the information of the column
2021-03-04 22:23:37 +08:00