Commit Graph

2840 Commits

Author SHA1 Message Date
771cb64290 Solve the situation that the hardware information of the Web UI home page cannot be loaded (#5585)
Solve the situation that the hardware information of the Web UI home page cannot be loaded

Co-authored-by: zhangjf@shuhaisc.com <zhangfeng800729>
2021-04-08 09:10:56 +08:00
79544d39cb [Metrics][LOG] Update metrics of 'max_compaction_score' and log for compaction (#5592)
* optimize compaction metrics and log

Co-authored-by: weizuo <weizuo@xiaomi.com>
2021-04-08 09:10:40 +08:00
621c89f7b9 [Enhance] Filter all non-journal db when getting journal db names (#5605) 2021-04-08 09:09:56 +08:00
bef2904f28 [Refactor] refactor the destructor in olap_meta.cpp (#5563) 2021-04-06 22:28:04 +08:00
18c2553ef8 [FE][Bug] Update Spark version to fix a security issue (#5593)
Fix CVE-2020-9480: Apache Spark RCE vulnerability in
auth-enabled standalone master
https://spark.apache.org/security.html#CVE-2020-9480
2021-04-06 11:02:04 +08:00
862a189570 update readme.md (#5596)
Co-authored-by: weizuo <weizuo@xiaomi.com>
2021-04-06 11:00:40 +08:00
5012fdc049 [FE][Fix]Update commons-collections to fix a security issue (#5595)
Fix CVE-2017-15708
https://www.cvedetails.com/cve/CVE-2017-15708/
2021-04-06 11:00:19 +08:00
68303ea7f3 [FE][Bug]Update log4j-web to fix a security issue (#5594)
Fix CVE-2017-5645

In Apache Log4j 2.x before 2.8.2, when using the
TCP socket server or UDP socket server to receive
serialized log events from another application, a
specially crafted binary payload can be sent that,
when deserialized, can execute arbitrary code.

https://www.cvedetails.com/cve/CVE-2017-5645/
2021-04-06 10:59:40 +08:00
05487e38ae [Bug] upgrade log4j version from 2.12.1 to 2.14.0 to fix performance issue in JDK11 using ZGC (#5591)
Co-authored-by: caiconghui <caiconghui@xiaomi.com>
2021-04-06 10:59:08 +08:00
dce66c7329 [bug-fix] Fix cluster property in hadoop/broker load (#5587)
The cluster property has been removed from load stmt which is incorrect. It caused user could not create hadoop/broker load.
Fix #5586
2021-04-02 11:54:46 +08:00
0490b156c8 [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout (#5564)
* [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout

*This PR is to fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout and fix some small issues with log

Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
2021-03-30 19:50:42 +08:00
08bfd1c762 [Bug] Fix some problem with downloading and building thirdparty (#5439) 2021-03-30 10:47:45 +08:00
e2c702c959 remove some unused file for compatible to older kernels (#5568) 2021-03-29 11:32:59 +08:00
1ddaac7832 [Compaction] Optimize cumulative compaction point calculate when after restart be (#5561)
This pr is to reduce a large number of useless cumulative compactions caused by
cumulative compaction point calculation problems after be restarts.
2021-03-27 18:59:50 +08:00
a56e7e2192 [Refactor] make uint24_t,OLAPIndexFixedHeader as a POD type (#5559) 2021-03-27 18:59:23 +08:00
86af8c76a3 [DOC] Add docs of load and export using S3 protocol (#5551)
Add docs of load and export using S3 protocol
2021-03-27 18:58:29 +08:00
ce015b364a [Enhance] Support to show more details for 'SHOW TABLES' (#5520)
Support to show more details for 'SHOW TABLES', it's useful for thirdparty monitoring systems.
2021-03-27 18:58:14 +08:00
09879a2927 [DOC] Update gcc10 compile docs (#5566)
* update docs of gcc10

* update docs

* update docs
2021-03-26 09:34:21 +08:00
ad67dd34a0 update gcc to gcc 10 and support c++17 (#5394)
* update gcc to gcc 10 and support c++17
    update brpc to 0.9.7
    update boost to 1.73
    remove third-party boost 1.54 for mysql

* update cmake version

* ignore jdk version

* remove unused patch

* avoid use SYS_getrandom call
2021-03-25 09:30:38 +08:00
23bd4a959a remove some links that contians indecent content (#5560) 2021-03-24 11:56:49 +08:00
fa9e5c18dc [FE] Improve StreamLoadPlanner error log. (#5549) 2021-03-24 09:33:08 +08:00
bfeb717abe [Refactor] fix some warning in gcc higher than 7 make decimal12_t as a POD type (#5547) 2021-03-23 09:37:10 +08:00
c8afd5646c [Doc] Modify doc for replace table (#5536) 2021-03-23 09:36:11 +08:00
71b6e93424 Update dynamic-partition.md (#5535)
[Docs] Fix typo in docs zh-CN  administrator-guide dynamic-partition.md
2021-03-23 09:35:51 +08:00
cef3cbc53a [Bug] Fix bug that the last column may be null when using multibytes separator (#5534) 2021-03-23 09:35:30 +08:00
ae3e9df8c7 (#5532) RollupJobV2 support identify cluster when replay (#5533) 2021-03-23 09:34:51 +08:00
4e0775c3df [Docs] fix a mistake in docs basic usage (#5525) 2021-03-23 09:33:58 +08:00
3a96b6dfbd [Bug] Fix the bug of rowset file being deleted by mistake (#5541)
* [Bug] Fix the bug of rowset file being deleted by mistake
2021-03-23 09:33:36 +08:00
95f4f70fce remove unused char at end of file (#5545) 2021-03-22 15:44:34 +08:00
253056ca53 [Docs] Modify schema change sql reference (#5530)
Modify the error examples
2021-03-22 10:50:55 +08:00
4085b194d6 [Bug] Fix multi broker load bug (#5543)
The storage type of broker desc should be set due to the type of load job
2021-03-22 09:53:17 +08:00
c97ddd7f35 [Enhance] Show brokers' hostname (#5522)
Show brokers' hostname in table to keep the same with show proc for backends and frontends.
2021-03-21 20:46:25 +08:00
a91888a68b [BUG] fix memory limit failure and optimize memory usage in join stage (#5514)
This patch works well on tpcds-1T query-24
2021-03-21 11:32:51 +08:00
8343abaad1 [Feature] Local Exechange (#5470)
Avoid network transmission when the data stream sender node and destination exchange node
are in same BE, to improve performance and save CPU.
2021-03-21 11:25:33 +08:00
029a8a046b [Build] Turn on glibc compatibility by default for upgrading gcc10 (#5528) 2021-03-21 11:22:53 +08:00
6d27ae5a10 [Enhance] Support unescape some invisible char in separator (#5524) 2021-03-21 11:22:33 +08:00
97134c64e4 [Docker] Update Dockerfile using gcc 10 (#5380)
Update Dockerfile ,using gcc 10 as default compiler
2021-03-21 11:21:54 +08:00
19b3a950de [ODBC] change SQL_DRIVER_COMPLETE_REQUIRED to SQL_DRIVER_NOPROMPT make mysql connect err clear (#5538) 2021-03-21 11:20:25 +08:00
b9d92e0fcb [Profile] Visualize the query plan and query profile (#5475)
Add command:
1. EXPLAIN GRAPGH SELECT ...
2. SHOW QUERY PROFILE "..."

Document will be added in next PR

Change-Id: Ifd9365e10b1f9ff4fdf8ae0556343783d97545f0
2021-03-21 11:18:50 +08:00
5fa56e162b [Bug] Fix compaction failed when doing incremental clone (#5540) 2021-03-21 11:18:15 +08:00
a4bd553fd8 [Apache] Add DISCLAIMER back (#5531)
Also modify the year in NOTICE.txt
2021-03-18 11:24:19 +08:00
087fd8159b [Bug] Fix bug that the stale rowset file will not be deleted (#5527)
1. If cumulative compaction compact only one rowset, the old rowset will not be put into `stale_rowset_meta_map`

2. Show rowset id in `/api/compaction/show`

Co-authored-by: xxiao2018 <benghua3_1@sina.com>
2021-03-17 22:31:05 +08:00
f8e89bbbe3 [Improve] Support set multi variables in SET_VAR (#5513)
Sometimes we need to set multi variables for a single query stmt. like:

SELECT /* SER_VAR(var1=xx, var2=xx) */ ...
2021-03-17 09:27:45 +08:00
a1bce25677 [BUG] Fix Memory Leak in SchemaChange And Fix some DCHECK error (#5491) 2021-03-17 09:27:05 +08:00
105a86d1cd Use O_SYNC instead of O_DIRECT to be fs agnostic. (#5518)
Some fs might not support O_DIRECT and O_SYNC is semantically the same to be used for disk checking.
2021-03-15 10:07:51 +08:00
1100a0f3a0 [Profile] Add more timer for scan thread (#5511)
1.
Add timer to count the time the transfer thread waits for the scaner thread to return rowbatch.
2.
Add timer to count the time that the scanner thread waits for the available worker threads in the thread pool.

Co-authored-by: chenmingyu <chenmingyu@baidu.com>
2021-03-15 10:07:11 +08:00
4d36bf734a [Refactor] Make sure the singleton thread is safe (#5428)
The following class
org.apache.doris.clone.ColocateTableBalancer
org.apache.doris.common.proc.ProcService
org.apache.doris.rpc.BackendServiceProxy
org.apache.doris.common.util.ProfileManager
org.apache.doris.qe.HelpModule
org.apache.doris.common.publish.ClusterStatePublisher
is not safe in multiple thread environment. This PR is to implement a secure singleton mode.
The class org.apache.doris.common.publish.ClusterStatePublisher singleton method is not used.
2021-03-15 10:06:32 +08:00
c462231555 [build] support ccache while build BE (#5508) 2021-03-14 15:40:02 +08:00
4b316e4c3f [Outfile] Support exporting query result to local disk (#5489)
1.
User can export query result to local disk like:

`select * from tbl into outfile ("file:///disk1/result_");`

And modify the return result to show the details of export:

```
mysql> select * from tbl1 limit 10 into outfile "file:///home/work/path/result_";
+------------+-----------+----------+--------------+
| FileNumber | TotalRows | FileSize | URL          |
+------------+-----------+----------+--------------+
|          1 |         2 |        8 | 192.168.1.10 |
+------------+-----------+----------+--------------+
```

2.
Support create a mark file after export successfully finished.

Co-authored-by: chenmingyu <chenmingyu@baidu.com>
2021-03-14 15:39:46 +08:00
e9a73ee278 [Bug] Fix the memory expand 10~1000x of compression algorithm (#5504)
Fix the memory expand 10~1000x of compression algorithm in load and compaction
2021-03-12 23:04:07 +08:00