Commit Graph

2814 Commits

Author SHA1 Message Date
4e0775c3df [Docs] fix a mistake in docs basic usage (#5525) 2021-03-23 09:33:58 +08:00
3a96b6dfbd [Bug] Fix the bug of rowset file being deleted by mistake (#5541)
* [Bug] Fix the bug of rowset file being deleted by mistake
2021-03-23 09:33:36 +08:00
95f4f70fce remove unused char at end of file (#5545) 2021-03-22 15:44:34 +08:00
253056ca53 [Docs] Modify schema change sql reference (#5530)
Modify the error examples
2021-03-22 10:50:55 +08:00
4085b194d6 [Bug] Fix multi broker load bug (#5543)
The storage type of broker desc should be set due to the type of load job
2021-03-22 09:53:17 +08:00
c97ddd7f35 [Enhance] Show brokers' hostname (#5522)
Show brokers' hostname in table to keep the same with show proc for backends and frontends.
2021-03-21 20:46:25 +08:00
a91888a68b [BUG] fix memory limit failure and optimize memory usage in join stage (#5514)
This patch works well on tpcds-1T query-24
2021-03-21 11:32:51 +08:00
8343abaad1 [Feature] Local Exechange (#5470)
Avoid network transmission when the data stream sender node and destination exchange node
are in same BE, to improve performance and save CPU.
2021-03-21 11:25:33 +08:00
029a8a046b [Build] Turn on glibc compatibility by default for upgrading gcc10 (#5528) 2021-03-21 11:22:53 +08:00
6d27ae5a10 [Enhance] Support unescape some invisible char in separator (#5524) 2021-03-21 11:22:33 +08:00
97134c64e4 [Docker] Update Dockerfile using gcc 10 (#5380)
Update Dockerfile ,using gcc 10 as default compiler
2021-03-21 11:21:54 +08:00
19b3a950de [ODBC] change SQL_DRIVER_COMPLETE_REQUIRED to SQL_DRIVER_NOPROMPT make mysql connect err clear (#5538) 2021-03-21 11:20:25 +08:00
b9d92e0fcb [Profile] Visualize the query plan and query profile (#5475)
Add command:
1. EXPLAIN GRAPGH SELECT ...
2. SHOW QUERY PROFILE "..."

Document will be added in next PR

Change-Id: Ifd9365e10b1f9ff4fdf8ae0556343783d97545f0
2021-03-21 11:18:50 +08:00
5fa56e162b [Bug] Fix compaction failed when doing incremental clone (#5540) 2021-03-21 11:18:15 +08:00
a4bd553fd8 [Apache] Add DISCLAIMER back (#5531)
Also modify the year in NOTICE.txt
2021-03-18 11:24:19 +08:00
087fd8159b [Bug] Fix bug that the stale rowset file will not be deleted (#5527)
1. If cumulative compaction compact only one rowset, the old rowset will not be put into `stale_rowset_meta_map`

2. Show rowset id in `/api/compaction/show`

Co-authored-by: xxiao2018 <benghua3_1@sina.com>
2021-03-17 22:31:05 +08:00
f8e89bbbe3 [Improve] Support set multi variables in SET_VAR (#5513)
Sometimes we need to set multi variables for a single query stmt. like:

SELECT /* SER_VAR(var1=xx, var2=xx) */ ...
2021-03-17 09:27:45 +08:00
a1bce25677 [BUG] Fix Memory Leak in SchemaChange And Fix some DCHECK error (#5491) 2021-03-17 09:27:05 +08:00
105a86d1cd Use O_SYNC instead of O_DIRECT to be fs agnostic. (#5518)
Some fs might not support O_DIRECT and O_SYNC is semantically the same to be used for disk checking.
2021-03-15 10:07:51 +08:00
1100a0f3a0 [Profile] Add more timer for scan thread (#5511)
1.
Add timer to count the time the transfer thread waits for the scaner thread to return rowbatch.
2.
Add timer to count the time that the scanner thread waits for the available worker threads in the thread pool.

Co-authored-by: chenmingyu <chenmingyu@baidu.com>
2021-03-15 10:07:11 +08:00
4d36bf734a [Refactor] Make sure the singleton thread is safe (#5428)
The following class
org.apache.doris.clone.ColocateTableBalancer
org.apache.doris.common.proc.ProcService
org.apache.doris.rpc.BackendServiceProxy
org.apache.doris.common.util.ProfileManager
org.apache.doris.qe.HelpModule
org.apache.doris.common.publish.ClusterStatePublisher
is not safe in multiple thread environment. This PR is to implement a secure singleton mode.
The class org.apache.doris.common.publish.ClusterStatePublisher singleton method is not used.
2021-03-15 10:06:32 +08:00
c462231555 [build] support ccache while build BE (#5508) 2021-03-14 15:40:02 +08:00
4b316e4c3f [Outfile] Support exporting query result to local disk (#5489)
1.
User can export query result to local disk like:

`select * from tbl into outfile ("file:///disk1/result_");`

And modify the return result to show the details of export:

```
mysql> select * from tbl1 limit 10 into outfile "file:///home/work/path/result_";
+------------+-----------+----------+--------------+
| FileNumber | TotalRows | FileSize | URL          |
+------------+-----------+----------+--------------+
|          1 |         2 |        8 | 192.168.1.10 |
+------------+-----------+----------+--------------+
```

2.
Support create a mark file after export successfully finished.

Co-authored-by: chenmingyu <chenmingyu@baidu.com>
2021-03-14 15:39:46 +08:00
e9a73ee278 [Bug] Fix the memory expand 10~1000x of compression algorithm (#5504)
Fix the memory expand 10~1000x of compression algorithm in load and compaction
2021-03-12 23:04:07 +08:00
543ed46bc3 (#5390)fix NPE when replay colocate group (#5391)
* (#5390)fix NPE when replay colocate group

* remove table id from colocate group when duplicate create table

* remove tablet id when duplicate create table,just like ddlexception

* add ut
2021-03-12 14:24:34 +08:00
c9a25aa29e [UT] fix memory tracker ut (#5501)
* [UT] fix memory tracker ut

* Update mem_limit_test.cpp
2021-03-12 13:45:04 +08:00
8ead0aaad8 [Enhance] Sort directories by available space when do trash sweep (#5498)
* [Enhance] Sort directories by available space when do trash sweep

In the case when one disk is about to be full, we want to sweep trash
data on this disk as quickly as possible. The currently trash sweep
function is to remove trashed files order by path's name, however, disk
data directories may have some large different available space because
of the load balance algorithm, this patch improve it to remove files by
directories' available space.

* add log
2021-03-12 13:43:27 +08:00
511ddab3b1 Create how-to-contribute.md (#5506)
change ‘micro-signal to WeChat-ID’.
同时这个旧的PR#5477作废
2021-03-12 13:39:14 +08:00
9254e78e57 Fix compatibility of glibc (#5502)
* fix compatibility

* remove eventfd.c because eventfd in differene glibc has different declarition
2021-03-12 11:44:43 +08:00
315b801561 [Bug] Fix bug that the log message is wrong when ReportHandler can not find tablet meta from TabletInvertedIndex (#5497)
* [Bug] Fix bug that the log message is wrong when ReportHandler can not find tablet meta from TabletInvertedIndex

Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
2021-03-12 10:43:07 +08:00
64fa305c06 [Doc] correct format errors in English doc (#5487)
Some formate errors in English doc.
They are very straightforward and should not break any existing build.
2021-03-11 22:34:54 +08:00
689602e686 [Enhancement] Support Pallralel Merge In Exchange Node (#5468)
Support Parallel Merge In Exchange Node
2021-03-11 22:34:18 +08:00
0131c33966 [Enhance] Improve the readability of memtrackers' name (#5455)
Improve the readability of memtrackers' name, then you will be happy to read website be_ip:port/mem_tracker
2021-03-11 22:33:31 +08:00
e5c7a6dd9f [Bug] hll serialize 160 items cause backend crash(#5424) (#5425)
Co-authored-by: lanhuajian <lanhuajian@sankuai.com>
2021-03-11 22:24:01 +08:00
89bfbdf811 [Bug][Spark Load] Spark Repository throws an exception when user configures spark_ resource_ path (#5417) 2021-03-11 22:23:26 +08:00
6cbbc36ea1 [Export] Expand function of export stmt (#5445)
1. Support where clause in export stmt which only export selected rows.

The syntax is following:

Export table [table name]
    where [expr]
To xxx
xxxx

It will filter table rows.
Only rows that meet the where condition can be exported.

2. Support utf8 separator

3. Support export to local

The syntax is following:

Export table [table name]
To (file:///xxx/xx/xx)

If user export rows to local, the broker properties is not requried.
User only need to create a local folder to store data, and fill in the path of the folder starting with file://

Change-Id: Ib7e7ece5accb3e359a67310b0bf006d42cd3f6f5
2021-03-11 20:43:32 +08:00
7a8fbe5db8 [internal] [doris-1084] support compressed csv file in stream load (#5463) 2021-03-11 10:53:05 +08:00
a6046049d3 [DOCS] Add missing en doc Setting Up dev env for FE - IntelliJ IDEA (#5492) 2021-03-10 20:51:17 +08:00
ee91c047fa [GitHub] Fix PR Template (#5494) 2021-03-10 19:13:43 +08:00
079b764136 [Bug] Fix thread safe problem when check whether olap table is stable (#5478) 2021-03-10 19:12:00 +08:00
620de71052 Update basic-usage.md (#5493)
[Docs] Fix typo in docs zh-CN getting-started basic-usage.md
2021-03-10 18:36:56 +08:00
7aad412539 [MINOR] Remove offensive words (#5495) 2021-03-10 18:36:02 +08:00
b4c8e9de7f [Bug] Fix create view faild when StringLiteral in SelectStmt contains single quotes (#5484)
* [Bug] Fix create view faild when StringLiteral in SelectStmt contains single quotes

Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
2021-03-10 14:47:02 +08:00
36002bec48 Fix bug that sql with limit query statistics are wrong (#5347)
Co-authored-by: weixiang <weixiang06@meituan.com>

When querying sql is with limit, the query statistics will be enlarged by merging the query statistics of each batch.

Fix #5340
2021-03-10 10:24:12 +08:00
1d1a2569aa Update install-deploy.md (#5482)
[Docs] fix a mistake in docs zh-CN installing
2021-03-10 10:23:06 +08:00
bd53f407aa [Bucket Shuffle Join] Support the some featrue of Bucket Shuffle Join (#5459)
1.Support Bucket Shuffle Join when left table is colocate table or Colocate/Bucket Bucket Shuffle Join
2.Enable Local Rumtime Filter when there is Bucket Shuffle Join and Colocate Join
3.Add Doc for Bucket Shuffle Join
2021-03-09 14:47:59 +08:00
e023ef5404 [Load] Support multi bytes LineDelimiter and ColumnSeparator (#5462)
* [Internal][Support Multibytes Separator] doris-1079
support multi bytes LineDelimiter and ColumnSeparator
2021-03-09 09:35:39 +08:00
a1160bcd99 [Bug] Fix bug that data lost when doing backup job (#5473)
Backup job may delete the tablet of existing table, which cause data lost.
2021-03-08 09:32:16 +08:00
43dd583cfc Fix dlopen faild by upgrade cmake (#5481)
* fix dlopen faild

* remove useless code
2021-03-08 09:02:53 +08:00
db2120a7f2 [Build][BE] Fix GLIBC_COMPATIBILITY can not compile in centos6 (#5472)
Add option to disable glibc_compatibility
2021-03-07 20:47:13 +08:00