1.Support Bucket Shuffle Join when left table is colocate table or Colocate/Bucket Bucket Shuffle Join
2.Enable Local Rumtime Filter when there is Bucket Shuffle Join and Colocate Join
3.Add Doc for Bucket Shuffle Join
* [Bug] Fix bug that the image cannot be pulled after the new fe node is added
This is because httpv2 modified the response body of the "/info" api,
causing FE to fail to obtain info from this api.
And the system did not exit correctly.
This will also cause issues in issue #5292
from_unxitime is a cpu-exhausted function.
SQL: select filed from table where from_unixtime(field) > '2021-03-02',
if there are one million rows of data. Function from_unixtime will be called one million times,
which will make query very slow.
In issue #5443, we try to rewrite from_unixtime into timestamp to reduce calling this function.
This rewriting can bring 2 times query performance improvement.
Currently, FeMetaVersion.java is in fe-common, users may forget to copy fe-common.jar when upgrading the service.
It's really dangerous because the data may be corrupted and can not be recovered.
When there is count(*) function in query, we only need to scan the smallest column.
For example:
Query: select count(*) from (select k1, k2, k3 from base) tmp;
Only k1 which is the smallest column should be scanned.
The remaining columns (k2, k3) should be pruning.
This pr achieves this optimization of column pruning.
Fixed#5409
When a table has multiple partitions, each partition has it's own
version, the version doesn't represent whether it's newer or not. When a
partition has a large version, it may be considered as the largest one
currently, this will cause incorrect query result.
Suppose there are 2 partitions:
PartitionName | VisibleVersion | VisibleVersionTime
p1 | 123 | 2021-02-17 23:31:32
p2 | 23 | 2021-02-22 11:39:19
Partition p1 will be considered as the lastest partition, and there is a
cache before p2's last update time, the cache will hit and return an
error result.
* [doris-1008] support backup and restore directly to cloud storage via aws s3 protocol
* Internal][S3DirectAccess] Support backup,restore,load,export directlyconnect to s3
1. Support load and export data from/to s3 directly.
2. Add a config to auto convert broker access to s3 acces when available
Change-Id: Iac96d4b3670776708bc96a119ff491db8cb4cde7
(cherry picked from commit 2f03832ca52221cc7436069b96c45c48c4bc7201)
* [Internal][S3DirectAccess] File path glob compatible with broker
Change-Id: Ie55e07a547aa22c6fa8d432ca926216c10384e68
(cherry picked from commit d4fb25544c0dc06d23e1ada571ec3f8edd4ba56f)
* [internal] [doris-1008] fix log4j class not found
Change-Id: I468176aca0d821383c74ee658d461aba9e7d5be3
(cherry picked from commit 029adaa9d6ded8503acbd6644c1519456f3db232)
* add poms
Co-authored-by: yangzhengguo01 <yangzhengguo01@baidu.com>
If the difference between the two times exceeds Integer.MAX_VALUE,
the compare's return value will overflow and the flow exception may be triggered when sorting profile.
In the previous broker load, multiple OlapTableSinks would send data to the same LoadChannel,
and because of the lock granularity problem, LoadChannel could only process these requests serially,
which made it impossible to make full use of cluster resources.
This CL modifies the related locks so that LoadChannel can process these requests in parallel.
In the test, with a size of 20G, the load speed of 334 million rows of data in 3 nodes has been
increased from 9min to 5min, and after enabling 2 concurrency, it can be increased to 3min.
Also modify the profile of load job.
After an Alter job finished, the job's state is FINISHED, but table's state
may not be NORMAL for a while.
We need to make sure that table's state become NORMAL to continue next UT.
Support conditional filtering of original data in broker load and routine load
eg:
```
LOAD LABEL `label1`
(
DATA INFILE ('bos://cmy-repo/1.csv')
INTO TABLE tbl2
COLUMNS TERMINATED BY '\t'
(event_day, product_id, ocpc_stage, user_id)
SET (
ocpc_stage = ocpc_stage + 100
)
PRECEDING FILTER user_id = 1381035
WHERE ocpc_stage > 30
)
...
```
* [Bug] Fix bug of NPE when replaying spark load job.
The resourceDesc in spark load job may be null because it is not persisted.
So when replaying the job, we should check it.
* fix
* add ut
In our product environment, we use LVS to dispatch requests to FEs,
however, LVS will send probes to check whether FE is alive, and will
close the connection immediately. It will cause much verbose log,
this patch aim to reduce these log by catch related exceptions.
One Tablet have three replicas, the compaction progress is different.
Considering the following scenario, replica A have 3 versions(1, 2, 3);
replica B have 2 versions(1-2, 3);
replica C have 1 versions(1-3).
Now a column named city been added with default zero 1.
Replica A will be resulted as 3, replica B results as 2,
replica C results as 1.
So there is a necessity to restrict the default value to zero for SUM aggregation column.
The io related codes may be used by new modules, so It's better to move them to fe-common.
The modification to fe-core is frequent, but there are many generated java files by thrift
will slow down the compilation, so It's better to move thrift generation process to fe-common.
Currently both log4j1 and log4j2 are used, which leads to logs are written to wrong files.
Our modification will remove log4j1 from dependency, use slf4j + slf4j -> log4j2 instead.
This PR support following functions:
1. Support content properties in backup stmt. It means user can backup only metadata or
meta+data which use content [METADATA_ONLY| ALL]attribute to distinguish.
2. Support exclude some tables in backup and restore stmt. This means that some
very large and unimportant tables can be excluded when the entire database is backed up.
3. Support backup and restore whole database instead of declaring each table name
in the backup and restore statement.
The backup and restore api has changed as following:
```
BACKUP SNAPSHOT [db_name].{snapshot_name}
TO 'repo_name'
[ON|EXCLUDE (
'table_name' [partition (p1,...)]
)]
[properties (
"content" = "metadata_only|all"
)]
RESTORE SNAPSHOT [db_name].{snapshot_name}
TO 'repo_name'
[EXCLUDE|ON (
'table_name' [partition (p1,...)]
)]
[properties (
)]
```
Support delete statement like:
1. delete from table partitions(p1, p2) where xxx; // apply to p1, p2
2. delete from table where xxx; // apply to all partitions
Also remove code about the deprecated sync/async delete job.
This CL changes FE meta version to 94
[BackupAndRestore] Support backup and restore view and external odbc table
1. Support backup and restore view and odbc table. The syntax is the same as that of the backup and restore table.
2. If the table associated with the view does not exist in the snapshot,
the view can still be backed up successfully, but the TableNotFound exception will be thrown when querying the view.
3. If the odbc table associated with the odbc resource, the odbc resource will be backuped and restored together.
4. If the same view, odbc table and resource already exists in the database, it will compare whether the metadata of snapshot is consistent.
If it is inconsistent, the restoration will fail.
4. This pr also modified the json format of the backup information.
A `new_backup_objects` object is added to the root node to store backup meta-information other than olap table,
such as views and external tables.
```
{
"backup_objects": {},
"new_backup_objects": {
"view": [
{"name": "view1", "id": "10001"}
],
"odbc_table": [
{"name":"xxx", xxx}
]
"odbc_resources": [
{"name": "bj_oracle"}
]
}
}
```
5. This pr changes the serialization and deserialization method of backup information
from manual construction to automatic analysis by Gson tools.
Change-Id: I216469bf2a6484177185d8354dcca2dc19f653f3
