1. insert very large string value may coredump
2. some analitic functiuon and agg function result may be incorrect
3. string compare may be coredump when string type is too large
4. string type in delete condition can not process correctly
5. add text/blob as alias of string to compitable with mysql
6. fix string type min/max agg may process incorrectly
There are many historical job records in Doris, such as load jobs, alter jobs, export jobs and so on.
These historical jobs are generally cleaned up periodically by the cleanup thread, to avoid taking too much memory.
This PR reorganized the cleanup logic of historical jobs and optimized the cleanup logic of some historical jobs
to reduce the memory usage of historical jobs.
The following FE configuration items are related to historical job cleaning:
1. label_keep_max_second
Used to determine whether LoadJob, LoadJobV2, RoutineLoadJob or TransactionState are expired.
2. streaming_label_keep_max_second
Used to determine whether InsertJob, DeleteJob or TransactionState are expired.
Different from label_keep_max_second, this config is used to clean up these frequently submitted jobs or load transactions.
3. history_job_keep_max_second
Used to determine whether AlterJob, ExportJob are expired
The basic idea is adding a rewrite rule to rewriete the expr of `SlotRef`,
which type is fixedPointValue, aka Integer-Types, to a `CastExpr`, it will cast the fixedPoint value to string.
Fix bug that if a tablet belongs to a colocation table, and one of its
replica is in DECOMMISSION state. This tablet can not be repaired.
Also fix a bug that quota does not escape in show create table result.
```
COMMENT "a"bc" to COMMENT "a\"bc"
```
Fix#6551
`Truncate` operation will create new partition to replace base partition.
Tablets in new partition would be created based on table default bucket number.
If the number of tablet for a partition is different from table default bucket number,
the number of tablet will be different between the new created partition and base partition after truncate operation.
In any case, the new partition should have the same number of tablet as the base partition after truncate operation.
Tablets in new partition should be created based on tablet number of base partition rather than table default bucket number for truncate operation.
This pr mainly supports
1. Export query result sets concurrently
2. Query result set export supports s3 protocol
Among them, there are several preconditions for concurrently exporting query result sets
1. Enable concurrent export variables
2. The query itself can be exported concurrently
(some queries containing sort nodes at the top level cannot be exported concurrently)
3. Export the s3 protocol used instead of the broker
After exporting the result set concurrently,
the file prefix is changed to outfile_{query_instance_id}_filenumber.{file_format}
Counter of image write success or failure:
image_write{"type" = "success"}
image_write{"type" = "failed"}
Counter of image push to other FE nodes success or failure:
image_push{"type" = "success"}
image_push{"type" = "failed"}
Counter of old image clean success or failure:
image_clean{"type" = "success"}
image_clean{"type" = "failed"}
Counter of old edit log clean success or failure:
edit_log_clean{"type" = "success"}
edit_log_clean{"type" = "failed"}
add a rewrite rule of compoundPredicate 'OR' 'AND' to hit prefix index
unit test added.
`````
case true AND expr ==> expr
case expr AND true ==> expr
case false Or expr ==> expr
case expr Or false ==> expr
case false AND expr ==> false
case expr AND false ==> false
case true Or expr ==> true
case expr Or true ==> true
`````
When no backend(be) available, eg first time before setup or all be is down, we cannot login to fe using most sql tools, eg datagrip, querious.
This is because these tools call `select version()` or `select @@version_comment …` right after login, when there is no backend available, the login will fail.
I make this pr to support login when no be available, so we can add backend or modify some configuaration using GUI sql tools, especially the first setup time.
This pr is working at precondition that，sql tools only query very simple information that fe can handle, there is no need to send the request to be.
So I check the query type and BE status, if we can handle the query and no BE is available, we will intercept and process in fe.
Redirect the following http requests to master:
/rest/v2/api/cluster_overview
/rest/v2/manager/node/frontends
/rest/v2/manager/node/backends
/rest/v2/manager/node/brokers
If the colocate group of the table is not stable,
then even if it is a colocate table,
its data distribution is Random.
So the distribution info of OlapScanNode is also Random instead of Hash Partition.
Fixed#6558
#5902
This CL mainly changes:
1. Support setting tags for BE nodes:
```
alter system add backend "1272:9050, 1212:9050" properties("tag.location": "zoneA");
alter system modify backend "1272:9050, 1212:9050" set ("tag.location": "zoneB");
```
And for compatibility, all BE nodes will be set a "default" tag when upgrading: `"tag.location": "default"`.
2. Create a new class `ReplicaAllocation` to replace the previous `replication_num`.
`ReplicaAllocation` represents the allocation of the replicas of a tablet. It contains a map from
Tag to number of replicas.
For example, if user set a table's replication num to 3, it will be converted to a ReplicaAllocation
like: `"tag.location.default" : "3"`, which means the tablet will have 3 replicas and all of them will be
allocated in BE nodes with tag "default";
3. Support create table with replication allocation:
```
CREATE TABLE example_db.table_hash
(
k1 TINYINT
)
DISTRIBUTED BY HASH(k1) BUCKETS 32
PROPERTIES (
"replication_allocation"="tag.location.zone1:1, tag.location.zone2:2"
);
```
Also support set replica allocation for dynamic tables, and modify replica allocation at runtime.
For compatibility, user can still set "replication_num" = "3", and it will be automatically converted to:
` "replication_allocation"="tag.location.default:3"`
4. Support tablet repair and balance based on Tag
1. For tablets of non-colocate table, most of the logic is the same as before,
but when selecting the destination node for clone, the tag of the node will be considered.
If the required tag does not exist, it cannot be repaired.
Similarly, under the condition of ensuring that the replicas are complete, the tablet will be
reallocated according to the tag or the replicas will be balanced.
Balancing is performed separately within each resource group.
2. For tablets of colocate table, the backends sequence of buckets will be splitted by tag.
For example, if replica allocation is "tag.location.zone1:1, tag.location.zone2:2",
And zone1 has 2 BE: A, B; zone2 has 3 BE: C, D, F
there will be 2 backend sequences: one is for zone1, and the other is for zone2.
And one posible seqeunces will be:
zone1: [A] [B] [A] [B]
zone2: [C, D][D, F][F, C][C, D]
5. Support setting tags for user and restrict execution node with tags:
```
set property for 'cmy' 'resource_tags.location' : 'zone1, zone2';
```
After setting, the user 'cmy' can only query data stored on backends with tag zone1 and zone2,
And query can only be executed on backends with tag zone1 and zone2
For compatibility, after upgrading, the property `resource_tags.location` will be empty,
so that user can still query data stored on any backends.
6. Modify the Unit test frame of FE so that we can created multi backends with different mocked IP in unit test.
This help us to easily test some distributed cases like query, tablet repair and balance
The document will be added in another PR.
Also fix a bug described in #6194
fix#5378#5391#5688#5973#6155 and all replay NPE. All replay method can now throw MetaNotFoundException and caught to log a warning for potential inconsistent metadata cases.
try to establish a clear notice for future developer to check null.
* fix(sparkload): bitmap deep copy in `or` operator
fix multi rollup hold the same Ref of bitmapvalue which may be updated repeatedly.
* fix(sparkload): bitmap deep copy in `or` operator
fix multi rollup hold the same Ref of bitmapvalue which may be updated repeatedly.
Co-authored-by: weixiang <weixiang06@meituan.com>
* fix bugs with string type
1. not support string with agg type min/max
2. agg_update with large string may coredump
3. stringval with large string may coredump
4. not support string as partition key
Origin stream load column order transformation is unclear , a user is struggling for a long time in this part ,so i modified some expressions to make it clearer.
for issue #6474
```sql
create table test.table1 like test.table with rollup r1,r2 -- copy some rollup
create table test.table1 like test.table with rollup all -- copy all rollup
create table test.table1 like test.table -- only copy base table
```
Fix#6512
If there is missing replica for a tablet, clone task will be executed to restore missing replica from a healthy replica. Src replica selector will randomly choose a healthy replica as src replica.
It's better to choose the health replica with min version count as src replica so that it could avoid repetitive compaction task. In addition, replica with less version count is good for query performance.
fix#6447
1. FE master regularly triggers the remove operation
2. After the master completes the removal of deleteInfo, it is synchronized to the Follower through editlog for remove
3. When the DeleteInfo creation time is longer than the current time, it will be cleaned up, which is determined by the `delete_info_keep_max_second` configuration
Implement the lower_case_table_names variable of mysql. The value meaning is as follows:
0: the table names are case-sensitive.
1: table names are stored in lowercase and comparisons are not case sensitive.
2: table names are stored as given but compared case-insensitively.
1. Add license/total line/release badegs.
2. Add monthly active contributor and contributor growth graph
3. fix a pom.xml bug
4. Modify some routine load log on BE side
