in some cases:
s3.session_token/AWS_TOKEN will be expired after hours, and may be upload snapshot job will failed if data is big large;
as same reason, repo will be expired too when RepositoryMgr exec repo ping
so it need support alter s3 repo properties about ak/sk/token and update upload snapshot job properties to continue backup.
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling，Env of replay thread also can not be recycling
1. Fix data wrong using mv rewrite
2. Ignore case when getting mv related partition table
3. Enable infer expression column name without alias when create mv
If we specific target partition(s) when inserting overwrite an auto partition table,
before:
could create new partition
now:
behalf just like non-auto partition table
Drop stats use IN predicate to filter the column stats to delete. The default length of IN predicate is 1024, drop table stats with more than 1024 columns may fail.
This pr is to split the delete sql based on the IN predicate length.
- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite
Since scheduling itself consumes a certain amount of time, the start time of the time window should not be the current time, but the end time of the last schedule.
1. Use consistent hash algo as the default assigner for file query scan node
A consistent assignment can better utilize the page cache of BE node.
2. Cache the consistent hash ring
Init a consistent hash ring is time-consuming because there a thousands of virtual node need to be added.
So cache it for better performance
When string is too long, clucene will throw an error.
And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default.
We add an poperty `ignore_above` for user to customize.
- create/alter mtmv support refresh_partition_num
- mtmv task according to refresh_partition_num executes refresh tasks in batches
- `tasks` tvf add column `CompletedPartitions` and `progress`
- fix mtmv can not `show temp partition` and `drop temp partition`
- fix task can not get error msg when insert overwrite error
- fix when the partition field is capitalized, the verification of creating a mtmv does not pass
In some cases, users need to get the data size of single replica of a table, and evaluate certain actions based on this, such as estimating the precise backup size.
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
Add a new FE Config `sql_convertor_service`.
If this config is set, and the session variable `sql_dialect` is set,
Doris will try to use a standalone sql converter service to convert user input sql to
specified sql dialect. eg:
```
mysql> set sql_dialect="presto";
Query OK, 0 rows affected (0.02 sec)
Database changed
mysql> select * from db1.tbl1 where "k1" = 1; # will be converted to select * from db1.tbl1 where `k1` = 1;
+------+------+
| k1 | k2 |
+------+------+
| 1 | 2 |
+------+------+
1 row in set (0.08 sec)
```
The sql converter service should be a http service.
The request and response body can be found in `SQLDialectUtils.java`
Introduced from #27861
The `dbName` saved in `CreateTableInfo` has `default_cluster` prefix, it should be removed.
Also modify the entry of `getDb` in internal catalog. This is a cover-up plan in case there may still
db name exist with `default_cluster` prefix.
