1. Add a new dynamic partition property `create_history_partition`.
If set to true, Doris will create all partitions from `start` to `end`.
2. Add a new FE config `max_dynamic_partition_num`
To limit the number of partitions created when creating one table.
1. Fix NPE in ReplicasProcNode when backend does not exist
2. Forbid the create table like statement to specify the view.
3. Check self ip when starting FE to see if it use the origin ip.
4. Modify the error msg of tablet sink to show more detail errors.
In the previous code, the routine load task did not catch the exception of opening the transaction.
As a result, although the task cannot be executed,
no exceptions can be seen during show routine load, only the routine load job is stuck.
The PR catch the QuotaExceedException when opening a transaction.
If the routine load task cannot be executed due to the exhaustion of the quota,
the routine load will be paused and an error message will be presented to the user.
Similarly, other load method will also catch similar exceptions and cancel job.
1. Add /api/compaction/run_status to show the running compaction tasks.
2. Support do base and cumulative compaction for one tablet at same time.
3. Modify some log level.
4. Add a feedback document.
The previous replay logic does not record the size of the map, which eventually resulted in EOF when reading the log.
This pr replaces the replay logic directly with json.
At the same time, the replay logic of image is supplemented.
The pr ensure that the attributes 'lastStreamLoadTime' of backend can be correctly recorded in the image.
Although the table lock can control the simultaneous modification of the table by different threads.
But it cannot control the drop operation of the table by other threads.
For example, when drop table and table update occur at the same time.
1. get table object by thread 1
2. drop table by thread 2 with table lock
3. update table object by thread 1
The above process is possible.
At this time, step 3 actually operates a table that no longer exists, which will eventually cause the wrong metadata to be recorded.
Fixed#5687
This CL mainly changes:
1. Add a config to control the expire time of load job
Add a new FE config "streaming_label_keep_max_second" to control
the expire time of some high frequency load job such as INSERT and STREAM LOAD.
2. Remove expired txn in batch to avoid holding transaction lock for a long time
Record finished stream load job (both successful job and failed job) into audit log
so that we can see when the stream load job was executed and check the details of stream load jobs.
When replacing table with swap = false, the origin table's tablets
should be removed from tablet inverted index.
Co-authored-by: xxiao2018 <benghua3_1@sina.com>
The old colocate join can only cover the case where the child is hash or scan.
In fact, as long as the child's data distribution meets the requirements,
no matter what the plan node on the child node is, a colocate join can be performed.
1. The MaterializedViewSelector should be reset for each scan node
2. On the BE side, columns with delete conditions must be added to the return column.
* [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout
*This PR is to fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout and fix some small issues with log
Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
* update gcc to gcc 10 and support c++17
update brpc to 0.9.7
update boost to 1.73
remove third-party boost 1.54 for mysql
* update cmake version
* ignore jdk version
* remove unused patch
* avoid use SYS_getrandom call
Add command:
1. EXPLAIN GRAPGH SELECT ...
2. SHOW QUERY PROFILE "..."
Document will be added in next PR
Change-Id: Ifd9365e10b1f9ff4fdf8ae0556343783d97545f0
1. If cumulative compaction compact only one rowset, the old rowset will not be put into `stale_rowset_meta_map`
2. Show rowset id in `/api/compaction/show`
Co-authored-by: xxiao2018 <benghua3_1@sina.com>
The following class
org.apache.doris.clone.ColocateTableBalancer
org.apache.doris.common.proc.ProcService
org.apache.doris.rpc.BackendServiceProxy
org.apache.doris.common.util.ProfileManager
org.apache.doris.qe.HelpModule
org.apache.doris.common.publish.ClusterStatePublisher
is not safe in multiple thread environment. This PR is to implement a secure singleton mode.
The class org.apache.doris.common.publish.ClusterStatePublisher singleton method is not used.
