1. Print the last few tablets of decommission backend in fe.log for debug.
2. OlapTableSink should get replica on alive Backends, not only available Backends.
3. When decommission multi Backends, we should drop the redundant replicas before creating a new one.
4. Replicas on decommissioning Backends should be not added to catalog again.
5. Decommissioning Backends should not be chosen as destination of tablet repairing.
* Change label of broker load txn
1. put broker load label into txn label
2. fix the bug of `label is already used`
3. fix partition error of new broker load
* Fix count error in mini load and broker load
There are three params (num_rows_load_total, num_rows_load_filtered, num_rows_load_unselected) which are used to count dpp.norm.ALL and dpp.abnorm.ALL.
num_rows_load_total is the number rows of source file.
num_rows_load_unselected is the not satisfied (where conjuncts) rows of num_rows_load_total
num_rows_load_filtered is the rows (quality not good enough) of (num_rows_load_total-num_rows_load_unselected)
This change include the show load of loadv2 and some bug fix of loadv2.
Firstly, the show load will perform both load and loadv2 info. According to loadv2, the ETL progress of loadv2 is N/A during the period of loading.
Secondly, the loadv2 will be created when version of property is v2.
This is a temporary property which will not influence the old broker load.
After the loadv2 is finished, the default load will be changed to loadv2.
Finally, there are some bug in LoadingTaskPlanner fixed by this change.
* Enhence the usabilities
1. Add metrics to monitor transactions and steaming load process in BE.
2. Modify BE config 'result_buffer_cancelled_interval_time' to 300s.
3. Modify FE config 'enable_metric_calculator' to true.
4. Add more log for tracing broker load process.
5. Modify the query report process, to cancel query immediately if some instance failed.
* Fix bugs
1. Avoid NullPointer when enabling colocation join with broker load
2. Return immediately when pull load task coordinator execution failed
* Add new scheduler of load in fe
1. New scheduler only support the broker load now.
2. The stage of load consist of PENDING -> LOADING -> FINISHED
3. The LoadScheduler will divide job into a pending task. There are preparations that need to be done on pending task.
4. OnPendingTaskFinished will be invoked after pending task. It is used to submit the loading task which is created based on attachment of pending task.
5. OnLoadingTaskFinished will be invoked after loding task. It is used to record the commit info and commit txn when all of task has been finished.
.
* Combine pendingTask and loadingTask into loadTask
1. The load task callback include two methods: onTaskFinished, onTaskFailed
* Add txn callback of load job
1. isCommittting is true when beforeCommitted in txn
2. job could not be cancelled when isCommitting is true
3. job will be finished after txn is visible
4. old job will be cleaned when (CurrentTS - FinishedTs) / 1000 > Config.label_keep_seconds
5. LoadTimeoutChecker is performed to cancel timeout job
1. max io util of disks
2. max network send/receive bytes rate of all network devices
3. base/cumulative compaction request counter and failure counter
1. TaskScheduler will process one task per round
2. TaskScheduler will be blocked till queue tasks a new task
3. TaskScheduler will submit tasks when queue is empty
4. Add a example of creating a broker table by BOS
5. Change syntax of show routine load job
1. Add running/finished state proc to show specified state txns.
2. Add max disk used percent info in backends proc dir.
* add missing file
* fix bug
* Update fe/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
Co-Authored-By: morningman <morningman@163.com>
* Update fe/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
Co-Authored-By: morningman <morningman@163.com>
* Fix bug of listener
* Change txnStateChangeListener to txnStateChangeCallback
* Fix the logic of beforeAborted
1. It task is not belong to job, the txn attachment will be set to null.
* Txn will be abort normally without attachment.
* Job will not be updated by this task which attachment is null.
1. Add Config.max_routine_load_concurrent_task_num instead of the old one
2. Fix a bug that SHOW ALTER TABLE COLUMN may throw Nullpointer exception
3. Fix some misspelling of docs
