1. Add a new dynamic partition property `create_history_partition`.
If set to true, Doris will create all partitions from `start` to `end`.
2. Add a new FE config `max_dynamic_partition_num`
To limit the number of partitions created when creating one table.
1. Fix NPE in ReplicasProcNode when backend does not exist
2. Forbid the create table like statement to specify the view.
3. Check self ip when starting FE to see if it use the origin ip.
4. Modify the error msg of tablet sink to show more detail errors.
In the previous code, the routine load task did not catch the exception of opening the transaction.
As a result, although the task cannot be executed,
no exceptions can be seen during show routine load, only the routine load job is stuck.
The PR catch the QuotaExceedException when opening a transaction.
If the routine load task cannot be executed due to the exhaustion of the quota,
the routine load will be paused and an error message will be presented to the user.
Similarly, other load method will also catch similar exceptions and cancel job.
1. Add /api/compaction/run_status to show the running compaction tasks.
2. Support do base and cumulative compaction for one tablet at same time.
3. Modify some log level.
4. Add a feedback document.
The distinct count result of bitmap/hll column may be incorrect in the spark load mode.
Fix some bugs in spark load to solve the above problem.
1. FE is big end but BE is little end. BitmapValues should be transfered to little end in FE's serialization
2. BitmapUnionAggregator/HllUnionAggregator ignore `null` value
3. Make sure encodeVarint64 in FE is consistent with BE
Co-authored-by: weixiang <weixiang06@meituan.com>
The previous replay logic does not record the size of the map, which eventually resulted in EOF when reading the log.
This pr replaces the replay logic directly with json.
At the same time, the replay logic of image is supplemented.
The pr ensure that the attributes 'lastStreamLoadTime' of backend can be correctly recorded in the image.
Although the table lock can control the simultaneous modification of the table by different threads.
But it cannot control the drop operation of the table by other threads.
For example, when drop table and table update occur at the same time.
1. get table object by thread 1
2. drop table by thread 2 with table lock
3. update table object by thread 1
The above process is possible.
At this time, step 3 actually operates a table that no longer exists, which will eventually cause the wrong metadata to be recorded.
Fixed#5687
This CL mainly changes:
1. Add a config to control the expire time of load job
Add a new FE config "streaming_label_keep_max_second" to control
the expire time of some high frequency load job such as INSERT and STREAM LOAD.
2. Remove expired txn in batch to avoid holding transaction lock for a long time
Record finished stream load job (both successful job and failed job) into audit log
so that we can see when the stream load job was executed and check the details of stream load jobs.
When replacing table with swap = false, the origin table's tablets
should be removed from tablet inverted index.
Co-authored-by: xxiao2018 <benghua3_1@sina.com>
The old colocate join can only cover the case where the child is hash or scan.
In fact, as long as the child's data distribution meets the requirements,
no matter what the plan node on the child node is, a colocate join can be performed.
1. The MaterializedViewSelector should be reset for each scan node
2. On the BE side, columns with delete conditions must be added to the return column.
Fix CVE-2017-5645
In Apache Log4j 2.x before 2.8.2, when using the
TCP socket server or UDP socket server to receive
serialized log events from another application, a
specially crafted binary payload can be sent that,
when deserialized, can execute arbitrary code.
https://www.cvedetails.com/cve/CVE-2017-5645/
* [Bug] Fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout
*This PR is to fix bug that call frontend service failed when rpc_timeout is equal with publish_timeout and fix some small issues with log
Co-authored-by: caiconghui [蔡聪辉] <caiconghui@xiaomi.com>
* update gcc to gcc 10 and support c++17
update brpc to 0.9.7
update boost to 1.73
remove third-party boost 1.54 for mysql
* update cmake version
* ignore jdk version
* remove unused patch
* avoid use SYS_getrandom call
