The txn attachment maybe null when broker load has been cancelled without attachment.
The end log of broker load has been record but the callback id of txnState hasn't been removed
So the callback of txn is executed when log of txn aborted is replayed.
The reason for validate failure is the cloned file's names
may conflict and load segment read file througth cache and
cache key is file name, so index may read wrong file. The
solution is load index without use file handle cache.
The Operator wants to known when the job being scheduled as PENDING
and LOADING. And how long it takes to finish these sub states.
Also add 2 metrics on BE to monitor the memtable's flush time.
`memtable_flush_total` and `memtable_flush_duration_us`
Use RowwiseIterator to uniform all data fetch in storage engine.
All objects in storage engine can be read in iterator format.
For example: Segment, Rowset.
This patch implement two generic iterators: UnionRowwiseIterator,
MergeRowwiseIterator. These two class will add iterator as its inputs.
To implement iterators, we define a new class RowBlockV2, all data read
from iterator is in this format. We define a new class other than use
old version's RowBlock is because we want to keep old code work
normally.
When doing rollup, seek_columns equals to the complete set of tablet's columns.
There is no necessity to set it.
Related to commit 36df6ebe4e5f0abd3f07c1e454710590f1de23c7
The `_profile` in OlapTableSink may not be initialized if `prepare()`
method is not called. So when close the OlapTableSink, we should
check if `_profile` is initialized.
When query result reach limit, the Coordinator in FE will send a cancel
request to BE to cancel the query. And when being cancelled, BE will report
query status to FE for debug purpose. But actually it is not necessary
and will generate too many logs.
So I add a CancelReason to distinguish the difference between 'normally'
cancellation and 'internal error' cancellation. if 'normally' cancelled,
no status will be reported from BE.
When query reach limit, or user cancel it actively, it is being cancelled 'normally'.
Otherwise, the query is cancelled due to internal error, which will need
a report from BE.
The function of miniLoadBegin will return the txn_id.
If the backend sends the duplicated request to frontend, frontend will return the txn_id which was created by the same mini load.
The issue is that frontend returns the txn_id when the last same request hasn't been begun the txn.
The frontend returns the zero which is initialized txn_id and the be could not execute the load plan with a error txn_id.
The commit conbines the `createLoadJob` and `execute` together in the write lock. It protects the atomicity of `create` and `beginTxn`.
So the duplicated request cannot get the txn id before the last same request is finished.
AlphaRowsetWriter validate rowset failed when build rowset
because rowset's num_rows is not equal to segment groups'
num_rows when add_rowset api is called. So add some log to
trace the process to debug the problems. The logs will be
deleted in the future.
FE uses partition_id to publish version. BE should check whether all tablets related with this partition have the version. But Tablet in BE does not have partition id in its metadata. So that BE could not check it.
This patch will add partition id to tablet meta during report task.
Sync at most 10k tablets during set tablet meta.
Currently, GRANT_PRIV can only be granted on global level, which means
it can only be granted on *.*. Grant it on db.* or db.tbl are not allowed.
This will not be able to meet the requirement to create a user who has privilege
to grant privileges to other users on specified database or table, such as:
GRANT SELECT_PRIV ON db1.* TO cmy@'%';
So I extend the range of GRANT_PRIV. User can now grant GRANT_PRIV on
database or even table level, such as:
GRANT GRANT_PRIV ON db1.* TO cmy@'%';
And after being granted, the user cmy@'%' can now grant GRANT_PRIV on db1.* to
other users.
Mini load is now using stream load framework. But we should keep the
mini load return behavior and result json format be same as old.
So PUBLISH_TIMEOUT error should be treated as OK in mini load.
Also add 2 counters for OlapTableSink profile:
SerializeBatchTime: time of serializing all row batch.
WaitInFlightPacketTime: time of waiting last send packet
Mini load timeout needs to be added in plan options.
The timeout property has been added in request of process put.
Otherwise, the timeout of mini load is useless.
Add log of label, txn and query id in mini load
NOTE: This patch would modify all Backend's data.
And this will cause a very long time to restart be.
So if you want to interferer your product environment,
you should upgrade backend one by one.
1. Refactoring be is to clarify the structure the codes.
2. Use unique id to indicate a rowset.
Nameing rowset with tablet_id and version will lead to
many conflicts among compaction, clone, restore.
3. Extract an rowset interface to encapsulate rowsets
with different format.
Currently, Doris only support UTF-8 encoded data. All data will be
shown to user in UTF-8 format. So if data loaded in Doris does not
UTF-8 encoded, user will see garbled data when querying.
I introduce a fast UTF-8 validator from
https://github.com/lemire/fastvalidate-utf-8
This validator is highly optimized that it only takes 0.7 CPU cycles
to validata a 64k string. And by testing 1GB data load to Doris, the
validator has no impact on performance.
