* Fix bug that <=> operator and in operator get wrong result
* Add some comment to get_result_for_null
* Add an new Binary Operator to replace is_safe_for_null for handleing '<=>' operator
* Add EQ_FOR_NULL to TExprOpcode
* Remove macro definition last backslash
Use same UUID as query ID and load ID of a load execution plan.
Each load execution plan has a load ID, and as a plan, there is also a query ID.
We can use same UUID as query ID and load ID, for tracing the load process more easily.
Change the load ID when retrying a load execution plan.
When a load execution plan retry, the load ID should be changed, otherwise BE can not
distinguish the old and new load requests.
Cancel the running loading task when cancelling the broker load.
When user cancel a broker load, the running loading task should also be cancelled, or
it may occupies the worker thread for a long time.
Remove the unnecessary query report when doing load execution plan.
Only the last query report is needed.
Add a new BE config tablet_writer_rpc_timeout_sec.
It is used for RPC of tablet sink. The default is 600 seconds. which is long enough for flushing
about 6GB data. The long timeout config will reduce the possibility of encountering fail to send batch error when loading.
Use streaming_load_max_mb instead of mini_load_max_mb in BE config.
Add more logs for tracing a broker load process easily.
The default value of null is forbidden in insert into stmt while null column has not been mentioned in stmt.
This is a bug because the unmentioned column has default value. The values should be inserted successfully although the default value is null.
So the column may simply be not assigned default value when the column is not allowed null and the default value of column is null.
Add dict encoding page for binary/string type data.
Construct a dict for original data, and save encoded id instead of
origin data to save space. If the dict is too big, then will automatically fall
back to plain encoding.
The catalogue of load docs:
---- load-manual.md
---- broker-load-manual.md
---- insert-into-manual.md
---- stream-load-manual.md
This commit also changes max/min_stream_load_timeout to max/min_load_timeout.
The old config named stream_load_timeout means the max timeout suited for all types of load.
So the config name has been changed.
The txn attachment maybe null when broker load has been cancelled without attachment.
The end log of broker load has been record but the callback id of txnState hasn't been removed
So the callback of txn is executed when log of txn aborted is replayed.
The reason for validate failure is the cloned file's names
may conflict and load segment read file througth cache and
cache key is file name, so index may read wrong file. The
solution is load index without use file handle cache.
The Operator wants to known when the job being scheduled as PENDING
and LOADING. And how long it takes to finish these sub states.
Also add 2 metrics on BE to monitor the memtable's flush time.
`memtable_flush_total` and `memtable_flush_duration_us`
Use RowwiseIterator to uniform all data fetch in storage engine.
All objects in storage engine can be read in iterator format.
For example: Segment, Rowset.
This patch implement two generic iterators: UnionRowwiseIterator,
MergeRowwiseIterator. These two class will add iterator as its inputs.
To implement iterators, we define a new class RowBlockV2, all data read
from iterator is in this format. We define a new class other than use
old version's RowBlock is because we want to keep old code work
normally.
When doing rollup, seek_columns equals to the complete set of tablet's columns.
There is no necessity to set it.
Related to commit 36df6ebe4e5f0abd3f07c1e454710590f1de23c7
The `_profile` in OlapTableSink may not be initialized if `prepare()`
method is not called. So when close the OlapTableSink, we should
check if `_profile` is initialized.
When query result reach limit, the Coordinator in FE will send a cancel
request to BE to cancel the query. And when being cancelled, BE will report
query status to FE for debug purpose. But actually it is not necessary
and will generate too many logs.
So I add a CancelReason to distinguish the difference between 'normally'
cancellation and 'internal error' cancellation. if 'normally' cancelled,
no status will be reported from BE.
When query reach limit, or user cancel it actively, it is being cancelled 'normally'.
Otherwise, the query is cancelled due to internal error, which will need
a report from BE.
The function of miniLoadBegin will return the txn_id.
If the backend sends the duplicated request to frontend, frontend will return the txn_id which was created by the same mini load.
The issue is that frontend returns the txn_id when the last same request hasn't been begun the txn.
The frontend returns the zero which is initialized txn_id and the be could not execute the load plan with a error txn_id.
The commit conbines the `createLoadJob` and `execute` together in the write lock. It protects the atomicity of `create` and `beginTxn`.
So the duplicated request cannot get the txn id before the last same request is finished.
AlphaRowsetWriter validate rowset failed when build rowset
because rowset's num_rows is not equal to segment groups'
num_rows when add_rowset api is called. So add some log to
trace the process to debug the problems. The logs will be
deleted in the future.
FE uses partition_id to publish version. BE should check whether all tablets related with this partition have the version. But Tablet in BE does not have partition id in its metadata. So that BE could not check it.
This patch will add partition id to tablet meta during report task.
Sync at most 10k tablets during set tablet meta.
Currently, GRANT_PRIV can only be granted on global level, which means
it can only be granted on *.*. Grant it on db.* or db.tbl are not allowed.
This will not be able to meet the requirement to create a user who has privilege
to grant privileges to other users on specified database or table, such as:
GRANT SELECT_PRIV ON db1.* TO cmy@'%';
So I extend the range of GRANT_PRIV. User can now grant GRANT_PRIV on
database or even table level, such as:
GRANT GRANT_PRIV ON db1.* TO cmy@'%';
And after being granted, the user cmy@'%' can now grant GRANT_PRIV on db1.* to
other users.
Mini load is now using stream load framework. But we should keep the
mini load return behavior and result json format be same as old.
So PUBLISH_TIMEOUT error should be treated as OK in mini load.
Also add 2 counters for OlapTableSink profile:
SerializeBatchTime: time of serializing all row batch.
WaitInFlightPacketTime: time of waiting last send packet
Mini load timeout needs to be added in plan options.
The timeout property has been added in request of process put.
Otherwise, the timeout of mini load is useless.
Add log of label, txn and query id in mini load
NOTE: This patch would modify all Backend's data.
And this will cause a very long time to restart be.
So if you want to interferer your product environment,
you should upgrade backend one by one.
1. Refactoring be is to clarify the structure the codes.
2. Use unique id to indicate a rowset.
Nameing rowset with tablet_id and version will lead to
many conflicts among compaction, clone, restore.
3. Extract an rowset interface to encapsulate rowsets
with different format.
