AlphaRowsetWriter validate rowset failed when build rowset
because rowset's num_rows is not equal to segment groups'
num_rows when add_rowset api is called. So add some log to
trace the process to debug the problems. The logs will be
deleted in the future.
FE uses partition_id to publish version. BE should check whether all tablets related with this partition have the version. But Tablet in BE does not have partition id in its metadata. So that BE could not check it.
This patch will add partition id to tablet meta during report task.
Sync at most 10k tablets during set tablet meta.
Currently, GRANT_PRIV can only be granted on global level, which means
it can only be granted on *.*. Grant it on db.* or db.tbl are not allowed.
This will not be able to meet the requirement to create a user who has privilege
to grant privileges to other users on specified database or table, such as:
GRANT SELECT_PRIV ON db1.* TO cmy@'%';
So I extend the range of GRANT_PRIV. User can now grant GRANT_PRIV on
database or even table level, such as:
GRANT GRANT_PRIV ON db1.* TO cmy@'%';
And after being granted, the user cmy@'%' can now grant GRANT_PRIV on db1.* to
other users.
Mini load is now using stream load framework. But we should keep the
mini load return behavior and result json format be same as old.
So PUBLISH_TIMEOUT error should be treated as OK in mini load.
Also add 2 counters for OlapTableSink profile:
SerializeBatchTime: time of serializing all row batch.
WaitInFlightPacketTime: time of waiting last send packet
Mini load timeout needs to be added in plan options.
The timeout property has been added in request of process put.
Otherwise, the timeout of mini load is useless.
Add log of label, txn and query id in mini load
NOTE: This patch would modify all Backend's data.
And this will cause a very long time to restart be.
So if you want to interferer your product environment,
you should upgrade backend one by one.
1. Refactoring be is to clarify the structure the codes.
2. Use unique id to indicate a rowset.
Nameing rowset with tablet_id and version will lead to
many conflicts among compaction, clone, restore.
3. Extract an rowset interface to encapsulate rowsets
with different format.
Currently, Doris only support UTF-8 encoded data. All data will be
shown to user in UTF-8 format. So if data loaded in Doris does not
UTF-8 encoded, user will see garbled data when querying.
I introduce a fast UTF-8 validator from
https://github.com/lemire/fastvalidate-utf-8
This validator is highly optimized that it only takes 0.7 CPU cycles
to validata a 64k string. And by testing 1GB data load to Doris, the
validator has no impact on performance.
The new class named 'AuthorizationInfo' is used to save the auth info in jobs.
The job doesn't need to retrieve the auth info by meta id which maybe throw the exception when db or table has been dropped or renamed.
The persistence of 'AuthorizationInfo' take effect in META_VERSION 56
This can happen if the Doris cluster is deployed with all, for example, SSD medium,
but create all tables with HDD storage medium property. Then getLoadScore(SSD) will
always return 0.0, so that no replica will be chosen when try to delete redundant
replicas.
Currently, the load error log on BE will be cleaned along with the
intermediate data of load, configured by 'load_data_reserve_hours'.
Sometimes user want to reserve the error log for longer time.
Before changing default insert operation to streaming load, if the select result
of a insert stmt is empty, a label will still be returned to the user, and user
can use this label to check the insert load job's status.
After changing the insert operation, if the select result is empty, a exception
will be thrown to user client directly without any label.
This new usage pattern is not friendly to already existed users, which is forcing
them to change their way of using insert operation.
So I add a new FE config 'using_old_load_usage_pattern', default is false.
If set to true, a label will be returned to user even if the select result is empty.
The issue is following:
Request1:
BE aborts the txn.
Attachment of txn is set.
Attachment of txn is set to null without lock of txn because the task has been aborted by FE.
Request2:
BE aborts the txn again.
Attachment of txn is set again.
Request1:
The attachment is not null so the job wants to find the task and commit it.
The job could not find the task so it is paused. (NullPointer Exception)
In this commit, The commit request will check if task exists instead of checking txn attachment
is not null.
Without dbId parameter, the backend report version can not be
updated when publish task report to FE, which may cause incorrect
order of report.
Related commit: 5c1b4f6
If one child of a binary predicate is column, and the other is a constant expr,
set the compatible type to column's type.
eg:
k1(int):
... WHERE k1 = "123" --> k1 = cast("123" as int);
k2(varchar):
... WHERE 123 = k2 --> cast(123 as varchar) = k2
This optimization is for the case that some users may use a int column to save date, eg: 20190703,
but query with predicate: col = "20190703".
If not casting "20190703" to int, query optimizer can not do partition prune correctly.
When a rollup table contains value columns of REPLACE aggregation type,
all key columns of base table should be included in this rollup.
Without all key columns, the order of rows is undefined.
For example, table schema k1,k2,k3,v1(REPLACE)
1,2,3,A
1,2,4,B
1,2,5,C
A rollup with column(k1,k2,v1):
1,2,A
1,2,B
1,2,C
No matter A or B or C be the last winner, it is meanless to user.
Also fix a bug that set password for current user on non-master FE
will throw NullPointerException.
This commit change the idToTable to concurrent hashmap in database. It don't need to lock the database before getTable.
The database will be locked in GlobalTxnManager. The load job will be locked after that.
So the lock sequence is: database, load manager, load job.
