In the original tablet reporting information, the version missing information is done by combining
two pieces of information as follows:
1. the maximum consecutive version number
2. the `version_miss` field
The logic of this approach is confusing and inconsistent with the logic of checking for missing versions when querying.
After the change, we directly use the version checking logic used in the query, and set `version_miss` to true
if a missing version is found
and on the FE processing side. Originally, only the **bad replica** information was syncronized among FEs,
but not the **version missing** information. As a result, the non-master FE is not aware of the missing version information.
In the new design, we deprecate the original log persistence class `BackendTabletsInfo` and use the new
`BackendReplicasInfo` to record replica reporting information and write both **bad** and **version missing**
information to metadata so that other FEs can synchronize these information.
1.
Before executing LoadLoadingTask of a broker load, we should check if the job is cancelled.
2.
Add a new column `runningTransactionNum` for `show proc "/transactions"`.
So that we can view all running txns in each db in one command.
1. support SHOW LAST INSERT
In the current implementation, the insert operation returns a json string to describe the result information
of the insert. But this information is in the session track field of the mysql protocol,
and it is difficult to obtain programmatically.
Therefore, I provide a new syntax `show last insert` to explicitly obtain the result of the latest insert operation,
and return a normal query result set to facilitate the user to obtain the result information of the insert.
2. the `ReturnRows` field in fe.audit.log of insert operation will be set to the loaded row num of the insert.
3. Fix a bug described in #8354
SQL to reproduce:
```
SELECT * FROM table WHERE where FROM_UNIXTIME(d_datekey,'%Y-%m-%d %H:%i:%s') != '1970-08-20 00:11:43';
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: Illegal pattern character 'i'
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:584) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:345) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:318) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:221) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:361) ~[palo-fe.jar:3.4.0]
at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:562) ~[palo-fe.jar:3.4.0]
at org.apache.doris.mysql.nio.ReadListener.lambda$handleEvent$0(ReadListener.java:50) ~[palo-fe.jar:3.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
```
Describe the overview of changes.
Just support:
yyyy-MM-dd HH:mm:ss
yyyy-MM-dd
yyyyMMdd
1. mysql-connector-java
mysql-connector-java is under GLPv2 license, which is not compatible with APLv2, and Doris does not use it.
2. org.json
org.json is under JSON license, which is not compatible with APLv2. I use `json-simple` to replace it.
1. optimize compaction too slow replica process, will set to ok if the replica compaction is done.
And will not set bad if more than half replica is too slow,
2. fix field in grouping_id() is not the same name will cause error lile `select list expression not produced by
aggregation output (missing from GROUP BY clause?): id`
* [refactor] remove types_test
1. remove types_test, it will cause core dump in higher version GCC or
clang, because of memory align, some code will be vectorized in higher
GCC or clang
2. Change string type length to 2 GB instead of -1
3. modify inaccessible code
This reverts commit df7e848cbbc8170c7bd83d812d7cac58b5574570.
Reverts apache/incubator-doris#8218
Because when using grpc 1.44.1, the corresponding `protoc-gen-grpc-java` plugin
requried GLIBC_2.14, which is not found in CentOS 6.
So I suggest to revert this commit this time. And considering upgrading this component
after most systems have reached glibc version 2.14.
And for Mac M1, you may have to change this version manually for now
This bug is introduced from #8112.
Also , I change the `grpc-netty` dependency to `grpc-netty-shaded`, to avoid dependency conflict:
```
java.lang.NoSuchMethodError: io.netty.buffer.PooledByteBufAllocator.
```
In some scenarios, users cannot find a suitable hash key to avoid data skew, so we need to provide an additional data distribution for olap table to avoid data skew
example:
CREATE TABLE random_table
(
siteid INT DEFAULT '10',
citycode SMALLINT,
username VARCHAR(32) DEFAULT '',
pv BIGINT SUM DEFAULT '0'
)
AGGREGATE KEY(siteid, citycode, username)
DISTRIBUTED BY random BUCKETS 10
PROPERTIES("replication_num" = "1");
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
There are many old codes in FE for old FE meta version such as `if (FeMetaVersion < VERSION_45) xxxxx`,
but the latest FE meta version is 107, these code maybe never reached,
but we do not remove these code because "sometimes" there are old code.
Add minimum required version check to allow us remove these old codes.
This Bug is introduced by PR #7936 , which change key type of connectionMap from Long to Integer,
which cause connectionMap could not find connectContext by connectionId
Enable to check the Java version when Doris starts, to prevent the user experience caused by the inconsistency
between the compiled version and the running version.
If the Java version is compiled and the Java version is run, it will not start, and a prompt message will be given.
1. Fix the problem of BE crash caused by destruct sequence. (close#8058)
2. Add a new BE config `compaction_task_num_per_fast_disk`
This config specify the max concurrent compaction task num on fast disk(typically .SSD).
So that for high speed disk, we can execute more compaction task at same time,
to compact the data as soon as possible
3. Avoid frequent selection of unqualified tablet to perform compaction.
4. Modify some log level to reduce the log size of BE.
5. Modify some clone logic to handle error correctly.
Hive Bitmap UDF provides UDFs for generating bitmap and bitmap operations in hive tables.
The bitmap in Hive is exactly the same as the Doris bitmap.
The bitmap in Hive can be imported into Doris through spark bitmap load.
