This CL mainly changes:
1. Add 2 new FE modules
1. fe-common
save all common classes for other modules, currently only `jmockit`
2. spark-dpp
The Spark DPP application for Spark Load. And I removed all dpp related classes to this module, including unit tests.
2. Change the `build.sh`
Add a new param `--spark-dpp` to compile the `spark-dpp` alone. And `--fe` will compile all FE modules.
the output of `spark-dpp` module is `spark-dpp-1.0.0-jar-with-dependencies.jar`, and it will be installed to `output/fe/spark-dpp/`.
3. Modify some bugs of spark load
This PR is to support grammar like the following: INSTALL PLUGIN FROM [source] [PROPERTIES("KEY"="VALUE", ...)]
user can set md5sum="xxxxxxx", so we don't need to provide a md5 uri.
When Fe uploads the spark archive, the broker may fail to write the file,
resulting in the bad file being uploaded to the repository.
Therefore, in order to prevent spark from reading bad files, we need to
divide the upload into two steps.
The first step is to upload the file, and the second step is to rename the file with MD5 value.
This PR is to ensure that dropped db , table or partition can be with normal state after recovered by user. Commited txns can not be aborted, because the partitions's commited versions have been changed, and some tablets may already have new visible versions. If user just don't want the meta(db, table or partition) anymore, just use drop force instead of drop to skip committed txn check.
This PR is to use LongAdder or volatile long to replace AtomicLong in some scenarios.
In the statistical summation scenario, LongAdder(introduced by jdk1.8) has better performance than AtomicLong in high concurrency update scenario. And if we just want to keep get and set operation for variable to be atomic, just add volatile at the front of the variable is enough, use AtomicLong is a little heavy.
NOTE: LongAdder is usually preferable to AtomicLong when multiple threads update a common sum that is used for purposes such as collecting statistics, not for fine-grained synchronization control, such as auto-incremental ids.
### Resume
When users use spark load, they have to upload the dependent jars to hdfs every time.
This cl will add a self-generated repository under working_dir folder in hdfs for saving dependecies of spark dpp programe and spark platform.
Note that, the dependcies we upload to repository include:
1、`spark-dpp.jar`
2、`spark2x.zip`
1 is the dpp library which built with spark-dpp submodule. See details about spark-dpp submodule in pr #4146 .
2 is the spark2.x.x platform library which contains all jars in $SPARK_HOME/jars
**The repository structure** will be like this:
```
__spark_repository__/
|-__archive_1_0_0/
| |-__lib_990325d2c0d1d5e45bf675e54e44fb16_spark-dpp.jar
| |-__lib_7670c29daf535efe3c9b923f778f61fc_spark-2x.zip
|-__archive_2_2_0/
| |-__lib_64d5696f99c379af2bee28c1c84271d5_spark-dpp.jar
| |-__lib_1bbb74bb6b264a270bc7fca3e964160f_spark-2x.zip
|-__archive_3_2_0/
| |-...
```
The followinng conditions will force fe to upload dependencies:
1、When fe find its dppVersion is absent in repository.
2、The MD5 value of remote file does not match the local file.
Before Fe uploads the dependencies, it will create an archive directory with name `__archive_{dppVersion}` under the repository.
#4076
1. The visibleVersionTime is updated when insert data to partition
2. GlobalTransactionMgr call partition.updateVisibleVersionAndVersionHash(version, versionHash) when fe is restarted
3. If fe restart, VisibleVersionTime may be changed, but the changed value is newer than the old value
Now, we only check database used data quota when create or alter table, or in some old type load job, but not for routine load job and stream load job. This PR provide a uniform solution to check db used data quota when data load job begin a new txn.
This PR is to optimize the logic of processing unfinishedTask when transaction is publishTimeout, we find errorReplica by
"db -> table -> partition -> index -> tablet(backendId) -> replica" path.
This PR is mainly do following three things:
1. Add thread name in fe log to make trace problem more easy.
2. Add agent_task_resend_wait_time_ms config to escape sending duplicate agent task to be.
3. Skip to continue to update replica version when new version is lower than replica version in fe.
This commit mainly supports creating bitmap_union, hll_union, and count materialized views.
* The main changes are as follows:
1. When creating a materialized view, doris judge the semantic analysis of the newly supported aggregate function.
Only bitmap_union(to_bitmap(column)), hll_union(hll_hash(column)) and count(column) are supported.
2. Match the correct materialized view when querying.
After the user sends the query, if there is a possibility of matching the materialized view, the query will be rewritten firstly.
Such as:
Table: k1 int, k2 int
MV: k1 int, mv_bitmap_union_k2 bitmap mv_bitmap_union
mv_bitmap_union = to_bitmap(k2)
Query: select k1, count(distinct k2) from Table
Found that there is a match between the materialized view column and the query column, the query is rewritten as:
Rewritten query: select k1, bitmap_union_count(mv_bitmap_union_k2) from table
Then when the materialized view is matched, it can be matched to the query materialized view table.
Sometimes the rewritten query may not match any materialized view, which means that the rewriting failed. The query needs to be re-parsed and executed again.
This commit mainly supports load bitmap_union, hll_union, and count materialized views.
The main changes are as follows:
1、insert stmt support load extend column
2、load stmt support load extend column
Issue : #3344
Co-authored-by: HangyuanLiu <460660956@qq.com>
When set `disable_colocate_balance` to false and set some BE to decommission,
`Coloratebalancer#balanceGroup` will choose decommissioned BE to locate tablets,
which is not right
Fix#4102
+ Building the materialized view function for schema_change here based on defineExpr.
+ This is a trick because the current storage layer does not support expression evaluation.
+ count distinct materialized view will set mv_expr with to_bitmap or hll_hash.
+ count materialized view will set mv_expr with count.
+ Support to regenerate historical data when a new materialized view is created in BE。
+ Support to_bitmap function
+ Support hll_hash function
+ Support count(field) function
For #3344
The output columns of query should be collected by all of tupleIds
in BaseTableRef rather than the top tupleIds of query.
The top tupleIds of count(*) is Agg tuple which does not expand the star.
Fixed#4065
[Bug]Fix some schema change not work right
This CL mainly fix some schema change to varchar type not work right
because forget to logic check && Add ConvertTypeResolver to add
supported convert type in order to avoid forget logic check
This PR is mainly to add `thrift_client_retry_interval_ms` config in be for thrift client
to avoid avalanche disaster in fe thrift server and fix some typo and some rpc
setting problems at the same time.
fix#4047#3886 has certain relevance to this case。
the sql : `bigtable t1 join mysqltable t2 join mysqltable t3 on t1.k1 = t3.k1`
1. after reorder:
t1, t2, t3
2. choose join t1 with t2:
t1 join t2 with no conditions, and Doris choose cross join
3. choose join (t1 join on t2) with t3:
in old code, the t2 is mysqlTable, so the cardinality is zero,
and "the cross join t1 with t2" 's cardinality is t1.cardinality multiply t2.cardinality,
for t2 is mysql, so t2.cardinality is zero, and "the cross join t1 with t2" is zero.
t3 is mysqltable, t3's cardinality is zero.
**If two tables need to be joined both are zero，we will choose the shuffle join**
So I change the mysql table ‘s cardinality from 0 to 1, the cross join's cardinality is not zero.
Its time to enable some features by default.
1. Enable FE plugins by setting `plugin_enable=true`
2. Enable dynamic partition by setting `dynamic_partition_enable=true`
3. Enable nio mysql server by setting `mysql_service_nio_enabled=true`
Also modify installation doc, add download link of MySQL client.
Fixes#3995
## Why does it happen
When SetOperations encounters that the previous node needs Aggregate, the timing of add AggregationNode is wrong. You should add AggregationNode first before add other children.
## Why doesn't intersect and union have this problem
intersect and union conform to the commutation law, so it doesn't matter if the order is wrong
## Why this problem has not been tested before
In the previous test case, not cover the previous node was not AggregationNode
After PR #3454 was merged, we should refactor and reorganize some logic for long-term sustainable iteration for Doris On ES.
To facilitate code review，I would divided into this work to multiple PRs (some other WIP work I also need to think carefully)
This PR include:
1. introduce SearchContext for all state we needed
2. divide meta-sync logic into three phase
3. modify some logic processing
4. introduce version detect logic for future using
This CL mainly support set replication_num property in dynamic partition
table if dynamic_partition.replication_num is not set, the value is the
same as table's default replication_num.
Now, FE use ThreadPoolManager to manage and monitor all Thread,
but there are still some threads are not managed. And FE use `Timer` class
to do some scheduler task, but `Timer` class has some problem and is out of date,
It should replace by ScheduledThreadPool.
Doris only support TThreadPoolServer model in thrift server, but the
server model is not effective in some high concurrency scenario, so this
PR introduced new config to allow user to choose different server model
by their scenario.
Add new FE config: `thrift_server_type`
