* [fix](ut) try to fix ConcurrentModifycationException bug
* [Regression](Fix) fix the regression of pipeline and ConcurrentModificationException failed
Co-authored-by: morningman <morningman@163.com>
Fix a bug that JDBC catalog/database/table should be add to GsonUtil
Fix a class loader issue that sometime it will cause ClassNotFoundException
Fix regression test to use different catalog name.
Comment out 2 regression tests:
regression-test/suites/query_p0/system/test_query_sys.groovy
regression-test/suites/statistics/alter_col_stats.groovy
Need to be fixed later
When a user has multiple catalogs and switch several times, he may forget which catalog is using. So I add a iscurrent column in show catalogs result for help.
mysql> show catalogs;
+-----------+-------------+----------+-----------+
| CatalogId | CatalogName | Type | IsCurrent |
+-----------+-------------+----------+-----------+
| 136591 | es | es | |
| 130100 | hive | hms | yes |
| 0 | internal | internal | |
+-----------+-------------+----------+-----------+
* merge proj-proj
* v2this pr guarantees that the physical plan does not contains consecutive physical projects.
Like rewrite rule "merge projects", it works on physical plan, not logical plan.
* move merge-proj code into Project.java
1. fix NPE exception #14740
2. fix issue:
mysql> alter catalog xyz set properties ('hive.metastore.uris'='thrift://172.21.0.1:7004');
ERROR 1105 (HY000): errCode = 2, detailMessage = Can't modify the type of catalog property with name: xyz
3. change behavior. The original logic is use props in set properties clause to replace all exists props, now change to only replace the listed props in set properties clause, and new props will be added. Make it behavior like alter table property stmt.
1. Map muiltiple tasks to one Job
2. Remove the codes for analyzing whole default db, since this feature is not available and would create too many tasks and related code is confusing
3. support analyze materialized view
4. abstract the common logic to BaseTask
Add an event producer, channel, consumer system to support the feature as title and you can turn it on using set
enable_nereids_event = true;
For more information, please see fe/fe-core/src/main/java/org/apache/doris/nereids/metrics/README.md
In the k8s environment, the ip of the pod can be changed, but the hostname of pod is stable. When the host machine of the pod fails, the k8s can schedule the failed pod to the new host machine for reconstruction. After that, the newly created pod's hostname remains unchanged, and the ip address has been changed. The change of the be node's ip address can be detected by FQDNManager when enable_fqdn_mode is true
Co-authored-by: caiconghui1 <caiconghui1@jd.com>
# Proposed changes
Issue Number: close#14280
## Problem summary
The AggregateDisassemble rule is refactored and split into two rules, which are not dependent on each other.
1. AggregateDisassemble splits the agg into two phases: Local, Global.
1.1. For count function, the implementation is as follows：distinct_multi_count(update)+ distinct_multi_count(merge)
2. DistinctAggregateDisassemble splits the agg into 4 stages: Local, Global, Distinct Local, Distinct GLobal.
2.1. For count function, the implementation is as follows：distinct_multi_count(update)+ distinct_multi_count(merge)+sum(update)+ sum(merge)
The logger Doris Fe uses is log4j, while netty might use slf4j to choose one logger.
And it's reported some confusing occasions would happen under such circumstance.
And this binding doesn't take effect if move the bind logic to other file or other place within PaloFe.java,
so I have to leave it before the main function.
Issue Number: close #xxx
I add jdbc catalog for doris multi-catalog feature.
Currently, the jdbc catalog only supports MYSQL DBMS.
TODO:
support for postgre DB
Support for other databases.
Problem summary
For jdbc catalog, we can create catalog like:
CREATE CATALOG jdbc4 PROPERTIES (
"type"="jdbc",
"jdbc.user"="root",
"jdbc.password"="123456",
"jdbc.jdbc_url" = "jdbc:mysql://127.0.0.1:13396/demo?yearIsDateType=false",
"jdbc.driver_url" = "file:/mnt/disk2/ftw/tools/jar/mysql-connector-java-5.1.47/mysql-connector-java-5.1.47.jar",
"jdbc.driver_class" = "com.mysql.jdbc.Driver"
);
Note:
yearIsDateType is a param of jdbc:
If yearIsDateType configuration property is set to false, then the returned object type is java.sql.Short. If set to true (the default), then the returned object is of type java.sql.Date with the date set to January 1st, at midnight.
To compat with mysql, we force the use of yearIsDateType=false in FE. if user sets yearIsDateType=true, doris FE will force to change yearIsDateType=false.
estimate plan cost without column statistics.
change list:
1. remove original StatsCalculator, it is replaced by StatsCalculatorV2. rename StatsCalculatorV2 to StatsCalculator
2. remove FilterSelectivityCalculator, it is replaced by FilterEstimation
3. remove session var:ENABLE_NEREIDS_STATS_DERIVE_V2
4. add ColumnStatistics.isUnKnown, which means the column is not analyzed, and its stats is not accurate.
5. add estimatedRowCount() function for OLAP tables
6. add unit tests for FilterEstimation and StatsCalculator
the cold-hot separation feature is still
under development. And seems there are some unsolved feature remains.
So I add a fe config enable_storage_policy, and default is false, to disable the creation and usage of storage policy by default.
So that user can aware that he is using an experimental feature on his own, and it will not be released formally in v1.2.0.
Disable storage policy by default, user can not use or create storage policy. Configured by enable_storage_policy.
Remove property remote_storage_policy, it is duplicate with storage_policy
Change the persist field in DataProperty.java.
And remove remoteCooldownTime from DataProperty, because it can be got from StoragePolicy.
Add tpch 1g orc test case in hive docker
Refactor some suites dir of catalog test cases.
And "-internal" for dlf endpoint, to support access oss with aliyun vpc.
