Commit Graph

7481 Commits

Author SHA1 Message Date
wxy
e141664339 [fix](statistics) fix missing scanBytes and scanRows in query statist… (#14750)
* [fix](statistics) fix missing scanBytes and scanRows in query statistics when enable_vectorized_engine=true.

Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
2022-12-05 09:17:51 +08:00
ed96442b85 [fix](multi-catalog) fix persist issue about jdbc catalog and class loader issue #14794
Fix a bug that JDBC catalog/database/table should be add to GsonUtil

Fix a class loader issue that sometime it will cause ClassNotFoundException

Fix regression test to use different catalog name.

Comment out 2 regression tests:

regression-test/suites/query_p0/system/test_query_sys.groovy
regression-test/suites/statistics/alter_col_stats.groovy
Need to be fixed later
2022-12-05 09:05:13 +08:00
5be8f9432e [fix](DOE) Support ES index which contains dynamic_templates (#14762)
Support ES index with dynamic_templates. And do not support index mapping without explicit mapping.
2022-12-05 08:33:51 +08:00
852b03729f [Improvement](meta)add IsCurrent column in show catalogs result #14700
When a user has multiple catalogs and switch several times, he may forget which catalog is using. So I add a iscurrent column in show catalogs result for help.

mysql> show catalogs;
+-----------+-------------+----------+-----------+
| CatalogId | CatalogName | Type | IsCurrent |
+-----------+-------------+----------+-----------+
| 136591 | es | es | |
| 130100 | hive | hms | yes |
| 0 | internal | internal | |
+-----------+-------------+----------+-----------+
2022-12-05 08:32:16 +08:00
ce95da8dfb [improvement](multi-catalog) support specify hadoop username (#14734)
Support setting "hadoop.username" property when creating hms catalog.
2022-12-04 21:09:39 +08:00
90e44fdd72 [typo](docs)Documentation 1.2 Feature version annotation (#14795) 2022-12-04 19:12:22 +08:00
3a3c43fe23 [typo](docs)add Doris version information to the head of array function (#14780)
[Docs]add Doris version information to the head of array function
2022-12-04 15:18:02 +08:00
bb2d6addaf [typo](docs)Upgrade 1.2, Java UDF function depends on the installation method (#14785) 2022-12-04 08:53:42 +08:00
97dcd2b13a [feature](nereids) merge proj-proj in post process (#14730)
* merge proj-proj

* v2this pr guarantees that the physical plan does not contains consecutive physical projects.
Like rewrite rule "merge projects", it works on physical plan, not logical plan.

* move merge-proj code into Project.java
2022-12-03 23:41:02 +08:00
7bb2343505 [chore](github) Optimize the workflow FE Code Style Checker (#14751)
We can reduce the times of checking out the codebase to boost the performance of the workflow.
2022-12-03 13:26:09 +08:00
Pxl
484e9bcece [Chore](workflow) move clang-tidy to code-checks (#14758)
move clang-tidy to code-checks
2022-12-03 10:43:18 +08:00
283b23f6da [fix](planner) wrong results when select from view which has with clause (#14747) 2022-12-02 18:10:52 +08:00
12304bc0ee [Pipeline](exec) Support pipeline exec engine (#14736)
Co-authored-by: Lijia Liu <liutang123@yeah.net>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: Pxl <952130278@qq.com>
Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: Gabriel <gabrielleebuaa@gmail.com>

## Problem Summary:

### 1. Design

DSIP: https://cwiki.apache.org/confluence/display/DORIS/DSIP-027%3A+Support+Pipeline+Exec+Engine

### 2. How to use:

Set the environment variable `set enable_pipeline_engine = true; `
2022-12-02 17:11:34 +08:00
505019e1dd [minor](decimal) degrade log level for overflow decimal (#14748) 2022-12-02 15:55:26 +08:00
5c7964d396 [minor](tools) delete unused script (#14752) 2022-12-02 15:20:01 +08:00
Pxl
c804024e5d [Chore](workflow) add clang-tidy workflow (#14737)
add clang-tidy workflow
2022-12-02 14:10:29 +08:00
15bd56cd43 [Docs]merge memory-management directory into maint-monitor (#14755) 2022-12-02 14:05:48 +08:00
7627defc88 [fix](regression-test) Add test data for test_mysql_jdbc_catalog and fix mysql-5.7.yaml about UTF8 (#14749)
Fix two things:
1. Fix that the MySQL table displays the garbled code even if the UTF8 is specified for table.
2. Fix that `test_mysql_jdbc_catalog.out` lack of returned data for table `ex_tb13`.
2022-12-02 11:58:11 +08:00
07e8af7808 [regression](test) add external regression-test base on emr environment 1.0 11-29 (#14666)
* add external regression-test base on emr environment 1.0 11-29

* delete ak sk info from regression-conf.groovy
2022-12-02 11:30:07 +08:00
ae6a007c4e [test](jdbc)add new extremum case (#14692) 2022-12-02 11:28:11 +08:00
4ef6efb59b [Docs](memory) fix document directory and fix memtracker accuracy bug (#14710) 2022-12-02 11:27:18 +08:00
f9d7983671 [fix](doris-demo)fix flink-connector demo maven dependency conflict (#14714) 2022-12-02 10:18:27 +08:00
e9799fab09 [refactor](datev2) refine function expr for datev2 (#14697)
* [refactor](datev2) refine function expr for datev2

* update
2022-12-02 10:13:11 +08:00
3dde97bff1 (compaction) opt compaction task producer and quick compaction (#13495) (#14535)
1.remove quick_compaction's rowset pick policy, call cu compaction when trigger
quick compaction
2. skip tablet's compaction task when compaction score is too small

Co-authored-by: yixiutt <yixiu@selectdb.com>
2022-12-02 10:07:44 +08:00
228e9ed01c [fix](improvement)(meta) fix alter catalog properties issues and reformat code (#14745)
1. fix NPE exception #14740
2. fix issue:
mysql> alter catalog xyz set properties ('hive.metastore.uris'='thrift://172.21.0.1:7004');
ERROR 1105 (HY000): errCode = 2, detailMessage = Can't modify the type of catalog property with name: xyz
3. change behavior. The original logic is use props in set properties clause to replace all exists props, now change to only replace the listed props in set properties clause, and new props will be added. Make it behavior like alter table property stmt.
2022-12-02 09:34:13 +08:00
5b29489c7f (tcmalloc) gc does not work in somecases (#14732)
gc does not work in some cases
2022-12-02 09:18:23 +08:00
f484bc5380 [github](ci) change required checks of branch 1.1 (#14743)
* [github](branch-1.1) change required checks of branch 1.1

* modify
2022-12-02 09:11:49 +08:00
e5000c708e [feature](statistics) Support for collecting statistics on materialized view (#14676)
1. Map muiltiple tasks to one Job
2. Remove the codes for analyzing whole default db, since this feature is not available and would create too many tasks and related code is confusing
3. support analyze materialized view
4. abstract the common logic to BaseTask
2022-12-01 22:34:13 +08:00
94a6ffb906 [feature](compaction) support vertical_compaction & ordered_data_compaction (#14524) 2022-12-01 22:15:41 +08:00
2be8235d95 [feature](nereids) support timestampdiff function (#14662)
complete timeStampDiff
supported timeunit:
 - YEAR 
 - MONTH
 - WEEK
 - DAY
 - HOUR
 - MINUTE
 - SECOND
2022-12-01 22:11:55 +08:00
9a01716a57 [chore](fix)build docs fix #14727 2022-12-01 21:57:29 +08:00
14e208354d [Feature](Nereids) support nereids event for logging the cascades states and transformation. (#13659)
Add an event producer, channel, consumer system to support the feature as title and you can turn it on using set
enable_nereids_event = true;
For more information, please see fe/fe-core/src/main/java/org/apache/doris/nereids/metrics/README.md
2022-12-01 21:42:40 +08:00
5ca6596ca3 [fix](tpch-tools) disable join reorder for Q12 in TPC-H tools (#14728) 2022-12-01 20:55:17 +08:00
302da03b18 [enhancement](Nereids): Use long bitmap in DPHyp (#14725) 2022-12-01 20:47:45 +08:00
ba9a777554 [fix](function) StringRef should not be key of timezone cache (#14719) 2022-12-01 16:31:47 +08:00
9dd1d989e8 [test](decimalv3) add regression test cases for decimalv3 (#14672) 2022-12-01 15:18:40 +08:00
f496d1972a [improvement](multi-catalog) return root cause of exception (#14708) 2022-12-01 14:58:05 +08:00
176f519fa1 [enhancement](memtracker) Optimize exec node memory tracking (#14711) 2022-12-01 14:52:21 +08:00
b4d32a0c44 [fix](join) runtime filter shared from other instance wasn't be published (#14717) 2022-12-01 14:17:23 +08:00
Pxl
bba77fa9dd [Enhancement](profile) enhance column predicates display on profile (#14664) 2022-12-01 13:07:12 +08:00
3c6b96b9be [enhancement](Nereids) avoid add project that output same with child to memo (#14180) 2022-12-01 10:49:44 +08:00
7873bc95a6 [Enhancement](bitmapfilter) Support bitmap filter to apply zone_map index to filter pages (#14635) 2022-12-01 10:41:09 +08:00
ce9a160d16 [enhancement](macOS) Make CLion work out of the box (#14689)
We can't build the project after import it to CLion on macOS. Some options must be provided by default.
2022-12-01 10:40:04 +08:00
12791f1c79 [opt](docs) Add select except usage in select doc (#14696) 2022-12-01 10:06:17 +08:00
2a3a758c75 [doc](community) update release-verify doc when gpg import (#14706) 2022-12-01 10:04:58 +08:00
6c70d794f6 [fix](bitmapfilter) fix core dump caused by bitmap filter (#14702) 2022-12-01 09:56:22 +08:00
36737fe9f4 [feature](Nereids): Add cache to avoid repeatly calculation in DPhyp (#14585) 2022-11-30 21:35:45 +08:00
9bbbcf031c [enhancement](k8s) Support fqdn mode for be in k8s enviroment (#9172)
In the k8s environment, the ip of the pod can be changed, but the hostname of pod is stable. When the host machine of the pod fails, the k8s can schedule the failed pod to the new host machine for reconstruction. After that, the newly created pod's hostname remains unchanged, and the ip address has been changed. The change of the be node's ip address can be detected by FQDNManager when enable_fqdn_mode is true

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2022-11-30 20:42:15 +08:00
80baca2643 [Docs](memory) Admin-manual adds mem tracker, memory exceeds limit, OOM analysis (#14419) 2022-11-30 18:02:05 +08:00
738c36109f rename tpch dir (#14668) 2022-11-30 17:59:13 +08:00