Commit Graph

7337 Commits

Author SHA1 Message Date
097641b543 [fix](Nereids): fix AssertNumRows StatsCalculator (#30053) 2024-01-19 15:48:15 +08:00
f1462f6cf4 [fix](Nereids): eliminate redundant join condition after inferring condition (#30093)
eliminate redundant join when find hashing join condition
such as for plan:
```
T1 join T2 on T1.id = T2.id join T3 on T1.id = T3.id and T2.id = T3.id 
```
we infer a new predicate T1.id = T2.id which is redundant. Therefore we need to eliminate it when find hash condition
2024-01-19 15:48:15 +08:00
7d3a3fee65 [fix](Nereids) update assignment column name should case insensitive (#30071) 2024-01-19 15:48:15 +08:00
Pxl
2ccb69dbed [Feature](materialized-view) support some case unmached to materialized-view (#30036)
same column appears in key and value like select id,count(id) group by id;
complex expr in sum select sum(if(xxx));
2024-01-18 12:03:07 +08:00
0ccd706a30 [Enhancement](Jdbc Catalog) Map Jdbc Catalog JSON Type to String for Improved Performance and Compatibility (#30035)
This PR proposes mapping external catalog JSON types to String instead of JsonB in Apache Doris. This change is motivated by the realization that JDBC retrieves JSON data as a String JSON string, regardless of its storage format (Json(String) or Json(Binary)). Mapping to String streamlines data retrieval, simplifies write-backs, and ensures compatibility with all JSON(String) and JSON(Binary) functions, despite potentially misleading displays of JSON data as Strings in Doris. This approach avoids the performance overhead and complexity of converting each row of data from JsonB to String, making the process more efficient and elegant.

About Upgrade
To ensure query compatibility with existing Catalogs in the upgraded version,we currently still retain the capability to query external JSON types as JSONB. However, once you upgrade to the new version and either refresh the Catalog or create a new one, all external JSON types will be treated as Strings. To ensure consistent behavior,and possible future removal of support for JSON as JSONB query code, it is highly recommended that you manually refresh your Catalog as soon as possible after upgrading to the new version.
2024-01-18 12:03:07 +08:00
c1722a6ab0 [Fix](schema change) fix tablet meta write race (#30081) 2024-01-18 12:03:07 +08:00
d554d5478c [improvement](fe) skip attach profile during send fragment (#30085)
Sometime attach profile may cost a lot of time in large machine with many instances.


---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-01-18 12:03:07 +08:00
4608bcd7bf [fix](ut) fixed test cases failure for showCreateFunction (#28593)
2 tests cases (ShowCreateFunctionTest and ShowFunctionTest) have function name clash;
because of which one of the test fails to complete setup() and eventually fail for not able to create funciton.

Changed the name of the test function in 1 of the file.
2024-01-18 12:03:07 +08:00
7d1b3d4704 [feature](statistics, metadata)Meta data place holder for statistics (#29867)
Meta data place holder for statistics in version 2.1.x. Users could upgrade to this version, but doesn't support rollback.
After this change, statistics related functions doesn't need to change meta data any more in the 2.1 series.
2024-01-18 12:03:07 +08:00
990d5d8664 [cleanup](insert-into) clean up some insert into log (#30063) 2024-01-18 12:03:07 +08:00
65b5a5436a [minor](Nereids): remove useless method isMysqlCompatibleDatabase (#30064) 2024-01-18 12:03:07 +08:00
44ba9e102c [feature](statistics)support statistics for iceberg/paimon/hudi table (#29868) 2024-01-18 12:03:07 +08:00
ade720470d [Improve](config)delete confused config for nested complex type (#29988) 2024-01-18 12:03:07 +08:00
dcc9cf9dd3 [config](Nereids): add SessionVariable to control nereids timeout (#30048) 2024-01-18 10:04:21 +08:00
e73c0f4077 [fix](fe) Fix drop table write too many times editlog (#29966) 2024-01-18 10:04:21 +08:00
d4195b3cc0 [fix](Nereids): fix enable mv rewrite is useless in dphyp (#30042) 2024-01-18 10:04:21 +08:00
Pxl
b0c49024cb [Feature](materialized-view) support match function with alias in materialized-view (#30025)
support match function with alias in materialized-view
2024-01-18 10:04:21 +08:00
2ca3c82135 [fix](fe ut) fix unstable TabletRepairAndBalanceTest (#30041) 2024-01-18 10:04:21 +08:00
3deee14680 [fix](Nereids): find hash condition after infer predicate (#30026) 2024-01-18 10:03:01 +08:00
3f2a794c2e [refactor](insert) remove unused insert code in FE #29924 2024-01-18 09:00:32 +08:00
d20103cad4 [fix](compatibility) remove cluster prefix in user property info (#30078) 2024-01-18 08:55:14 +08:00
2dcdf07dd4 [Feature](profile)Support active_queries TVF (#29999) 2024-01-16 21:25:02 +08:00
d658a44cef [improvement](catalog) Change the push-down parameters of the predicate function of the table query SQL into variables (#30028)
In this PR, we will control whether the external data source query is a push-down function parameter in the filter condition, changing the enable_fun_pushdown of fe conf to the enable_ext_func_pred_pushdown of the variable
2024-01-16 21:14:35 +08:00
8090a395bd change sessionvariable.java style 2024-01-16 20:40:36 +08:00
e693249f5b [fix](Nereids) remove useless substr of string like type cast (#29997) 2024-01-16 20:23:09 +08:00
4e41e1d797 [feat](Nereids) persist constraint in table (#29767) 2024-01-16 18:49:29 +08:00
168afdb965 [fix](disk balance) Change disk rebalance unpick time to configurable (#28949) 2024-01-16 18:49:04 +08:00
74991c4af2 [bugfix](paimon)support native and jni to read paimon for minio/cos #29933 2024-01-16 18:49:01 +08:00
4bf4239d7a [feature](Nereids): optimize logical group expression in dphyp (#30000) 2024-01-16 18:48:20 +08:00
6ce9c67383 [fix](Nereids) struct type coercion call wrong api (#30018) 2024-01-16 18:48:20 +08:00
55a0f90003 [fix](jobs)fix create task int job (#30011) 2024-01-16 18:46:19 +08:00
75cafa8672 [enhancement](jdbc catalog) Enhance function pushdown of Jdbc Oracle Catalog (#29972) 2024-01-16 18:46:19 +08:00
f53d2c28cb [improvement](catalog) fix jdbc mysql catalog to_date fun pushdown (#29900) 2024-01-16 18:46:19 +08:00
22978726e3 [opt](nereids) if column stats are unknown, 10-20 table-join optimization use cascading instead of dphyp (#29902)
* if column stats are unknown, do not use dphyp
tpcds query64 is optimized in case of no stats
sf500, query64 improved from 15sec to 7sec on hdfs, and from 4sec to 3.85sec on olaptable
2024-01-16 18:46:19 +08:00
e1bcdc35fd (fix)[group-commit] Fix some group commit case (#30008) 2024-01-16 18:46:19 +08:00
07de535c4c [fix](Nereids) should not fold constant when do ordinal group by (#29976) 2024-01-16 18:46:19 +08:00
ce2e84e9a6 [fix](nereids)need validate auto partition columns in DDL (#29985) 2024-01-16 18:46:19 +08:00
f7e12605a2 [fix](Nereids): fix NPE InferPredicates (#29978)
PredicatePropagation shouldn't add null into List.
2024-01-16 18:46:19 +08:00
2bf8c51baa [test](regression) Add debug level log of editlog for p0 p1 (#29992) 2024-01-16 18:42:09 +08:00
66513d57f9 [feature](function) support ip function named ipv6_cidr_to_range(addr, cidr) (#29812) 2024-01-16 18:42:09 +08:00
0ea5f0f5de [fix](Nereids): fix offset in PlanTranslator (#29789)
Current BE operator don't support `offset`, we need add offset into `ExchangeNode`
2024-01-16 18:41:21 +08:00
d5dcdf3e07 [Improve](array) support array_enumerate_uniq and array_suffle for nereids (#29936) 2024-01-16 18:40:32 +08:00
94bfa6e46d [fix](Nereids): fix wrong api calling (#29977) 2024-01-16 18:40:32 +08:00
b99b9efda8 [fix](Nereids) top-n with top project should hit top-n opt (#29971)
in PR #29312, we limit top-n opt only enable in simplest cases.
in this PR, we let go of some restrictions.
2024-01-16 18:40:32 +08:00
e09118eb9c extract agg in struct info node (#29853) 2024-01-16 18:39:37 +08:00
f6dc6ea13b [improvement](catalog) Escape characters for columns in recovery predicate pushdown in SQL (#29854)
In the previous logic, when we restored the Column in the predicate pushdown based on the logical syntax tree for JdbcScanNode, in order to avoid query errors caused by keywords such as `key`, we added escape characters for it, but before we only Binary predicates are processed, which is imperfect. We should add escape characters to all columns that appear in the predicate to avoid errors with keywords or illegal characters.
2024-01-16 18:39:00 +08:00
e35b26f4fc [feature](auditlog)Add runtime cpu time/peak memory metric (#29925) 2024-01-16 18:39:00 +08:00
97955da749 [enhancement](fe-memory) support label num threshold to reduce fe memory consumption (#22889)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-01-16 18:38:54 +08:00
b47e289560 [refactor](Nereids): avoid ConnectContext.get() ASAP to improve proformance (#29952) 2024-01-16 18:37:44 +08:00
7a574df9fc [fix](pipelineX) fix multi be may be missing profiles #29914 2024-01-16 18:37:44 +08:00