Commit Graph

6425 Commits

Author SHA1 Message Date
1c2831e06f [fix](spill) disable spill of sort and agg for now to avoid diisk overflow (#26209) 2023-11-01 17:20:26 +08:00
683832230c [Performance](Nereids): pass ConnectContext to avoid ThreadLocal.get() (#26165) 2023-11-01 16:16:42 +08:00
1770224322 [fix](planner) Fix decimal precision and scale wrong when create table like (#25802)
Use field datatype such as decimal(10, 0) to create table like. Because the scale is 0, the precision and scale will lost when create table like done. this will fix the bug.

**Before fix, create table with following SQL**:
CREATE TABLE IF NOT EXISTS db_test.table_test
(
    `name` varchar COMMENT "1m size",
    `id` SMALLINT COMMENT "[-32768, 32767]",
    `timestamp0` decimal null comment "c0",
    `timestamp1` decimal(38, 0) null comment "c1"
)
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES ('replication_num' = '1');

**and Then run**
CREATE TABLE db_test.table_test_like LIKE db_test.table_test
SHOW CREATE TABLE db_test.table_test_like;

the field `timestamp1` will be decimal(9, 0), it's wrong. this will fix it.
2023-11-01 03:11:43 -05:00
5efc6c4233 [fix](auth)Fix can not insert overwrite who have load priv (#25711)
Fix can not insert overwrite who have load priv
2023-11-01 15:21:16 +08:00
HB
6de1cc7a0a [fix](udf)java udf does not support overloaded evaluate method (#22681)
Co-authored-by: morningman <morningman@163.com>
2023-11-01 15:05:37 +08:00
268c69971d [fix](stats) Store max/min by base64 2023-11-01 14:31:35 +08:00
eaed0dea5e [refactor](executor)Add % suffix for cpu_hard_limit #26174 2023-11-01 10:27:48 +08:00
18dabe7386 [fix](compile) fe compile failed when generate doc and FE UT failed (#26164)
1. FE could not compile because below error. Intro by PR #25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - #25951
> - #26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR #25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (#26145)"
> This reverts commit 8d7abf60f94d2d1208b71e96b9290ea02122b8d8.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (#26130)"
> This reverts commit 19122b55cd95af097b4ef7b6eb809f37db29765f.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (#26084)"
> This reverts commit 0d956e90cf920039b8baa79c170a298be56a128d.
2023-10-31 20:50:44 -05:00
ac7678926f [improvement](tablet clone) add config for decommission tablet wait time (#26000) 2023-10-31 20:36:26 +08:00
08c78a1135 [Feature](Job)Support manual and refactor some execution logic (#26082)
Supports manually triggered JOBs and Tasks
Optimize JOB&TASK display logic
Refactor the executor to support context passing
2023-10-31 20:35:55 +08:00
b98744ae90 [Bug](iceberg)fix read partitioned iceberg without partition path (#25503)
Iceberg does not require partition values to exist on file paths, so we should get the partition value from `PartitionScanTask.partition`.
2023-10-31 18:09:53 +08:00
b137f03921 [feature](profile) add MaxActiveTime and add exectime in pipelineX (#26148) 2023-10-31 17:53:52 +08:00
Pxl
271b074f42 Bug](view) make forbiddenMVRewrite work on InlineViewRef (#26162)
make forbiddenMVRewrite work on InlineViewRef
2023-10-31 17:30:59 +08:00
6f316b117f [Fix](Checkpoint)AnalysisMgrV2 does not implement the corresponding read and write interface (#26161) 2023-10-31 17:24:35 +08:00
994db8b400 [fix](multi-catalog)add the FAQ for Aliyun DLF and add the fs.xx.impl check (#25594)
1. add the FAQ for Aliyun DLF,include the DLF configuration and Jindo SDK locations
2. add the fs.xx.impl check when load cache files
2023-10-31 17:09:52 +08:00
958c7fdfd9 [fix](planner)should keep at least one slot materialized in agg node (#26116) 2023-10-31 16:52:45 +08:00
a6e010f722 [opt](nereids) use rowcount to compute runtime filter size if ndv is not available (#26022) 2023-10-31 16:52:17 +08:00
a259ae9417 [fix](mtmv)add mtmvJobManager to DEPRECATED_MODULE_NAMES (#26137)
Inherited from:26041
2023-10-31 16:28:43 +08:00
19122b55cd [enhancement](Nereids): optimize GroupExpressionMatching (#26130)
1. pattern can't be SubTreePattern in CBO phase.
2. optimize getInputSlotExprId()
2023-10-31 15:47:13 +08:00
111b8e2b4f [minor](planner) remove len on integral type toSql (#25951) 2023-10-31 02:37:08 -05:00
feefebe0da [chore](log opt) opt mysql load log (#26149) 2023-10-31 14:59:52 +08:00
95c765f5fe [fix](Nereids) values list could not process only one column (#26123) 2023-10-31 01:58:54 -05:00
ea1e8fa15d [Fix](Nereids) fix create table as select of view with unknowed length character type (#25471)
Problem:
when create table as select from a view with unknown length character type, be would return an error of inserting data failed

Example:
doris/regression-test/suites/ddl_p0/test_ctas.groovy

Reason & Solved:
BE can not derive varchar length automaticly so FE should tell BE to maximize the size of varchar type
2023-10-31 01:50:28 -05:00
0449a240f4 [Fix](from_unixtime) Keep consistent with MySQL & bug fix (#25966)
Bug fix: implicit convert from int32 -> int64 makes negative time stamp valid, so change signature to int64
Consistent: keep consistent with mysql.
2023-10-31 14:31:24 +08:00
Pxl
8d7abf60f9 [Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan when match ba… (#26145)
* SelectMaterializedIndexWithAggregate do not change plan when match base index

* forbid table withh alias
2023-10-31 14:11:08 +08:00
588d4c150f [bugfix](DateTimeV2Literal) The start and end dates of daylight saving time vary from year to year, which can lead to different test results at different times (#26083)
The start and end dates of daylight saving time vary depending on the year. Daylight saving time usually starts on the last Sunday in March and ends on the last Sunday in October. During this time, the offset for Europe/London is UTC+1. Outside of daylight saving time, the Europe/London offset is UTC+0.
2023-10-31 13:39:38 +08:00
809510f8b2 [bug](udf) Fix method invoking (#26131) 2023-10-31 11:46:14 +08:00
78204f7c92 [Fix](statistics)Fix external couldn't analyze database bug (#26025) 2023-10-31 11:32:47 +08:00
b3f31f9e1a [enhancement](Nereids): speedup graphsimplifier (#26066)
1. fix some bugs in graphsimplifier
2. remove some time costed code
3. add shape check for graphsimplifier
2023-10-30 20:13:13 +08:00
2a74d9a8c8 [feature](mtmv)(1)remove old mtmv code (#26041)
remove old mtmv code,we will implement mtmv in a new way
2023-10-30 19:49:45 +08:00
f97c40d0e1 [fix](Nereids) fix plan shape of query64 unstable (#26012)
don't remove the physical plan after optimizing the plan in dphyper.
2023-10-30 19:49:28 +08:00
f70dfe27d0 [fix](Nereids) init comment as null when create struct field from catalog type (#26105) 2023-10-30 06:26:09 -05:00
6eb2cb2d48 [date](parser) Support DateV1 keyword (#25414) 2023-10-30 18:39:22 +08:00
844b7c8cba [enhancement](Nereids): check stats unreliable when deriving stats (#26103)
check stats unreliable when deriving stats
2023-10-30 17:21:42 +08:00
3a954cd1aa [fix](function)return NULL rather than 'null' if path not found (#25880)
fix json_extract not return NULL but null
2023-10-30 14:26:44 +08:00
0d956e90cf [Performance](Nereids): optimize GroupExpressionMatching (#26084) 2023-10-30 14:05:25 +08:00
eb2cbae6e3 [Fix](MySqlLoad) Fix meaningless thread creation every time checkpoint mysql load (#26031)
Add a unified thread name setting method
2023-10-30 13:55:40 +08:00
00c30f075f [fix](nereids)only push down subquery in non-window agg functions (#26034) 2023-10-30 11:32:10 +08:00
bb814ccff8 [fix](Nereids) bound error when in diff value of table name case config (#25957)
- when lower_case_table_names = 0, reference of table name should case sensitive
- when lower_case_table_names = 1 and 2 cte name should case insensitive
more details, please read doris doc: https://doris.apache.org/docs/advanced/variables#supported-variables
2023-10-29 22:04:42 -05:00
d47a946aab [fix](auth)Fix no priv to insert to mysql catalog (#25678)
fix user can not insert to mysql catalog even if he has load permission
2023-10-30 10:59:53 +08:00
28dce9f226 [opt](nereids) let DBA ignore some runtime filters (#25933)
example:
set ignore_runtime_filter_ids="3, 1";
after this setting, RF003 and RF001 will be ignored
2023-10-29 21:39:25 +08:00
99b45e1938 [fix](Outfile) Export DateTimev2 type of doris to ORC's TimeStamp type (#25470)
Previously,doris's `DateTimev2` was exported to orc as a `String` type.
Now, export doris's `DateTimev2` to orc timestamp type.
2023-10-29 15:59:38 +08:00
1e5ed91e4d [fix](fe-meta) make FE meta be compatible with branch 2.0 (2.0.2.1 or later) (#26048)
In branch 2.0, we changed the read/write method of AnalysisManager,
and rename the image module name to AnalysisMgrV2.
So we need to make the same change in master branch, so that user
can upgrade Doris from branch-2.0 to master branch.

After this PR, user can:

- upgrade from 2.0.x(or branch-2.0) to master
2023-10-29 14:45:21 +08:00
c32cd5b25a FE ssl certificates are of various formats #26039 #26044 2023-10-28 22:37:38 +08:00
365fdd2f4d [feature](backup) add property to remove snapshot before creating repo (#25847)
Doris is not responsible for managing snapshots, but it needs to clear all
snapshots before doing backup/restore regression testing, so a property is
added to indicate that existing snapshots need to be cleared when creating a
repo.

In addition, a regression test case for backup/restore has been added.
2023-10-27 21:03:26 +08:00
c715facafa [fix](window_function) window function first_value/last_value should be always nullable (#26014) 2023-10-27 20:53:48 +08:00
d953e5c8f4 [feature](Nereids): Convert topn(x, 1) to max(x) (#26004) 2023-10-27 18:45:21 +08:00
2464a22ac5 [fix](meta) fix follower sync image timeout after checkpoint. (#26003)
The image file of our cluster reaches 2.3G. After the checkpoint, Followers synchronize the image timeout, resulting in the continuous increase of the bdb directory.

related pr: #25768
2023-10-27 16:58:42 +08:00
165957658a [fix](Nereids) could not run multi group_concat distinct (#25851)
could not run multi group_concat distinct with more than one parameters.
This bug is not just for group_concat, but we usually use literal as
parameters in group_concat. So group_concat brought the problem to light.

In the original logic, we think only distinct aggregate function with
zero or one parameter could run in multi distinct mode. But it is wrong.
We could process all distinct aggregate function with not more than one
input slots.

Think about sql:
```sql
SELECT
  group_concat(distinct c1, ','), group_concat(distinct c2, ',')
FROM t
GROUP BY c3
```
2023-10-27 03:42:00 -05:00
d57fc263cf [fix](Nereids) support create dup table without key (#25954) 2023-10-27 01:57:39 -05:00