Commit Graph

3104 Commits

Author SHA1 Message Date
638d4b6f27 [Bug](function) Fix function for cast string as date/datetime (#35637) (#37891)
## Proposed changes

Issue Number: close #35635
cherry-pick https://github.com/apache/doris/pull/35637 from master to
branch-2.1

<!--Describe your changes.-->
Cast rules:Consistent with mysql.
String:Date
The first part is 1-digit x: 000x-month-day
The first part is 2-digit xy: 20xy-month-day / 19xy-month-day The first
part is 3-digit xyz: 20xy-0z-day / 19xy-0z-day The first part is 4-digit
xyzw: xyzw-month-day

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 17:22:03 +08:00
46c06b3d97 [enhancement](regression-test) sync unique case to 2.1 (#37272)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-07-16 16:59:33 +08:00
9fd4795080 [opt](iceberg)Add a new appearance to display the pushDown count for 2.1 (#37046) (#34928) (#37810)
## Proposed changes

bp: #37046 #34928
2024-07-16 16:03:44 +08:00
80ea98b371 [fix](nereids)subquery unnesting get wrong result if correlated conjuncts is not slot_a = slot_b (#37683)
pick from master https://github.com/apache/doris/pull/37644

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 15:06:40 +08:00
2c4b5519e9 [cherry-pick](branch-2.1) let insert statement support CTE (#36150) (#36265)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

cherry-pick: #36150
2024-07-16 14:50:53 +08:00
8e42871228 [fix](in expr) fix error result when in expr has null value and lef… (#37800)
https://github.com/apache/doris/pull/36024

## Proposed changes

```
create table t2 (id int, c1 int);
insert into t2 values(1, null);
 select 0 in (c1, null) from t2; -- should return null,but 1
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 14:04:35 +08:00
253f929234 [cherry-pick](branch-2.1) fix inverted index file size (#37836)
## Proposed changes

pick from master #37232
pick from master #37564
2024-07-16 11:28:47 +08:00
2957fdc039 [branch2.1] pick [fix](show) show create table show index comment err… (#37034)
## Proposed changes
pick https://github.com/apache/doris/pull/36306
2024-07-16 11:19:27 +08:00
3cb1d4e842 [feature](json)support explode_json_object func #36887 (#37378) 2024-07-16 10:59:11 +08:00
e64f2997e9 [fix](function) fix core when input not null array in foreach functio… (#37798)
## Proposed changes
https://github.com/apache/doris/pull/37349
error code
```C++
return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
                                                                      nested_function);
```
"transform_arguments is an internal type of array. All internal types of
the array are null, so an array that is not null was mistakenly treated
as a null array."

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 10:57:11 +08:00
e7a001c420 [enhance](mtmv)support partition tvf (#37795)
pick from: https://github.com/apache/doris/pull/36479 and
https://github.com/apache/doris/pull/37201
2024-07-16 09:27:44 +08:00
aa2b902633 [cherry-pick](branch-21) fix broadcast join running when hash table build not finished (#37844)
cherry-pick from master https://github.com/apache/doris/pull/37792

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 09:20:06 +08:00
78eb9d8e33 [case](restart_fe) add demo case for restart_fe test (#37091) (#37313)
pick from master #37091

Co-authored-by: stephen <hello-stephen@qq.com>
2024-07-15 19:42:20 +08:00
63c2d22513 [cherry-pick](branch-2.1) Pick "[Fix](delete command) Mark delete sign when do delete command in MoW table (#35917)" (#37594)
Pick #35917 and #37151
2024-07-15 18:54:01 +08:00
03e21dddff [cherry-pick](branch-21) fix cast string to int return wrong result (#36788) (#37803)
## Proposed changes
cherry-pick from master:
https://github.com/apache/doris/pull/36788
https://github.com/apache/doris/pull/36505

<!--Describe your changes.-->
2024-07-15 18:48:49 +08:00
Pxl
e5219467dd [Bug](join) avoid overflow on bucket_size+1 (#37807)
## Proposed changes
pick from #37493
2024-07-15 18:47:36 +08:00
b7dbd5c186 [feature](inverted index) add ordered functionality to match_phrase query (#37751)
## Proposed changes

1. select count() from tbl where b match_phrase 'the brown ~2+';
2024-07-15 18:42:48 +08:00
Pxl
d1fc4e2e60 [Bug](query) fix meet invalid column when direct scan on mow mv (#37806)
pick from #36483
2024-07-15 18:29:30 +08:00
e5339a4014 [feature](ES Catalog)Support control scroll level by config #37180 (#37290)
## Proposed changes

backport #37180
2024-07-15 16:41:38 +08:00
7bd6818350 [branch-2.1][improvement](jdbc catalog) Added support for Oracle Raw type (#37776)
pick (#37078)
In previous versions, we adopted the strategy of reading the object
address for Oracle's raw type, which would lead to unstable and
meaningless results. Here I changed it to read hexadecimal or UTF8
2024-07-15 14:43:05 +08:00
2759383365 [branch-2.1](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) (#37269)
pick https://github.com/apache/doris/pull/37062

1. revert https://github.com/apache/doris/pull/25097. we decide to rely
on OS. not maintain independent tzdata anymore to keep result
consistency
2. refactor timezone load. removed rwlock.

before:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (6.88 sec)
```
now:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (2.61 sec)
```
3. now don't support timezone offset format string like 'UTC+8', like we
already said in
https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage
4. support case-insensitive timezone parsing in nereids.
5. a bug when parse timezone using nereids. should check DST by input,
but wrongly by now before. now fixed.

doc pr: https://github.com/apache/doris-website/pull/810
2024-07-15 10:56:48 +08:00
8de13c5cc8 [fix](function) error scale set in unix_timestamp (#36110) (#37619)
## Proposed changes

```
mysql [test]>set DEBUG_SKIP_FOLD_CONSTANT = true;
Query OK, 0 rows affected (0.00 sec)

mysql [test]>select cast(unix_timestamp("2024-01-01",'yyyy-MM-dd') as bigint);
+------------------------------------------------------------+
| cast(unix_timestamp('2024-01-01', 'yyyy-MM-dd') as BIGINT) |
+------------------------------------------------------------+
|                                           1704038400000000 |
+------------------------------------------------------------+
```
now
```
mysql [test]>select cast(unix_timestamp("2024-01-01",'yyyy-MM-dd') as bigint);
+------------------------------------------------------------+
| cast(unix_timestamp('2024-01-01', 'yyyy-MM-dd') as BIGINT) |
+------------------------------------------------------------+
|                                                 1704038400 |
+------------------------------------------------------------+
1 row in set (0.01 sec)
```

The column does not have a scale set, but the cast uses the scale to
perform the cast.


<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-15 10:00:04 +08:00
b55dd6f644 [fix](delete) fix the error message for valid decimal data for 2.1 (#37710)
## Proposed changes

cherry-pick : #36802

<!--Describe your changes.-->
2024-07-15 09:54:42 +08:00
16de141743 [regression](kerberos)add hive kerberos docker regression env (#37657)
## Proposed changes
pick:
[regression](kerberos)fix regression pipeline env when write hosts 
(#37057)
[regression](kerberos)add hive kerberos docker regression env (#36430)
2024-07-15 09:35:39 +08:00
56a207c3f0 [case](paimon/iceberg)move cases from p2 to p0 (#37276) (#37738)
bp #37276

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-07-13 10:01:05 +08:00
f2556ba182 [feature](insert)support external hive truncate table DDL (#37659)
pick: #36801
2024-07-12 22:37:47 +08:00
ffa9e49bc7 [feature](mtmv) pick some mtmv pr from master (#37651)
cherry-pick from master
pr: #36318
commitId: c1999479

pr: #36111
commitId: 35ebef62

pr: #36175
commitId: 4c8e66b4

pr: #36414
commitId: 5e009b5a

pr: #36770
commitId: 19e2126c

pr: #36567
commitId: 3da83514
2024-07-12 10:35:54 +08:00
6214d6421f [Fix](planner) fix bug of char(255) toSql (#37340) (#37671)
cherry-pick #37340 from master
2024-07-12 10:33:24 +08:00
4dc933bb28 [cherry-pick] (branch-2.1) fix query errors caused by ignore_above (#37685)
## Proposed changes
pick from master #37679
2024-07-12 09:31:45 +08:00
79a208259e [cherry-pick] (branch-2.1) Remove the check for inverted index file exists #36945 (#37423) 2024-07-11 21:35:52 +08:00
217eac790b [pick](Variant) pick some refactor and fix #34925 #36317 #36201 #36793 (#37526) 2024-07-11 21:25:34 +08:00
d7cae940d2 [fix](test) fix case conflict between test_tvf_based_broker_load and test_broker_load #37622 (#37631)
cherry pick from #37622
2024-07-11 17:52:21 +08:00
1eb04cf538 [feature](mtmv) Support query rewrite by materialized view when query is aggregate and materialized view has no aggregate (#36278) (#37497)
cherry-pick from master
pr: #36278
commitId: 649f9bc6
2024-07-11 10:54:50 +08:00
e6b8ebc847 [Fix](Short Circuit) fix no project list in OlapScanNode (#37121) (#37504)
pick from #37121
2024-07-11 10:04:28 +08:00
e1cb568d11 [Optimize] Add session variable `max_fetch_remote_schema_tablet_count… (#37505)
pick from #37217
2024-07-11 10:04:20 +08:00
0a95757a4d [opt](test) Optimize execution time of test_s3_load case #37562 (#37612)
cherry pick from #37562
2024-07-10 19:09:46 +08:00
a4aaace481 [Fix](branch-2.1) Fix partial update regression test (#37560)
Version 2.1 does not support partial update for variant type data, so we
delete this case.
2024-07-10 10:44:24 +08:00
db4d061a68 [fix](Nereids) null type result with alias name should keep alias name (#37457) (#37524)
pick from master #37457
2024-07-09 20:46:51 +08:00
8ef83259ff [fix](planner) fix bug of select stmt toSql(#37274) (#37344)
cherry-pick from master #37274
2024-07-09 20:34:57 +08:00
9b075bc873 [fix](nereids) derive column stats for 'expr and A is not null' (#37235) (#37498)
pick from #37235 
the algorithm for computing stats for "expr1 and expr2" predicate is as
following:
1. compute output stats of expr1 based on input stats. the result stats
is denoted by leftStats
2. compute stats of expr2 based on leftStats after step1, leftStats
should be normalized to avoid abnormal cases, such as ndv > rowCount or
numNulls > rowCount

Issue Number: close #xxx

<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-09 17:46:57 +08:00
2d5eb55310 [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add some partial update case (#36385)" (#37382)
#36385
2024-07-09 10:52:04 +08:00
ca0e44f83f [fix](case) fix struct format out files (#37350) (#37499)
bp #37350
2024-07-09 10:11:50 +08:00
0a103aa11f [improve](json)improve json support empty keys #36762 (#37351) 2024-07-08 19:04:51 +08:00
fbc954e8be [feat](mtmv) Support grouping_sets rewrite when query rewrite by materialized view (#36056) (#37436)
cherry pick from master
pr: #36056
commitId: 569c9772
2024-07-08 15:06:16 +08:00
779a51570e [opt](mtmv) Set query rewrite by materialized view default enable (#35897) (#36949)
cherry pick from master
pr: #35897
commitId: 603fa82f
2024-07-08 14:29:38 +08:00
c66df8d9e6 [branch-2.1](load) fix no error url if no partition can be found (#36831) (#37401)
## Proposed changes

pick #36831

before
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]Encountered unqualified data, stop processing",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0
}
```

after
```
Stream load result: {
    "TxnId": 2014,
    "Label": "83ba46bd-280c-4e22-b581-4eb126fd49cf",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 1,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 1,
    "NumberUnselectedRows": 0,
    "LoadBytes": 1669,
    "LoadTimeMs": 58,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 10,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 47,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "http://XXXX:8040/api/_load_error_log?file=__shard_4/error_log_insert_stmt_c6461270125a615b-2873833fb48d56a3_c6461270125a615b_2873833fb48d56a3"
}
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-08 10:41:33 +08:00
32529ecda2 [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add partial update mix cases (#37113)" (#37384)
#37113
2024-07-07 18:26:46 +08:00
c399a0e216 [opt](inverted index) reduce generation of the rowid_result if not necessary #35357 (#36569) 2024-07-06 21:33:03 +08:00
ef59af8df0 [branch-2.1] Picks "[fix](regression) Fix p0 case test_modify_reorder_column #37256" (#37332)
## Proposed changes

picks https://github.com/apache/doris/pull/37256
2024-07-05 22:20:14 +08:00
5de6aa74c0 [branch-2.1] Picks "[opt](autoinc) Forbid some schema change when the table has auto-increment column #37186" (#37331)
## Proposed changes

picks https://github.com/apache/doris/pull/37186
2024-07-05 21:59:30 +08:00