Commit Graph

3122 Commits

Author SHA1 Message Date
193be20c86 [feature](csv)Supports reading CSV data using LF and CRLF as line separators. (#37687) (#38099)
bp #37687
2024-07-22 22:53:04 +08:00
d9fd419e47 [Fix](JsonReader) fix json with duplicate key entry may result out of bound exception (#38147)
#38146
2024-07-19 22:53:02 +08:00
22b9cc6eb2 [fix](mysql)fix mysql row buf (#38145)
## Proposed changes
backport: https://github.com/apache/doris/pull/37936
Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-19 22:18:42 +08:00
bb2b7774df [feature](iceberg) iceberg write support insert overwrite and optimize hive write transaction statistics and (#37191) (#38097)
bp #37191

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
2024-07-19 09:45:41 +08:00
301ff6af22 [Fix](nereids) add backquote and qualifier to udf, column, column alias and table alias when create view (#37237) (#37984)
cherry-pick #37237 to branch-2.1

---------

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2024-07-19 00:56:26 +08:00
de2272ce48 [fix](round) fix round decimal128 overflow (#37733) (#37963)
cherry-pick #37733 to branch-2.1
2024-07-18 23:50:23 +08:00
8c6ff22e04 [Pick](Variant) fix heap use after free and optimize cases #37991 #37976 (#38037) 2024-07-18 16:53:09 +08:00
c30c1d2436 [branch-2.1] Picks "[opt](delete) Delete job should retry for failure that is not DELETE_INVALID_XXX #37834" (#38032)
## Proposed changes

picks https://github.com/apache/doris/pull/37834 and
https://github.com/apache/doris/pull/38043
2024-07-18 14:50:30 +08:00
ebc178af23 [fix](nereids)acos function should return null literal instead of NaN value (#37996)
pick from master https://github.com/apache/doris/pull/37932

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-18 09:28:56 +08:00
78d51ca320 [fix](Nereids) tablet prune wrong when decimal value scale is nagtive (#37889) (#38013)
pick from master #37889

we use unscaled value of BigDecimal in tablet prune. So we need to
ensure BigDecimal's precision and scale is same with the literal
contains it.
2024-07-17 20:12:14 +08:00
b2a4cff51b [fix](nereids)fix nullable property of ForEachCombinator (#37980)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37796

<!--Describe your changes.-->
2024-07-17 13:48:21 +08:00
ba66ff5768 [fix](multi-catalog)fix paimon meta properties convert (#37249) (#37958)
bp #37249

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
2024-07-17 01:04:33 +08:00
b6e5281a1c [Fix](bug) fix the divide zero in local shuffle: (#37948)
## Proposed changes

cherry pick #37906 

<!--Describe your changes.-->
2024-07-17 01:03:53 +08:00
21c6b854f7 [fix](explode-json-object)fix explode json object (#37956)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-17 01:03:07 +08:00
f7068b5658 [cherry-pick](branch-2.1) Make doris read hive text table parameters and behavior consistent with hive (#37840)
## Proposed changes

pick from master https://github.com/apache/doris/pull/37638

<!--Describe your changes.-->
2024-07-16 22:24:50 +08:00
b15ccdbe98 [Pick](Variant) pick some fix (#37922)
#37674
#37839
#37883 
#37857 
#37794
2024-07-16 21:38:47 +08:00
9ff129b630 [fix](stream_load) fix stream load may failed caused by column name with keyword (#35822) (#37890)
#35938 #35822 
let 
KW_SQL,
KW_CACHE,
KW_COLOCATE,
KW_COMPRESS_TYPE,
KW_DORIS_INTERNAL_TABLE_ID,
KW_HOTSPOT,
KW_PRIVILEGES,
KW_RECENT,
KW_STAGES,
KW_WARM,
KW_UP,
KW_CONVERT_LSC
be as non-reserved

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

---------

Co-authored-by: caiconghui1 <caiconghui1@jd.com>
2024-07-16 20:20:34 +08:00
8440303b91 [fix](delete) Incorrect precision detection for the decimal type in condition.​ (#37801) (#37904)
## Proposed changes

pick #37801

For precision like Decimal(7,7), the value "0.1234567" should be
valid(the integer part is 0).

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 19:02:02 +08:00
638d4b6f27 [Bug](function) Fix function for cast string as date/datetime (#35637) (#37891)
## Proposed changes

Issue Number: close #35635
cherry-pick https://github.com/apache/doris/pull/35637 from master to
branch-2.1

<!--Describe your changes.-->
Cast rules:Consistent with mysql.
String:Date
The first part is 1-digit x: 000x-month-day
The first part is 2-digit xy: 20xy-month-day / 19xy-month-day The first
part is 3-digit xyz: 20xy-0z-day / 19xy-0z-day The first part is 4-digit
xyzw: xyzw-month-day

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 17:22:03 +08:00
46c06b3d97 [enhancement](regression-test) sync unique case to 2.1 (#37272)
Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
2024-07-16 16:59:33 +08:00
9fd4795080 [opt](iceberg)Add a new appearance to display the pushDown count for 2.1 (#37046) (#34928) (#37810)
## Proposed changes

bp: #37046 #34928
2024-07-16 16:03:44 +08:00
80ea98b371 [fix](nereids)subquery unnesting get wrong result if correlated conjuncts is not slot_a = slot_b (#37683)
pick from master https://github.com/apache/doris/pull/37644

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 15:06:40 +08:00
2c4b5519e9 [cherry-pick](branch-2.1) let insert statement support CTE (#36150) (#36265)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

cherry-pick: #36150
2024-07-16 14:50:53 +08:00
8e42871228 [fix](in expr) fix error result when in expr has null value and lef… (#37800)
https://github.com/apache/doris/pull/36024

## Proposed changes

```
create table t2 (id int, c1 int);
insert into t2 values(1, null);
 select 0 in (c1, null) from t2; -- should return null,but 1
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 14:04:35 +08:00
253f929234 [cherry-pick](branch-2.1) fix inverted index file size (#37836)
## Proposed changes

pick from master #37232
pick from master #37564
2024-07-16 11:28:47 +08:00
2957fdc039 [branch2.1] pick [fix](show) show create table show index comment err… (#37034)
## Proposed changes
pick https://github.com/apache/doris/pull/36306
2024-07-16 11:19:27 +08:00
3cb1d4e842 [feature](json)support explode_json_object func #36887 (#37378) 2024-07-16 10:59:11 +08:00
e64f2997e9 [fix](function) fix core when input not null array in foreach functio… (#37798)
## Proposed changes
https://github.com/apache/doris/pull/37349
error code
```C++
return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
                                                                      nested_function);
```
"transform_arguments is an internal type of array. All internal types of
the array are null, so an array that is not null was mistakenly treated
as a null array."

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 10:57:11 +08:00
e7a001c420 [enhance](mtmv)support partition tvf (#37795)
pick from: https://github.com/apache/doris/pull/36479 and
https://github.com/apache/doris/pull/37201
2024-07-16 09:27:44 +08:00
aa2b902633 [cherry-pick](branch-21) fix broadcast join running when hash table build not finished (#37844)
cherry-pick from master https://github.com/apache/doris/pull/37792

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-16 09:20:06 +08:00
78eb9d8e33 [case](restart_fe) add demo case for restart_fe test (#37091) (#37313)
pick from master #37091

Co-authored-by: stephen <hello-stephen@qq.com>
2024-07-15 19:42:20 +08:00
63c2d22513 [cherry-pick](branch-2.1) Pick "[Fix](delete command) Mark delete sign when do delete command in MoW table (#35917)" (#37594)
Pick #35917 and #37151
2024-07-15 18:54:01 +08:00
03e21dddff [cherry-pick](branch-21) fix cast string to int return wrong result (#36788) (#37803)
## Proposed changes
cherry-pick from master:
https://github.com/apache/doris/pull/36788
https://github.com/apache/doris/pull/36505

<!--Describe your changes.-->
2024-07-15 18:48:49 +08:00
Pxl
e5219467dd [Bug](join) avoid overflow on bucket_size+1 (#37807)
## Proposed changes
pick from #37493
2024-07-15 18:47:36 +08:00
b7dbd5c186 [feature](inverted index) add ordered functionality to match_phrase query (#37751)
## Proposed changes

1. select count() from tbl where b match_phrase 'the brown ~2+';
2024-07-15 18:42:48 +08:00
Pxl
d1fc4e2e60 [Bug](query) fix meet invalid column when direct scan on mow mv (#37806)
pick from #36483
2024-07-15 18:29:30 +08:00
e5339a4014 [feature](ES Catalog)Support control scroll level by config #37180 (#37290)
## Proposed changes

backport #37180
2024-07-15 16:41:38 +08:00
7bd6818350 [branch-2.1][improvement](jdbc catalog) Added support for Oracle Raw type (#37776)
pick (#37078)
In previous versions, we adopted the strategy of reading the object
address for Oracle's raw type, which would lead to unstable and
meaningless results. Here I changed it to read hexadecimal or UTF8
2024-07-15 14:43:05 +08:00
2759383365 [branch-2.1](timezone) refactor tzdata load to accelerate and unify timezone parsing (#37062) (#37269)
pick https://github.com/apache/doris/pull/37062

1. revert https://github.com/apache/doris/pull/25097. we decide to rely
on OS. not maintain independent tzdata anymore to keep result
consistency
2. refactor timezone load. removed rwlock.

before:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (6.88 sec)
```
now:
```sql
mysql [optest]>select count(convert_tz(d, 'Asia/Shanghai', 'America/Los_Angeles')), count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) from dates;
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
| count(convert_tz(cast(d as DATETIMEV2(6)), 'Asia/Shanghai', 'America/Los_Angeles')) | count(convert_tz(dt, 'America/Los_Angeles', '+00:00')) |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
|                                                                            16000000 |                                               16000000 |
+-------------------------------------------------------------------------------------+--------------------------------------------------------+
1 row in set (2.61 sec)
```
3. now don't support timezone offset format string like 'UTC+8', like we
already said in
https://doris.apache.org/docs/dev/query/query-variables/time-zone/#usage
4. support case-insensitive timezone parsing in nereids.
5. a bug when parse timezone using nereids. should check DST by input,
but wrongly by now before. now fixed.

doc pr: https://github.com/apache/doris-website/pull/810
2024-07-15 10:56:48 +08:00
8de13c5cc8 [fix](function) error scale set in unix_timestamp (#36110) (#37619)
## Proposed changes

```
mysql [test]>set DEBUG_SKIP_FOLD_CONSTANT = true;
Query OK, 0 rows affected (0.00 sec)

mysql [test]>select cast(unix_timestamp("2024-01-01",'yyyy-MM-dd') as bigint);
+------------------------------------------------------------+
| cast(unix_timestamp('2024-01-01', 'yyyy-MM-dd') as BIGINT) |
+------------------------------------------------------------+
|                                           1704038400000000 |
+------------------------------------------------------------+
```
now
```
mysql [test]>select cast(unix_timestamp("2024-01-01",'yyyy-MM-dd') as bigint);
+------------------------------------------------------------+
| cast(unix_timestamp('2024-01-01', 'yyyy-MM-dd') as BIGINT) |
+------------------------------------------------------------+
|                                                 1704038400 |
+------------------------------------------------------------+
1 row in set (0.01 sec)
```

The column does not have a scale set, but the cast uses the scale to
perform the cast.


<!--Describe your changes.-->

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-07-15 10:00:04 +08:00
b55dd6f644 [fix](delete) fix the error message for valid decimal data for 2.1 (#37710)
## Proposed changes

cherry-pick : #36802

<!--Describe your changes.-->
2024-07-15 09:54:42 +08:00
16de141743 [regression](kerberos)add hive kerberos docker regression env (#37657)
## Proposed changes
pick:
[regression](kerberos)fix regression pipeline env when write hosts 
(#37057)
[regression](kerberos)add hive kerberos docker regression env (#36430)
2024-07-15 09:35:39 +08:00
56a207c3f0 [case](paimon/iceberg)move cases from p2 to p0 (#37276) (#37738)
bp #37276

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-07-13 10:01:05 +08:00
f2556ba182 [feature](insert)support external hive truncate table DDL (#37659)
pick: #36801
2024-07-12 22:37:47 +08:00
ffa9e49bc7 [feature](mtmv) pick some mtmv pr from master (#37651)
cherry-pick from master
pr: #36318
commitId: c1999479

pr: #36111
commitId: 35ebef62

pr: #36175
commitId: 4c8e66b4

pr: #36414
commitId: 5e009b5a

pr: #36770
commitId: 19e2126c

pr: #36567
commitId: 3da83514
2024-07-12 10:35:54 +08:00
6214d6421f [Fix](planner) fix bug of char(255) toSql (#37340) (#37671)
cherry-pick #37340 from master
2024-07-12 10:33:24 +08:00
4dc933bb28 [cherry-pick] (branch-2.1) fix query errors caused by ignore_above (#37685)
## Proposed changes
pick from master #37679
2024-07-12 09:31:45 +08:00
79a208259e [cherry-pick] (branch-2.1) Remove the check for inverted index file exists #36945 (#37423) 2024-07-11 21:35:52 +08:00
217eac790b [pick](Variant) pick some refactor and fix #34925 #36317 #36201 #36793 (#37526) 2024-07-11 21:25:34 +08:00
d7cae940d2 [fix](test) fix case conflict between test_tvf_based_broker_load and test_broker_load #37622 (#37631)
cherry pick from #37622
2024-07-11 17:52:21 +08:00