Commit Graph

3444 Commits

Author SHA1 Message Date
182f37f837 [fix](planner) NullLiteral should always having a correct Type and set to be analyzed (#43371) 2024-11-10 10:10:40 +08:00
f6ddd818cd [tests](inverted index)Add and improve add/drop index cases (#36811) (#43463)
bp #36811
2024-11-08 09:38:54 +08:00
7de7541ec6 branch-2.1: [fix](ES Catalog)Only like on keyword can be applied to wildcard query (#43400)
Cherry-picked from #41176

Co-authored-by: qiye <jianliang5669@gmail.com>
2024-11-07 22:43:42 +08:00
12bb38eab3 [fix](build index)Fix build index failed on renamed column (#42882)(#43246)(#43336) (#43266)
bp #42882 #43246 #43336
We pick these PRs together because they all fix one problem and if not
do so, the regression test won't be passed.

---------

Co-authored-by: Sun Chenyang <csun5285@gmail.com>
Co-authored-by: qidaye <luen@selectdb.com>
2024-11-07 21:32:02 +08:00
c3e6702d28 branch-2.1: [fix](array-funcs)fix array funcs which ret can be nullable (#43354)
Cherry-picked from #43243

Co-authored-by: amory <wangqiannan@selectdb.com>
2024-11-07 18:28:50 +08:00
876fae6f71 [fix](regression) test_primary_key_partial_update fail due to the U.S. change from DST to standard time (#43288) (#43300)
cherry-pick #43288
2024-11-06 14:10:29 +08:00
46afbfca01 branch-2.1: [fix](ip) fix datatype serde for ipv6 with rowstore (#43252)
Cherry-picked from #43065

Co-authored-by: amory <wangqiannan@selectdb.com>
2024-11-05 20:09:14 +08:00
09f2516c7c [fix](test) fix some test cases #43217 (#43216)
bp #43217
2024-11-04 22:15:30 +08:00
98d3db03b1 [fix](regression-test) fix test_hive_serde_prop #42886 (#43098)
cherry pick from #42886

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-11-01 23:11:45 +08:00
c5374be1bc [cherry-pick](branch-2.1) Pick "[Feature](schema change) Support add column bitmap with default value bitmap_empty (#42331)" (#42702)
Pick #42331
2024-11-01 21:46:38 +08:00
355170a921 [cherry-pick](branch2.1) impl scalar functions trim_in、ltrim_in and rtrim_in (#42641)
pick https://github.com/apache/doris/pull/41681
2024-11-01 09:55:50 +08:00
e85e6c89ba [fix](auditlog) add missing audit log fields and duplicate audit log error #42262 (#43015)
bp #42262
2024-11-01 08:24:55 +08:00
c573351e4e [fix](tvf) fix FE cannot start when replay alter view from tvf (#42866)
bp: #40872
2024-10-31 14:15:48 +08:00
6c3d42e09a [cherry-pick](branch-21) cherry-pick pr about (#42488) (#42099) (#42055) (#42916)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-31 14:14:19 +08:00
cc30a7e78e [fix](test) fix some unstable external p0 test cases (#42685) (#42943)
cherry-pick #42685

Co-authored-by: daidai <2017501503@qq.com>
2024-10-31 12:36:03 +08:00
cdd32d9582 [enhance](hive) support reading hive table with OpenCSVSerde #42257 (#42940)
cherry pick from #42257

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-31 11:12:07 +08:00
0d008b5a43 [feat](nereids)disable join reorder if column stats is invalid #41790 (branch-2.1) (#42902)
## Proposed changes
pick #41790
Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-30 23:47:03 +08:00
e43d255815 [fix](array) fix add_array_null_value for inverted index_builder (#42899)
## Proposed changes
Pick #42638 to doris branch 2.1
Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-30 19:16:59 +08:00
2042726ceb [Fix](regression case) fix some unstable regression cases (#42601)
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-29 18:12:40 +08:00
0baca54200 [fix](Nereids) offset do more than once when have shuffle after limit (#42576) (#42577)
pick from master #42576

intro by #39316. it want to fix a problem intro by #36699. but forgot to
remove all wrong code in #36699.

after #39316, we should not set offset on exchange, when the exchange is
on the top of a limit with offset.
2024-10-28 19:48:13 +08:00
9eef393e2a [pick]support cgroup v2 (#42465)
## Proposed changes

pick #39991   #39374  #36663
2024-10-25 20:13:27 +08:00
4a62d9e44b Revert "[2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool" (#42481)
Reverts apache/doris#42255

We have found that after closing the connection pool, there will be
class loading problems and connection release problems for some data
sources. We will remove this function first and re-add it after solving
and testing it completely.
2024-10-25 19:37:36 +08:00
0700449e8b [fix](schema change) fix delete predicate incorrect comparisons result with empty strings during schema change (#41064) (#42427)
```
insert into t1 (k1,  k2,  v1) value(1, '', 2);
delete form t1 where k1 = 1 and k2 = '';
alter table modify column v1 string
select * from t1 //  expect 0 rows in return, but get 1 row.
```
2024-10-25 01:05:55 +08:00
cbdaaa62b2 [feature](function) hour/minute/second functions support time as an a… 41008 (#42232)
…rgument. (#41008)

## Proposed changes

```
mysql> select cast(4562632 as time),hour(cast(4562632 as time)), minute(cast(4562632 as time)),second(cast(4562632 as time));
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
| cast(4562632 as TIME) | hour(cast(4562632 as TIME)) | minute(cast(4562632 as TIME)) | second(cast(4562632 as TIME)) |
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
| 456:26:32             |                         456 |                            26 |                            32 |
+-----------------------+-----------------------------+-------------------------------+-------------------------------+
```

<!--Describe your changes.-->

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: Dongyang Li <hello_stephen@qq.com>
2024-10-24 11:09:36 +08:00
a3510d3b73 [cherry-pick](branch-21)fix the exchange not skip offset rows (#42190) (#42330)
cherry-pick from master (#42190)
2024-10-24 11:08:17 +08:00
73fc5c734e [fix](mtmv)fix in the scenario of recreating a table, the materialize… (#42339)
…d view may assume that the data has not changed (#41762)

pick: https://github.com/apache/doris/pull/41762
2024-10-24 11:07:29 +08:00
64d0c55e1d [fix](decimal256) support decimal256 for many functions (#42136) (#42353)
BP #42136
2024-10-24 10:52:28 +08:00
e3b059e339 [fix](nereids) Fix not check column name when create or alter view (#42206) (#42323)
pr: https://github.com/apache/doris/pull/42206
commitId: 2bcaa5b4
2024-10-24 10:11:32 +08:00
44e9368c78 Pick some fix from master to 21(#41472) (#40106)(#40173) (#42212)
## Proposed changes

pr: https://github.com/apache/doris/pull/41472
commitId: 2745e044


pr: https://github.com/apache/doris/pull/40106
commitId: 0fdb1ee0


pr: https://github.com/apache/doris/pull/40173
commitId: 0d07e3d1
2024-10-24 10:09:55 +08:00
a5b3520cbb [fix](mtmv) regression test unstable and error (#41145) (#42185)
## Proposed changes

pr: https://github.com/apache/doris/pull/41145
commitId: 5e6e4bf6
2024-10-23 14:42:53 +08:00
e7395198d4 [fix](estimate) func call with not filter will estimate some statisti… (#42302)
…cs (#41989)

before this pr: use ! or not in or condition when table has been
analyzed we will meet
```
SELECT
  count(1)
FROM
  table_30_un_pa_ke_pr_di4
where
  col_int_undef_signed_not_null < -128
  or not array_contains(col_array_bigint__undef_signed, col_int_undef_signed_not_null);

ERROR 1105 (HY000): errCode = 2, detailMessage = Not-predicate meet unexpected child:
  array_contains(col_array_bigint__undef_signed, cast(col_int_undef_signed_not_null as BIGINT))
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-23 14:33:41 +08:00
2defa90be7 [test](ES Catalog)Add mapping _routing test case (#42074) (#42282)
## Proposed changes

bp #42074
2024-10-23 10:14:12 +08:00
c9acd71ad6 [fix](inverted index) Fix errors caused by enable_need_read_data_opt #42064 (#42247)
cherry pick from #42064

Co-authored-by: Sun Chenyang <csun5285@gmail.com>
2024-10-23 09:18:19 +08:00
bf90d918ca [fix](regression-test) Make test_decimal256_outfile_csv export the data to S3 rather than local file system (#36592)(#42211) (#42270)
bp  (#36592)(#42211)

---------

Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
2024-10-22 23:57:30 +08:00
157d67e7ca [enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug #42200 (#42273)
cherry pick from #42200

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-22 23:56:57 +08:00
bde8e2d474 [2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool (#42255)
pick (#41992)

We initially introduced jdbc connection pool to improve the connection
performance of jdbc catalog, but we always found that connection pool
would bring some unexpected errors, so we chose to add a catalog
property: `enable_connection_pool` to choose whether to enable the jdbc
connection pool of jdbc catalog, and the default false.However, the
created catalog will still open the connection pool when it is upgraded,
and only the newly created catalog will be false

And we conducted performance tests on this, the performance loss is
within the expected range.

- Enable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM mysql.test.test
limit 1;' --create-schema=mysql --delimiter=";" --verbose
Benchmark
        Average number of seconds to run all queries: 0.008 seconds
        Minimum number of seconds to run all queries: 0.004 seconds
        Maximum number of seconds to run all queries: 0.133 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1

- Disable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM
mysql_no_pool.test.test limit 1;' --create-schema=mysql --delimiter=";"
--verbose
Benchmark
        Average number of seconds to run all queries: 0.054 seconds
        Minimum number of seconds to run all queries: 0.047 seconds
        Maximum number of seconds to run all queries: 0.184 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1
2024-10-22 23:28:28 +08:00
e2bdac39fb [fix] Implementing match_phrase_edge without index query method (#41658) (#42098)
pick from #41658
2024-10-22 18:44:21 +08:00
c1d2b8d548 [2.1][improvement](jdbc catalog) Disallow non-constant type conversion pushdown and implicit conversion pushdown (#42242)
pick (#42102)

Add a variable `enable_jdbc_cast_predicate_push_down`, the default value
is false, which prohibits the pushdown of non-constant predicates with
type conversion and all predicates with implicit conversion. This change
can prevent the wrong predicates from being pushed down to the Jdbc data
source, resulting in query data errors, because the predicates with cast
were not correctly pushed down to the data source before. If you find
that the data is read correctly and the performance is better before
this change, you can manually set this variable to true

```
| Expression                                          | Can Push Down |
|-----------------------------------------------------|---------------|
| column type equals const type                       | Yes           |
| column type equals cast const type                  | Yes           |
| cast column type equals const type                  | No            |
| cast column type equals cast const type             | No            |
| column type not equals column type                  | No            |
| column type not equals cast const type              | No            |
| cast column type not equals const type              | No            |
| cast column type not equals cast const type         | No            |

```
2024-10-22 17:27:29 +08:00
7eec0f8fbb [branch-2.1](datetime) Fix date floor functions overflow (#35477) (#42238)
pick https://github.com/apache/doris/pull/35477
2024-10-22 15:54:53 +08:00
9c4ce73dfa [Pick](nereids) pick 40529 41464 40349 (#42073)
## Proposed changes
pick  #40529 #41464 #40349

Issue Number: close #xxx

<!--Describe your changes.-->
2024-10-22 15:04:50 +08:00
85a98df9c2 [Fix](count on index) fix count on index opt when count project expr #41772 (#42229)
cherry pick from #41772
2024-10-22 13:16:32 +08:00
47ff6f1300 [fix](OrcReader) fix the issue that orc_reader can not read DECIMAL(0,0) type of orc file #41795 (#42220)
cherry pick from #41795

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
2024-10-22 10:10:25 +08:00
1f8d685f26 [fix](inverted index) multi_match supports any, all, phrase. (#41663) (#42097)
https://github.com/apache/doris/pull/41663
2024-10-22 10:10:02 +08:00
e713b92321 [fix](multi-catalog) Disable string dictionary filtering when predicate express is not slot #42113 (#42222)
cherry pick from #42113

Co-authored-by: Socrates <suxiaogang223@icloud.com>
2024-10-22 09:43:29 +08:00
084434e25c [Test](tvf) add regression tests for testing orc reader #41606 #42188 (#42120)
cherry pick from #42031 #42188

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: TieweiFang <ftw2139@163.com>
2024-10-21 21:31:18 +08:00
bbd4970ed8 [feature](jdbc catalog) support gbase jdbc catalog #41027 #41587 (#42123)
cherry pick from #41027 #41587

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-10-21 16:52:23 +08:00
a32ad0b1f7 [cherry-pick](branch-2.1) support reading brotli compressed parquet file (#42162)
pick pr: https://github.com/apache/doris/pull/41875
2024-10-21 16:48:09 +08:00
a150d160ea [fix](jdbc catalog) fix and add mysql and doris extremum test #41679 (#42122)
cherry pick from #41679

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2024-10-21 16:39:40 +08:00
5ba0da4a84 [fix](test) fix unstable external p0 cases #42069 (#42153)
cherry pick from #42069
2024-10-21 15:04:40 +08:00
29f29f494f [chore](test) add log to find bug of unstable test (#37982) (#42129) 2024-10-19 18:00:57 +08:00