database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,619 Commits 1 Branch 0 Tags
443e87e20327eaa5577cc10f08a63ec1694de358
Commit Graph

20619 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhangdong
73fc5c734e [fix](mtmv)fix in the scenario of recreating a table, the materialize… (#42339) 
…d view may assume that the data has not changed (#41762)

pick: https://github.com/apache/doris/pull/41762
 2024-10-24 11:07:29 +08:00
TengJianPing
64d0c55e1d [fix](decimal256) support decimal256 for many functions (#42136) (#42353) 
BP #42136
 2024-10-24 10:52:28 +08:00
camby
32b13b285c [fix](index compaction) fix fd leak and mem leak while index compaction (#41915) (#42171) 
pick #41915 to branch-2.1
 2024-10-24 10:40:10 +08:00
seawinde
e3b059e339 [fix](nereids) Fix not check column name when create or alter view (#42206) (#42323) 
pr: https://github.com/apache/doris/pull/42206
commitId: 2bcaa5b4
 2024-10-24 10:11:32 +08:00
seawinde
44e9368c78 Pick some fix from master to 21(#41472) (#40106)(#40173) (#42212) 
## Proposed changes

pr: https://github.com/apache/doris/pull/41472
commitId: 2745e044


pr: https://github.com/apache/doris/pull/40106
commitId: 0fdb1ee0


pr: https://github.com/apache/doris/pull/40173
commitId: 0d07e3d1
 2024-10-24 10:09:55 +08:00
hui lai
0d76c16d14 [fix](routine load) reduce routine load task consume log (#42058) (#42268) 
pick (#42058)

There is too much routine load task log, five million logs were
generated in 10 minutes.

```
grep 'consumer meet partition eof' be.INFO.log.20240930-164533 | wc -l 
5369624
```
 2024-10-23 23:30:15 +08:00
Xin Liao
86fa6a37c7 [fix](load) Fix the issue of high-concurrency single-replica load getting stuck #42297 (#42317) 
cherry pick from #42297
 2024-10-23 23:28:05 +08:00
airborne12
ccff47e813 [Fix](inverted index) fix use after free when duplicate key in index dir when index file writer open index #42207 (#42301) 
cherry pick from #42207
 2024-10-23 16:45:25 +08:00
hui lai
036906f025 [fix](routine load) make the timeout of load channel consistent with routine load task (#42042) (#42267) 
pick (#42042)

Routine load task timeout is max_batch_interval * 10, but load channel
timeout is max_batch_interval * 2.
 2024-10-23 16:17:39 +08:00
yiguolei
0aa07e6118 [enhancement](blacklist) should check the real error reason when sendfragment timeout (#42314) 
## Proposed changes

1. send fragment in BE may run for long time.
2. timeout is not related with BE down.
3. cancel logic should not handle blacklist. because if a BE is down,
send fragment will find it.

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-10-23 16:16:04 +08:00
qiye
bc84734291 [opt](index compaction)Optimize logic of picking columns for index compaction(#42051) (#42287) 
## Proposed changes

bp #42051
 2024-10-23 15:21:23 +08:00
seawinde
a5b3520cbb [fix](mtmv) regression test unstable and error (#41145) (#42185) 
## Proposed changes

pr: https://github.com/apache/doris/pull/41145
commitId: 5e6e4bf6
 2024-10-23 14:42:53 +08:00
amory
e7395198d4 [fix](estimate) func call with not filter will estimate some statisti… (#42302) 
…cs (#41989)

before this pr: use ! or not in or condition when table has been
analyzed we will meet
```
SELECT
  count(1)
FROM
  table_30_un_pa_ke_pr_di4
where
  col_int_undef_signed_not_null < -128
  or not array_contains(col_array_bigint__undef_signed, col_int_undef_signed_not_null);

ERROR 1105 (HY000): errCode = 2, detailMessage = Not-predicate meet unexpected child:
  array_contains(col_array_bigint__undef_signed, cast(col_int_undef_signed_not_null as BIGINT))
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-23 14:33:41 +08:00
qiye
2defa90be7 [test](ES Catalog)Add mapping _routing test case (#42074) (#42282) 
## Proposed changes

bp #42074
 2024-10-23 10:14:12 +08:00
airborne12
c9acd71ad6 [fix](inverted index) Fix errors caused by enable_need_read_data_opt #42064 (#42247) 
cherry pick from #42064

Co-authored-by: Sun Chenyang <csun5285@gmail.com>
 2024-10-23 09:18:19 +08:00
Rayner Chen
bf90d918ca [fix](regression-test) Make test_decimal256_outfile_csv export the data to S3 rather than local file system (#36592)(#42211) (#42270) 
bp  (#36592)(#42211)

---------

Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-22 23:57:30 +08:00
Rayner Chen
157d67e7ca [enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug #42200 (#42273) 
cherry pick from #42200

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-10-22 23:56:57 +08:00
Rayner Chen
d7c3369ce7 [regression](case)fix mc regression test p2 case. #42217 (#42274) 
cherry pick from #42217

Co-authored-by: daidai <2017501503@qq.com>
 2024-10-22 23:43:51 +08:00
zy-kkk
bde8e2d474 [2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool (#42255) 
pick (#41992)

We initially introduced jdbc connection pool to improve the connection
performance of jdbc catalog, but we always found that connection pool
would bring some unexpected errors, so we chose to add a catalog
property: `enable_connection_pool` to choose whether to enable the jdbc
connection pool of jdbc catalog, and the default false.However, the
created catalog will still open the connection pool when it is upgraded,
and only the newly created catalog will be false

And we conducted performance tests on this, the performance loss is
within the expected range.

- Enable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM mysql.test.test
limit 1;' --create-schema=mysql --delimiter=";" --verbose
Benchmark
        Average number of seconds to run all queries: 0.008 seconds
        Minimum number of seconds to run all queries: 0.004 seconds
        Maximum number of seconds to run all queries: 0.133 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1

- Disable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM
mysql_no_pool.test.test limit 1;' --create-schema=mysql --delimiter=";"
--verbose
Benchmark
        Average number of seconds to run all queries: 0.054 seconds
        Minimum number of seconds to run all queries: 0.047 seconds
        Maximum number of seconds to run all queries: 0.184 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1
 2024-10-22 23:28:28 +08:00
zy-kkk
1fc1662e88 [2.1][fix](external) Fix case-insensitive table name mapping retrieval rules (#42259) 
pick (#38227)

```
Doris > CREATE CATALOG `oracle` PROPERTIES (
    -> "user" = "doris_test",
    -> "type" = "jdbc",
    -> "password" = "xxx",
    -> "jdbc_url" = "jdbc:oracle:thin:@xxx:1521:XE",
    -> "driver_url" = "ojdbc8-19.3.0.0.jar",
    -> "driver_class" = "oracle.jdbc.driver.OracleDriver"
    -> );
Query OK, 0 rows affected (2.16 sec)
```

```
Doris > show variables like '%lower_case%';
+------------------------+-------+---------------+---------+
| Variable_name          | Value | Default_Value | Changed |
+------------------------+-------+---------------+---------+
| lower_case_table_names | 1     | 0             | 1       |
+------------------------+-------+---------------+---------+
1 row in set (0.00 sec)

Doris > show tables from oracle.DORIS_TEST;
+----------------------+
| Tables_in_DORIS_TEST |
+----------------------+
| aa/d                 |
| aaad                 |
| lower_test           |
| student              |
| student2             |
| student3             |
| test_all_types       |
| test_char            |
| test_clob            |
| test_date            |
| test_insert          |
| test_int             |
| test_num             |
| test_number          |
| test_number2         |
| test_number3         |
| test_number4         |
| test_raw             |
| test_timestamp       |
+----------------------+
19 rows in set (0.01 sec)

```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (1.03 sec)

Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Table [TEST_INT] does not exist in database [DORIS_TEST].

```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)

Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)
```

```
Doris > show variables like '%lower_case%';
+------------------------+-------+---------------+---------+
| Variable_name          | Value | Default_Value | Changed |
+------------------------+-------+---------------+---------+
| lower_case_table_names | 2     | 0             | 1       |
+------------------------+-------+---------------+---------+
1 row in set (0.01 sec)

Doris > show tables from oracle.DORIS_TEST;
+----------------------+
| Tables_in_DORIS_TEST |
+----------------------+
| AA/D                 |
| AAAD                 |
| LOWER_TEST           |
| STUDENT              |
| TEST_ALL_TYPES       |
| TEST_CHAR            |
| TEST_CLOB            |
| TEST_DATE            |
| TEST_INSERT          |
| TEST_INT             |
| TEST_NUM             |
| TEST_NUMBER          |
| TEST_NUMBER2         |
| TEST_NUMBER3         |
| TEST_NUMBER4         |
| TEST_RAW             |
| TEST_TIMESTAMP       |
| student2             |
| student3             |
+----------------------+
19 rows in set (1.05 sec)
```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Table [test_int] does not exist in database [DORIS_TEST].
Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (1.07 sec)
```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.21 sec)

Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-22 23:28:01 +08:00
airborne12
3f71f0e738 [Test](bloom filter) enhance bloom filter test case with retry logic #42243 (#42264) 
cherry pick from #42243
 2024-10-22 21:50:53 +08:00
zy-kkk
56c2a0f523 [2.1][opt](Catalog) Remove unnecessary conjuncts handling on External Scan (#42261) 
pick (#41218)

In the previous FileScanNode, some parts that used conjuncts for
predicate conversion were placed in the init phase. However, for the
Nereids planner, pushing the filter down to the scan happens in the
Translator, which means that the ScanNode can only get the complete
conjuncts in the finalized phase. Therefore, in this PR, I have removed
all conjuncts variables in External for the Nereids planner. They no
longer need to store conjuncts themselves or add them to the ScanNode.
Instead, all places in the ScanNode that use conjuncts should be moved
to the finalized phase.

This refactor also fix a performance issue introduced from #40176 After
introducing the change of generating SelectNode for consecutive projects
or filters, FileScan still adds conjuncts too early in the init phase,
resulting in the discovery of consecutive filters when the upper layer
continues to translate, a selectnode was unexpectedly generated on the
scannode, causing the project to be unable to prune the scannode
columns. However, the Project node trims columns of SelectNode and
ScanNode differently, which causes ScanNode to scan unnecessary columns.

My modification removes the addition of conjuncts in the scannode step,
so that we can keep the structure from ScanNode to Project and achieve
correct column trimming.
 2024-10-22 21:42:55 +08:00
Kaijie Chen
25d7d0b255 [fix](move-memtable) abstract multi-streams to one logical stream (#42039) (#42250) 
backport #42039
 2024-10-22 20:26:42 +08:00
Socrates
38e529cd29 [cherry-pick](branch-2.1) support decimal256 for parquet reader (#42241) 
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41526
 2024-10-22 19:42:09 +08:00
zzzxl
e2bdac39fb [fix] Implementing match_phrase_edge without index query method (#41658) (#42098) 
pick from #41658
 2024-10-22 18:44:21 +08:00
Xinyi Zou
6f2bac012a [pick](branch-2.1) pick #39398 #41754 #41770 (#42231) 
pick #39398 #41754 #41770
 2024-10-22 18:05:40 +08:00
zy-kkk
c1d2b8d548 [2.1][improvement](jdbc catalog) Disallow non-constant type conversion pushdown and implicit conversion pushdown (#42242) 
pick (#42102)

Add a variable `enable_jdbc_cast_predicate_push_down`, the default value
is false, which prohibits the pushdown of non-constant predicates with
type conversion and all predicates with implicit conversion. This change
can prevent the wrong predicates from being pushed down to the Jdbc data
source, resulting in query data errors, because the predicates with cast
were not correctly pushed down to the data source before. If you find
that the data is read correctly and the performance is better before
this change, you can manually set this variable to true

```
| Expression                                          | Can Push Down |
|-----------------------------------------------------|---------------|
| column type equals const type                       | Yes           |
| column type equals cast const type                  | Yes           |
| cast column type equals const type                  | No            |
| cast column type equals cast const type             | No            |
| column type not equals column type                  | No            |
| column type not equals cast const type              | No            |
| cast column type not equals const type              | No            |
| cast column type not equals cast const type         | No            |

```
 2024-10-22 17:27:29 +08:00
zclllhhjj
7eec0f8fbb [branch-2.1](datetime) Fix date floor functions overflow (#35477) (#42238) 
pick https://github.com/apache/doris/pull/35477
 2024-10-22 15:54:53 +08:00
minghong
9c4ce73dfa [Pick](nereids) pick 40529 41464 40349 (#42073) 
## Proposed changes
pick  #40529 #41464 #40349

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-22 15:04:50 +08:00
Gabriel
8877267930 [pipeline](API) Add a new API to find pipeline tasks by a specific qu… (#42233) 
…ery ID (#35563)

pick #35563
 2024-10-22 14:03:45 +08:00
airborne12
85a98df9c2 [Fix](count on index) fix count on index opt when count project expr #41772 (#42229) 
cherry pick from #41772
 2024-10-22 13:16:32 +08:00
starocean999
803c052100 [fix](nereids)modify agg function nullability in PhysicalHashAggregate (#42018) 
## Proposed changes

pick from master https://github.com/apache/doris/pull/41943

<!--Describe your changes.-->
 2024-10-22 11:23:39 +08:00
yagagagaga
104d427afa [cherry-pick][chore](audit) Optimize the SQL (insert into values) length in audit logs (#37894) and let line comment work well (#40599) (#42186) 
## Proposed changes

cherry-pick from master #37894 and #40599

<!--Describe your changes.-->
 2024-10-22 10:16:05 +08:00
Rayner Chen
47ff6f1300 [fix](OrcReader) fix the issue that orc_reader can not read DECIMAL(0,0) type of orc file #41795 (#42220) 
cherry pick from #41795

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-22 10:10:25 +08:00
zzzxl
1f8d685f26 [fix](inverted index) multi_match supports any, all, phrase. (#41663) (#42097) 
https://github.com/apache/doris/pull/41663
 2024-10-22 10:10:02 +08:00
lihangyu
fc857ddf0d [regression-test](suites) use master ip to do server side prepared (#42146) 
(#41981)
 2024-10-22 10:08:28 +08:00
Lei Zhang
8ce65cab86 [chore](log) Adjust log level for replaying a batch editlog cost time(#41392) (#42182) 
* `replay journal cost too much time` is a counter for replaying a batch
editlog it is normal that cost too much time, the warning level can make
confused

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-22 10:06:11 +08:00
Rayner Chen
c6b35ba9b8 [cherry-pick](paimon) pick part of refactor code from #34496 (#42221) 
#34496

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-22 09:44:41 +08:00
Rayner Chen
e713b92321 [fix](multi-catalog) Disable string dictionary filtering when predicate express is not slot #42113 (#42222) 
cherry pick from #42113

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-10-22 09:43:29 +08:00
Rayner Chen
084434e25c [Test](tvf) add regression tests for testing orc reader #41606 #42188 (#42120) 
cherry pick from #42031 #42188

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: TieweiFang <ftw2139@163.com>
 2024-10-21 21:31:18 +08:00
Rayner Chen
ac3d64c010 [feature](hive)support create hive table for text format #41860 (#42195) 
cherry pick from #41860

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-10-21 21:30:11 +08:00
Rayner Chen
e7229c77c8 [fix](config) increase JVM memory of BE #42052 (#42194) 
bp #42052
 2024-10-21 20:14:43 +08:00
Rayner Chen
7568fe41f1 [fix](test) fix unstable external p0 tests #42158 (#42192) 
cherry pick from #42158
 2024-10-21 18:37:30 +08:00
lihangyu
720a4c9f30 [Fix](Branch-2.1) fix fallback to original planer when enable_server_side_prepared_statement = false (#42156) 2024-10-21 17:46:24 +08:00
Socrates
a3c1657c4b [cherry-pick](branch-2.1) check end of file when reading page (#42159) 
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41816
 2024-10-21 17:01:04 +08:00
Rayner Chen
bbd4970ed8 [feature](jdbc catalog) support gbase jdbc catalog #41027 #41587 (#42123) 
cherry pick from #41027 #41587

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
 2024-10-21 16:52:23 +08:00
Socrates
a32ad0b1f7 [cherry-pick](branch-2.1) support reading brotli compressed parquet file (#42162) 
pick pr: https://github.com/apache/doris/pull/41875
 2024-10-21 16:48:09 +08:00
Rayner Chen
a150d160ea [fix](jdbc catalog) fix and add mysql and doris extremum test #41679 (#42122) 
cherry pick from #41679

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
 2024-10-21 16:39:40 +08:00
Rayner Chen
b9e2738ee6 [Fix](orc-reader) Fix StringRef nullptr data by add checking string_values empty. #42061 (#42154) 
cherry pick from #42061

Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
 2024-10-21 16:26:23 +08:00
airborne12
da377e98cf [Pick 2.1](inverted index) enhace test_build_index case for quickly finished jobs (#42151) 
## Proposed changes

bp: #42150
 2024-10-21 15:52:57 +08:00