database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,504 Commits 1 Branch 0 Tags
bf90d918ca634529dcbdb562bc7747af5336dfd5
Commit Graph

20504 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rayner Chen
bf90d918ca [fix](regression-test) Make test_decimal256_outfile_csv export the data to S3 rather than local file system (#36592)(#42211) (#42270) 
bp  (#36592)(#42211)

---------

Co-authored-by: TengJianPing <18241664+jacktengg@users.noreply.github.com>
Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-22 23:57:30 +08:00
Rayner Chen
157d67e7ca [enhance](hive) Add regression-test cases for hive text ddl and hive text insert and fix reading null string bug #42200 (#42273) 
cherry pick from #42200

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-10-22 23:56:57 +08:00
Rayner Chen
d7c3369ce7 [regression](case)fix mc regression test p2 case. #42217 (#42274) 
cherry pick from #42217

Co-authored-by: daidai <2017501503@qq.com>
 2024-10-22 23:43:51 +08:00
zy-kkk
bde8e2d474 [2.1][improvement](jdbc catalog) Add catalog property to enable jdbc connection pool (#42255) 
pick (#41992)

We initially introduced jdbc connection pool to improve the connection
performance of jdbc catalog, but we always found that connection pool
would bring some unexpected errors, so we chose to add a catalog
property: `enable_connection_pool` to choose whether to enable the jdbc
connection pool of jdbc catalog, and the default false.However, the
created catalog will still open the connection pool when it is upgraded,
and only the newly created catalog will be false

And we conducted performance tests on this, the performance loss is
within the expected range.

- Enable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM mysql.test.test
limit 1;' --create-schema=mysql --delimiter=";" --verbose
Benchmark
        Average number of seconds to run all queries: 0.008 seconds
        Minimum number of seconds to run all queries: 0.004 seconds
        Maximum number of seconds to run all queries: 0.133 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1

- Disable connection pool: mysqlslap -uroot -h127.0.0.1 -P9030
--concurrency=1 --iterations=100 --query='SELECT * FROM
mysql_no_pool.test.test limit 1;' --create-schema=mysql --delimiter=";"
--verbose
Benchmark
        Average number of seconds to run all queries: 0.054 seconds
        Minimum number of seconds to run all queries: 0.047 seconds
        Maximum number of seconds to run all queries: 0.184 seconds
        Number of clients running queries: 1
        Average number of queries per client: 1
 2024-10-22 23:28:28 +08:00
zy-kkk
1fc1662e88 [2.1][fix](external) Fix case-insensitive table name mapping retrieval rules (#42259) 
pick (#38227)

```
Doris > CREATE CATALOG `oracle` PROPERTIES (
    -> "user" = "doris_test",
    -> "type" = "jdbc",
    -> "password" = "xxx",
    -> "jdbc_url" = "jdbc:oracle:thin:@xxx:1521:XE",
    -> "driver_url" = "ojdbc8-19.3.0.0.jar",
    -> "driver_class" = "oracle.jdbc.driver.OracleDriver"
    -> );
Query OK, 0 rows affected (2.16 sec)
```

```
Doris > show variables like '%lower_case%';
+------------------------+-------+---------------+---------+
| Variable_name          | Value | Default_Value | Changed |
+------------------------+-------+---------------+---------+
| lower_case_table_names | 1     | 0             | 1       |
+------------------------+-------+---------------+---------+
1 row in set (0.00 sec)

Doris > show tables from oracle.DORIS_TEST;
+----------------------+
| Tables_in_DORIS_TEST |
+----------------------+
| aa/d                 |
| aaad                 |
| lower_test           |
| student              |
| student2             |
| student3             |
| test_all_types       |
| test_char            |
| test_clob            |
| test_date            |
| test_insert          |
| test_int             |
| test_num             |
| test_number          |
| test_number2         |
| test_number3         |
| test_number4         |
| test_raw             |
| test_timestamp       |
+----------------------+
19 rows in set (0.01 sec)

```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (1.03 sec)

Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Table [TEST_INT] does not exist in database [DORIS_TEST].

```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)

Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)
```

```
Doris > show variables like '%lower_case%';
+------------------------+-------+---------------+---------+
| Variable_name          | Value | Default_Value | Changed |
+------------------------+-------+---------------+---------+
| lower_case_table_names | 2     | 0             | 1       |
+------------------------+-------+---------------+---------+
1 row in set (0.01 sec)

Doris > show tables from oracle.DORIS_TEST;
+----------------------+
| Tables_in_DORIS_TEST |
+----------------------+
| AA/D                 |
| AAAD                 |
| LOWER_TEST           |
| STUDENT              |
| TEST_ALL_TYPES       |
| TEST_CHAR            |
| TEST_CLOB            |
| TEST_DATE            |
| TEST_INSERT          |
| TEST_INT             |
| TEST_NUM             |
| TEST_NUMBER          |
| TEST_NUMBER2         |
| TEST_NUMBER3         |
| TEST_NUMBER4         |
| TEST_RAW             |
| TEST_TIMESTAMP       |
| student2             |
| student3             |
+----------------------+
19 rows in set (1.05 sec)
```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = Table [test_int] does not exist in database [DORIS_TEST].
Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (1.07 sec)
```

```
Doris > select * from oracle.DORIS_TEST.test_int limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.21 sec)

Doris > select * from oracle.DORIS_TEST.TEST_INT limit 1;
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| ID   | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
| 1    |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 |
+------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+
1 row in set (0.20 sec)
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-22 23:28:01 +08:00
airborne12
3f71f0e738 [Test](bloom filter) enhance bloom filter test case with retry logic #42243 (#42264) 
cherry pick from #42243
 2024-10-22 21:50:53 +08:00
zy-kkk
56c2a0f523 [2.1][opt](Catalog) Remove unnecessary conjuncts handling on External Scan (#42261) 
pick (#41218)

In the previous FileScanNode, some parts that used conjuncts for
predicate conversion were placed in the init phase. However, for the
Nereids planner, pushing the filter down to the scan happens in the
Translator, which means that the ScanNode can only get the complete
conjuncts in the finalized phase. Therefore, in this PR, I have removed
all conjuncts variables in External for the Nereids planner. They no
longer need to store conjuncts themselves or add them to the ScanNode.
Instead, all places in the ScanNode that use conjuncts should be moved
to the finalized phase.

This refactor also fix a performance issue introduced from #40176 After
introducing the change of generating SelectNode for consecutive projects
or filters, FileScan still adds conjuncts too early in the init phase,
resulting in the discovery of consecutive filters when the upper layer
continues to translate, a selectnode was unexpectedly generated on the
scannode, causing the project to be unable to prune the scannode
columns. However, the Project node trims columns of SelectNode and
ScanNode differently, which causes ScanNode to scan unnecessary columns.

My modification removes the addition of conjuncts in the scannode step,
so that we can keep the structure from ScanNode to Project and achieve
correct column trimming.
 2024-10-22 21:42:55 +08:00
Kaijie Chen
25d7d0b255 [fix](move-memtable) abstract multi-streams to one logical stream (#42039) (#42250) 
backport #42039
 2024-10-22 20:26:42 +08:00
Socrates
38e529cd29 [cherry-pick](branch-2.1) support decimal256 for parquet reader (#42241) 
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41526
 2024-10-22 19:42:09 +08:00
zzzxl
e2bdac39fb [fix] Implementing match_phrase_edge without index query method (#41658) (#42098) 
pick from #41658
 2024-10-22 18:44:21 +08:00
Xinyi Zou
6f2bac012a [pick](branch-2.1) pick #39398 #41754 #41770 (#42231) 
pick #39398 #41754 #41770
 2024-10-22 18:05:40 +08:00
zy-kkk
c1d2b8d548 [2.1][improvement](jdbc catalog) Disallow non-constant type conversion pushdown and implicit conversion pushdown (#42242) 
pick (#42102)

Add a variable `enable_jdbc_cast_predicate_push_down`, the default value
is false, which prohibits the pushdown of non-constant predicates with
type conversion and all predicates with implicit conversion. This change
can prevent the wrong predicates from being pushed down to the Jdbc data
source, resulting in query data errors, because the predicates with cast
were not correctly pushed down to the data source before. If you find
that the data is read correctly and the performance is better before
this change, you can manually set this variable to true

```
| Expression                                          | Can Push Down |
|-----------------------------------------------------|---------------|
| column type equals const type                       | Yes           |
| column type equals cast const type                  | Yes           |
| cast column type equals const type                  | No            |
| cast column type equals cast const type             | No            |
| column type not equals column type                  | No            |
| column type not equals cast const type              | No            |
| cast column type not equals const type              | No            |
| cast column type not equals cast const type         | No            |

```
 2024-10-22 17:27:29 +08:00
zclllhhjj
7eec0f8fbb [branch-2.1](datetime) Fix date floor functions overflow (#35477) (#42238) 
pick https://github.com/apache/doris/pull/35477
 2024-10-22 15:54:53 +08:00
minghong
9c4ce73dfa [Pick](nereids) pick 40529 41464 40349 (#42073) 
## Proposed changes
pick  #40529 #41464 #40349

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-22 15:04:50 +08:00
Gabriel
8877267930 [pipeline](API) Add a new API to find pipeline tasks by a specific qu… (#42233) 
…ery ID (#35563)

pick #35563
 2024-10-22 14:03:45 +08:00
airborne12
85a98df9c2 [Fix](count on index) fix count on index opt when count project expr #41772 (#42229) 
cherry pick from #41772
 2024-10-22 13:16:32 +08:00
starocean999
803c052100 [fix](nereids)modify agg function nullability in PhysicalHashAggregate (#42018) 
## Proposed changes

pick from master https://github.com/apache/doris/pull/41943

<!--Describe your changes.-->
 2024-10-22 11:23:39 +08:00
yagagagaga
104d427afa [cherry-pick][chore](audit) Optimize the SQL (insert into values) length in audit logs (#37894) and let line comment work well (#40599) (#42186) 
## Proposed changes

cherry-pick from master #37894 and #40599

<!--Describe your changes.-->
 2024-10-22 10:16:05 +08:00
Rayner Chen
47ff6f1300 [fix](OrcReader) fix the issue that orc_reader can not read DECIMAL(0,0) type of orc file #41795 (#42220) 
cherry pick from #41795

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-22 10:10:25 +08:00
zzzxl
1f8d685f26 [fix](inverted index) multi_match supports any, all, phrase. (#41663) (#42097) 
https://github.com/apache/doris/pull/41663
 2024-10-22 10:10:02 +08:00
lihangyu
fc857ddf0d [regression-test](suites) use master ip to do server side prepared (#42146) 
(#41981)
 2024-10-22 10:08:28 +08:00
Lei Zhang
8ce65cab86 [chore](log) Adjust log level for replaying a batch editlog cost time(#41392) (#42182) 
* `replay journal cost too much time` is a counter for replaying a batch
editlog it is normal that cost too much time, the warning level can make
confused

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-22 10:06:11 +08:00
Rayner Chen
c6b35ba9b8 [cherry-pick](paimon) pick part of refactor code from #34496 (#42221) 
#34496

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-22 09:44:41 +08:00
Rayner Chen
e713b92321 [fix](multi-catalog) Disable string dictionary filtering when predicate express is not slot #42113 (#42222) 
cherry pick from #42113

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-10-22 09:43:29 +08:00
Rayner Chen
084434e25c [Test](tvf) add regression tests for testing orc reader #41606 #42188 (#42120) 
cherry pick from #42031 #42188

---------

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
Co-authored-by: TieweiFang <ftw2139@163.com>
 2024-10-21 21:31:18 +08:00
Rayner Chen
ac3d64c010 [feature](hive)support create hive table for text format #41860 (#42195) 
cherry pick from #41860

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-10-21 21:30:11 +08:00
Rayner Chen
e7229c77c8 [fix](config) increase JVM memory of BE #42052 (#42194) 
bp #42052
 2024-10-21 20:14:43 +08:00
Rayner Chen
7568fe41f1 [fix](test) fix unstable external p0 tests #42158 (#42192) 
cherry pick from #42158
 2024-10-21 18:37:30 +08:00
lihangyu
720a4c9f30 [Fix](Branch-2.1) fix fallback to original planer when enable_server_side_prepared_statement = false (#42156) 2024-10-21 17:46:24 +08:00
Socrates
a3c1657c4b [cherry-pick](branch-2.1) check end of file when reading page (#42159) 
## Proposed changes
pick pr: https://github.com/apache/doris/pull/41816
 2024-10-21 17:01:04 +08:00
Rayner Chen
bbd4970ed8 [feature](jdbc catalog) support gbase jdbc catalog #41027 #41587 (#42123) 
cherry pick from #41027 #41587

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
 2024-10-21 16:52:23 +08:00
Socrates
a32ad0b1f7 [cherry-pick](branch-2.1) support reading brotli compressed parquet file (#42162) 
pick pr: https://github.com/apache/doris/pull/41875
 2024-10-21 16:48:09 +08:00
Rayner Chen
a150d160ea [fix](jdbc catalog) fix and add mysql and doris extremum test #41679 (#42122) 
cherry pick from #41679

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
 2024-10-21 16:39:40 +08:00
Rayner Chen
b9e2738ee6 [Fix](orc-reader) Fix StringRef nullptr data by add checking string_values empty. #42061 (#42154) 
cherry pick from #42061

Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
 2024-10-21 16:26:23 +08:00
airborne12
da377e98cf [Pick 2.1](inverted index) enhace test_build_index case for quickly finished jobs (#42151) 
## Proposed changes

bp: #42150
 2024-10-21 15:52:57 +08:00
Rayner Chen
5ba0da4a84 [fix](test) fix unstable external p0 cases #42069 (#42153) 
cherry pick from #42069
 2024-10-21 15:04:40 +08:00
Rayner Chen
9ac8b44d65 [bugfix](hive)Use the connected user to initialize the owner of the hive table #41876 (#42121) 
cherry pick from #41876

---------

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-10-21 14:41:13 +08:00
Rayner Chen
1eea8dcb2c [enhance](thirdparty) add boritli headers in thirdparty (#42017) (#42110) (#42124) 
bp  (#42017) (#42110)

---------

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-10-21 09:51:11 +08:00
924060929
29f29f494f [chore](test) add log to find bug of unstable test (#37982) (#42129) 2024-10-19 18:00:57 +08:00
yiguolei
dc438649d9 [bugfix](handshake) brpc handshake should not use light pool (#42115) (#42127) 
The light pool may be full. Handshake is used to check the connection
state of brpc. Should not be interfered by the thread pool logic.

---------
pick #42115

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-10-19 16:19:17 +08:00
yiguolei
15a7b53f59 [branch-2.1](case) fix invalid test case (#42126) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-10-19 13:54:17 +08:00
Rayner Chen
9383378f2e [Fix](Export) show export statement supports specify the catalog name #41662 (#42117) 
cherry pick from #41662

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-19 10:47:28 +08:00
zzzxl
d5fef266ec [fix](inverted index) Fix incorrect exception handling (#42094) 
https://github.com/apache/doris/pull/41874
 2024-10-19 10:45:32 +08:00
Rayner Chen
702184bd92 [fix](tvf) fix that FE OOM when using tvf #40981 (#42119) 
cherry pick from #40981

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-19 08:34:38 +08:00
Rayner Chen
bce89b65fb [Test](Orc) Add tests for exception orc files #40621 (#42118) 
cherry pick from #40621

Co-authored-by: Tiewei Fang <43782773+BePPPower@users.noreply.github.com>
 2024-10-19 08:34:18 +08:00
Rayner Chen
ede1253666 [fix](catalog) fix filtered database when use_meta_cache=true #41939 (#42114) 
cherry pick from #41939
 2024-10-19 08:33:35 +08:00
minghong
f361754a71 [fix](nereids) fix regression case:test_cte_filter_pushdown (#42103) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-19 01:33:20 +08:00
Rayner Chen
c65f1044f6 [opt](log) avoid print too much log when get partitions failed #41892 (#42116) 
cherry pick from #41892
 2024-10-19 00:28:49 +08:00
zhangdong
4cab8c5178 [fix](auth)fix show load priv bug (#41723) (#42108) 
pick: https://github.com/apache/doris/pull/41723
 2024-10-18 22:21:50 +08:00
starocean999
4edfbb5c76 [fix](nereids)modify split_part function to fold to null literal with correct data type (#42100) 
pick from master https://github.com/apache/doris/pull/41109

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-10-18 22:15:57 +08:00