database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,095 Commits 1 Branch 0 Tags
5f4d1931ffd82f2c1fc07130d14942c3ac47e4d7
Commit Graph

7587 Commits

Author SHA1 Message Date
qiye
5f4d1931ff [feature](ES Catalog)Add FE open API for ES Catalog (#40444) (#40526) 
bp #40444
 2024-09-09 14:25:43 +08:00
hui lai
4b72a4c04e [improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) (#40509) 
pick (#39975)

We encountered a scenario where a large number of small transactions
were generated, resulting in an impact on query performance: Kafka's
data comes in batches of very small data every very short time, which
leads to tasks being frequently scheduled and ending very quickly,
resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive
EOF, which would not delay data consumption, for perceiving EOF
indicates that the consumption speed is greater than the production
speed.
 2024-09-09 11:36:22 +08:00
zhangdong
01a651e573 [enhance](mtmv) During cache generation, no longer hold the write loc… (#40489) 
…k for mtmv (#40402)

pick: https://github.com/apache/doris/pull/40402
 2024-09-09 11:35:22 +08:00
seawinde
a67f20f073 [opt](mtmv) Support to contain select constant clause when create async materialized view (#40244) (#40435) 
## Proposed changes
commitId: 518a0fc0
pr: https://github.com/apache/doris/pull/40244
 2024-09-09 11:23:15 +08:00
zhangdong
4084ff2c4a [enhance](mtmv)Optimize the speed of obtaining the last update time o… (#40421) 
…f Hive (#40169)

pick: https://github.com/apache/doris/pull/40169
 2024-09-09 11:20:40 +08:00
zhangdong
fc4c2aed64 [enhance](mtmv) Prohibit creating materialized views in external cata… (#40420) 
…logs (#39557)

pick: https://github.com/apache/doris/pull/39557
 2024-09-09 11:16:40 +08:00
zhangdong
ecb75c2e7d [fix](mtmv) Mtmv support set both immediate and starttime (#39573) (#40418) 
pick: https://github.com/apache/doris/pull/39573
 2024-09-09 11:13:51 +08:00
zhangdong
a3eba2aad5 [fix](auth) ordinary users can see the processes of other users (#39747) (#40415) 
pick: https://github.com/apache/doris/pull/39747
 2024-09-09 11:13:18 +08:00
zhangdong
e1e09badfa [chore](mtmv)add plan log for UpdateMvByPartitionCommand (#40072) (#40413) 
pick: https://github.com/apache/doris/pull/40072
 2024-09-09 11:09:58 +08:00
zhangdong
93a2518d12 [fix](auth)Fix some issues with incorrect permission verification (#3… (#40410) 
…9726)

pick: https://github.com/apache/doris/pull/39726
 2024-09-09 11:04:49 +08:00
Siyang Tang
d373ca7da1 [enhancement](schema-change) Support light schema change on hash columns and agg key columns with varchar type to change length (#39319) (#40236) 
## Proposed changes

1. Schema change should rebuild distribution info after modifying
columns, especially distribution columns. Or it may cause dynamic
partition failed when checking distribution columns' equality.
2. Support hash key columns to do light schema change. For unique key or
dup key columns, could not be enabled temporarily due to some historical
reasons. See #39798 .
 2024-09-09 10:55:32 +08:00
zhiqiang
a963709fed [opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 (#40357) 
cherry pick from #39927
 2024-09-09 10:42:19 +08:00
zy-kkk
962c382077 [fix](jdbc catalog) Fix type recognition error when using query tvf to query doris (#40481) 
pick  (#40122)

Using string to match Doris type will not work with query tvf, so use
field matching instead
 2024-09-06 19:30:32 +08:00
qiye
8104b992d1 [fix](ES Catalog)Do not extract doc_values of field with ignore_above setting (#40314) (#40464) 
bp #40314
 2024-09-06 16:25:30 +08:00
abmdocrt
88750e7e5d [cherry-pick](branch-2.1) Pick "[Fix](group commit) Fix table not found fault when disable group commit (#39731)" (#40323) 
<!--Describe your changes.-->

## Proposed changes

Pick #39731 

<!--Describe your changes.-->
 2024-09-05 21:39:15 +08:00
zhiqiang
0928c9c6ed [fix](unary function) Fix wrong result of asin, acos and sqrt when processing invalid input #40267 (#40358) 
cherry pick from #40267
 2024-09-05 19:51:01 +08:00
Calvin Kirs
961d2c9af5 [feat](job)Internal job cancellation immediately and the strong association with the STARTS parameter (#36805) (#38110) 
…

## Proposed changes

For internal tasks, such as MTMV, the start time may already be set, or
the time may be adjusted immediately.

<!--Describe your changes.-->

(cherry picked from commit 904a6c0fc1a804520285533de874fe4d0ffff2c1)

## Proposed changes

Issue Number: close #36805

<!--Describe your changes.-->
 2024-09-05 16:28:35 +08:00
abmdocrt
52393f829d [cherry-pick](branch-2.1) Fix some group commit fault (#40319) 
## Proposed changes

Pick #39986 #40120

<!--Describe your changes.-->
 2024-09-05 15:26:59 +08:00
yiguolei
40d10bdb0a Revert "Revert "[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value"" (#40411) 
Reverts apache/doris#40359
 2024-09-05 14:59:01 +08:00
minghong
c6771da240 [fix](nereids) fix bug for A>n, where A.max is infinity #39936 (#40368) 
## Proposed changes
pick #39936 
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-05 08:33:04 +08:00
yiguolei
08fb91f161 Revert "[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value" (#40359) 
Reverts apache/doris#40283
 2024-09-04 17:55:25 +08:00
Tiewei Fang
b4beec8ea8 [fix](OrcWriter) fix be core when upgrading BE without upgrading FE (#40303) 
bp: #40282
 2024-09-04 10:24:41 +08:00
feiniaofeiafei
db06c678a3 [Fix](planner) legacy planner repeat group by has grouping report error (#40281) (#40305) 
cherry-pick #40281 to branch-2.1
 2024-09-04 08:50:21 +08:00
Jerry Hu
3cf5d15b49 [fix](fe) avoid setting the value of batch_size excessively high (#35941) (#40312) 
Issue Number: close #xxx

pick #35941

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-03 19:49:26 +08:00
abmdocrt
c49b345342 [cherry-pick](branch-2.1) Fix enable_mow_light_delete default value (#40283) 
## Proposed changes

Pick #40104 #40197

<!--Describe your changes.-->
 2024-09-03 11:19:05 +08:00
Mingyu Chen
92752b90e7 [feature](metacache) add system table catalog_meta_cache_statistics #40155 (#40210) 
bp #40155
 2024-09-02 23:23:35 +08:00
morrySnow
1527099e35 [opt](Nereids) forbid one step topn with distribute as child (#40066) (#40278) 
pick from master #40066
 2024-09-02 22:26:26 +08:00
Jibing-Li
a5efdc6085 [improve](statistics)Clean expired TableStatsMeta. (#39779) (#40253) 
backport: https://github.com/apache/doris/pull/39779
 2024-09-02 22:13:42 +08:00
Jibing-Li
8ff068eb7c [feature](statistics) Support get row count for JDBC external table. (#38889) (#40248) 
backport: https://github.com/apache/doris/pull/38889
 2024-09-02 16:39:42 +08:00
lihangyu
98e039ae31 [pick](Variant) truncate key when column is variant type when crea… (#40238) 
#39988
 2024-09-02 13:54:04 +08:00
Xinyi Zou
5db2c8b8bc [branch-2.1](arrow-flight-sql) Fix return result from FE Arrow Flight server error 0.0.0.0:xxx, connection refused (#40229) 
pick #40002
 2024-09-02 09:45:27 +08:00
Calvin Kirs
2a12251b83 [branch-2.1][Feat](Job)After a job is paused, it can be manually triggered to execute. (#40180) 
…

pick (#39565)
 2024-08-31 19:24:53 +08:00
feiniaofeiafei
32a32808a7 [improvement](nereids) support convert_tz in partition prune (#40047) (#40127) 
cherry-pick #40047 to branch-2.1
 2024-08-30 13:39:37 +08:00
walter
80508fb595 [fix](restore) Reset next version for remote table when restore #40118 (#40165) 
cherry pick from #40118

Co-authored-by: smallx <e9999e@163.com>
 2024-08-30 13:22:23 +08:00
starocean999
1288676b68 [fix](nereids)change lag/lead function signature (#40161) 
## Proposed changes

pick from master https://github.com/apache/doris/pull/40060

<!--Describe your changes.-->
 2024-08-30 12:05:47 +08:00
zclllhhjj
163193b1d4 [branch-2.1](function) fix random_bytes return same data for multi rows (#39891) (#40137) 
pick https://github.com/apache/doris/pull/39891

Issue Number: close #xxx

before:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a                      |
+------------------------+
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
+------------------------+
```

now:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a                      |
+------------------------+
| 0xd82cf60825b29ef2a0fd |
| 0x6f8c808415bdbaa6d257 |
| 0x7c26b5214297a151c25c |
| 0x43f02c77293063900437 |
| 0x5e5727569dec5e24f96b |
| 0x434f20bf74d7759640b7 |
| 0x087ed96b739750c733a6 |
| 0xdf05f6d7ede4972eb846 |
| 0xcefab471912264b5c54f |
| 0x1bddc019409d1926aa10 |
+------------------------+
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-30 10:43:42 +08:00
zclllhhjj
e4d543f060 [branch-2.1](load) fix commit txn timeout when loading to table with many tablet (#40031) (#40139) 
pick https://github.com/apache/doris/pull/40031
 2024-08-30 10:34:43 +08:00
zzzxl
70daa1f85d [opt](inverted index) Controls whether the in_list can execute fast_execute. (#40141) 
https://github.com/apache/doris/pull/40022
 2024-08-30 10:32:43 +08:00
yiguolei
ca07a00c93 Revert "[branch-2.1](hive) support hive write text table (#38549) (#4… (#40157) 
…0063)"

This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-08-30 10:25:38 +08:00
feiniaofeiafei
1ecc85b575 [Fix](ctas) forward the use_max_length_of_varchar_in_ctas (#39705) (#40138) 
cherry-pick from master #39705
 2024-08-30 10:11:29 +08:00
abmdocrt
0b9817ea4b [cherry-pick](branch-2.1) Pick "[Enhancement](txn) Block new insert into if schema change happens during transaction (#39483)" (#40115) 
## Proposed changes

Pick #39483 

<!--Describe your changes.-->
 2024-08-30 10:01:10 +08:00
seawinde
45e35e8a99 [improvement](mtmv) Optimize get available mv logic to avoid unnecessary rewrite (#39734) (#40136) 
## Proposed changes

commitId: 55ed1277
pr: https://github.com/apache/doris/pull/39734
 2024-08-30 08:02:27 +08:00
seawinde
e072ce7f68 [fix](nereids) Remove Nondeterministic to avoid ambiguity when decide a expression is nondeterministic or not (#39801) (#40130) 
## Proposed changes

commitId: 2b715a7e
pr: https://github.com/apache/doris/pull/39801
 2024-08-30 07:51:42 +08:00
TengJianPing
b86e4e8498 [fix](profile) fix possible coredump of rpc verbose profile (#40117) 
## Proposed changes

Issue Number: close #xxx

`_instance_to_rpc_stats_vec` may be updated when sorting in
`ExchangeSinkBuffer<Parent>::update_profile`, which may cause coredump.
 2024-08-29 23:54:52 +08:00
morrySnow
82fb747e6f [fix](Nereids) datetimev2 literal equals should compare microsecond (#40121) (#40150) 
pick from master #40121
 2024-08-29 23:41:51 +08:00
Socrates
c6df7c21a3 [branch-2.1](hive) support hive write text table (#38549) (#40063) 
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd

pick from https://github.com/apache/doris/pull/38549
 2024-08-29 16:50:40 +08:00
xzj7019
2054cc9fc9 [fix](nereids) fix union all instance number (#40099) 
pick from https://github.com/apache/doris/pull/39999

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-29 16:47:55 +08:00
bobhan1
eb6a040a03 [branch-2.1] Picks "[Fix](partial update) Fix wrongly update autoinc column in partial update #39996" (#40073) 
## Proposed changes

pick https://github.com/apache/doris/pull/39996
 2024-08-29 16:41:51 +08:00
Mingyu Chen
131238ff71 [fix](file-cache) change metric_value column in file_cache_statistics table to string (#40083) 
Make it more flexible
followup #39552
 2024-08-29 16:39:22 +08:00
Gabriel
7155711431 [cherry-pick](branch-2.1) Improve local shuffle strategy (#40030) 
pick #34122 #35454 #35716 #37195
 2024-08-29 14:16:16 +08:00