database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
19,956 Commits 1 Branch 0 Tags
db0724dfe0c39f71ccc4b9d44b6f9f030dac721f
Commit Graph

19956 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zclllhhjj
db0724dfe0 [Fix-2.1](function) fix function covar core for not null input (#39943) 
## Proposed changes

Issue Number: close #xxx

add testcases like:
```groovy
    qt_notnull1 "select covar_samp(non_nullable(x), non_nullable(y)) from test_covar_samp"
    qt_notnull2 "select covar_samp(x, non_nullable(y)) from test_covar_samp"
    qt_notnull3 "select covar_samp(non_nullable(x), y) from test_covar_samp"
```

before they will all coredump in 2.1
 2024-08-27 08:39:47 +08:00
zhangstar333
21bd4a4ac8 [bug](function)fix json_replace check return type error (#37014) (#39938) 
1. fix the return type dcheck error:
```
mysql [test]>select (json_replace(a, '$.fparam.nested_2', "qwe")) from json_table_2 limit 1;
ERROR 1105 (HY000): errCode = 2, detailMessage = (10.16.10.8)[INTERNAL_ERROR]Function json_replace get failed, expr is VectorizedFnCall[json_replace](arguments=a, String, String, String,return=Nullable(String)) and return type is Nullable(String).
```

2. improve the json_replace/json_insert/json_set function execute of not
convert const column, test about could faster 1s on 1000w table rows

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 08:03:48 +08:00
minghong
aacd61a98f [fix](nereids)filter estimation for slot=unknown #39592 (#39933) 
## Proposed changes
pick #39592
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-27 08:03:00 +08:00
Socrates
a5c8ed1cde [branch2.1][fix](cache) Catch the directory_iterator's error_code (#39922) 
## Proposed changes

Catch the directory_iterator's error_code to avoid exceptions causing
core dump
 2024-08-27 08:00:52 +08:00
shouchengShen
e4bf10ee96 [enhancement](Load)allow load data to the other partitions when some partitions are restoring (#39915) 
If broker load or stream load task execute in one table that is
restoring data, load task will failed with Exception.
Exception info :"Table [xxx] is under restore" or "Table [xxx] is in
restore process, can't load into it".

But mostly restoreJob only effects some partitions in this table, not
all of them, so that the other partitions still need to load data
successfully.
To achieve this goal, before checking olap table state, check partition
state first.

cherry pick from master branch, pr has been merged:
https://github.com/apache/doris/pull/39595

Co-authored-by: shenshoucheng <shenshoucheng@jd.com>
 2024-08-26 21:04:34 +08:00
walter
fb5002d723 [fix](binlog) Fix NPE when recover binlogs (#39909) (#39910) 
Cherry-pick #39909 

The field partition ID of DropPartitionInfo was added in PR:
apache/doris#37196, the old version doesn't contain this field so it
will be null.
 2024-08-26 20:17:26 +08:00
Gabriel
a6e485e54f [fix](followup) Fix wrong partition num (#39917) 2024-08-26 18:25:36 +08:00
Sun Chenyang
44c565840d [fix] (regression test) fix error tests (#39907) 
## Proposed changes

fix error tests: `test_dup_table_inverted_index`,
`test_single_compaction_with_variant_inverted`
 2024-08-26 17:48:52 +08:00
walter
9412ba56ae [fix](restore) Reset the db name of the materialized index stmt #39710 (#39848) 
cherry pick from #39710, #39855
 2024-08-26 16:51:50 +08:00
wangbo
357394bb3e [branch-2.1]Reset io limit default value (#39898) 
pick #39842
 2024-08-26 14:27:35 +08:00
Gabriel
5acd1279a9 [fix](2.1) Fix correctness in branch-2.1 (#39901) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-26 14:12:59 +08:00
walter
e21d2b82d6 [chore](backup) Fix the db name of the restored view (#38072) 
Cherry-pick #37412, #39855

Previously, during restore, the database name in the CREATE VIEW
statement was not modified, causing the restored view to be unviewable
with the SHOW VIEW command. This PR retains the original cluster's
database name in the BackupMeta and manually replaces it with the new
cluster's database name in the CREATE VIEW statement during restore.
 2024-08-26 14:12:47 +08:00
zhangdong
d87a220d2b [opt](privilege) Grant check name (#39597) (#39856) 
pick https://github.com/apache/doris/pull/39597
 2024-08-26 09:53:45 +08:00
Yongqiang YANG
4c1c67e03a [improvemen](overflow) Provide the user with a suggestion to avoid th… (#39631) (#39897) 
cherry-pick #39631 to branch-2.1
 2024-08-26 08:10:32 +08:00
yiguolei
d563621f6e [enhancement](thrift) add value number to thrift definition for enum type (#39880) (#39896) 
Issue Number: close #xxx
 cherry pick : https://github.com/apache/doris/pull/39880

---------

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-08-26 08:07:57 +08:00
Yongqiang YANG
f2a37d58fb [fix](stat) handle overflow of memory stat if load failed (#39621) (#39887) 
## Proposed changes

pick #39621 

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-25 18:24:08 +08:00
zclllhhjj
f7b6ed772b [Fix-2.1](testcase) Fix some testcases unstable results due to lack of orderby (#39892) 
## Proposed changes

Issue Number: close #xxx

some sql's order-by column has same value, add more order-by column to
fix it.
 2024-08-25 08:45:36 +08:00
zzzxl
44b80fb03b [fix](inverted index) Fix Session Variable Compatibility (#39884) 
https://github.com/apache/doris/pull/39889
 2024-08-25 08:42:36 +08:00
TengJianPing
8dbd73988a [fix](recvr) catch exception of transmit_block (#39882) 
BP #39881
 2024-08-25 00:25:20 +08:00
yiguolei
e0534c9bfc [bugfix](thrift) the definition number should consistent with master branch (#39879) 
## Proposed changes
introduced by pr https://github.com/apache/doris/pull/35103

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-08-25 00:22:19 +08:00
LiBinfeng
9997911ec9 [feat](Nereids) support nereids hint position detaction (#39113) (#39417) 
cherry-pick: #39113
When use hint in wrong position or use unsupport hint, use channel(2) to
filter it out

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-24 23:59:54 +08:00
Gavin Chou
8e140727ae [fix](udf) Fix exception thrown when replayDropFunction (#39868) 
This PR completes #25965 to fix uncaught runtime exception when calling
`org.apache.doris.catalog.GlobalFunctionMgr#replayDropFunction`


![img_v3_02e0_3889b1de-583e-48ba-8e7f-48e1d3ae799g](https://github.com/user-attachments/assets/b0dbfefb-16f3-4dbb-8a00-f4773978ba01)
 2024-08-24 23:56:48 +08:00
TengJianPing
2dea859bdb [debug](rpc) debug rpc time consumption problem (#39852) 
## Proposed changes

Issue Number: close #xxx

Add detail RPC time info for each channel, sorted by max rpc time of
channels:
```
                     DATA_STREAM_SINK_OPERATOR  (id=1,dst_id=1):
                          -  Partitioner:  Crc32HashPartitioner(64)
                          -  BlocksProduced:  74
                          -  BrpcSendTime:  2.689us
                          -  BrpcSendTime.Wait:  0ns
                          -  BytesSent:  89.35  KB
                          -  CloseTime:  680.152us
                          -  CompressTime:  0ns
                          -  ExecTime:  160.663ms
                          -  InitTime:  263.608us
                          -  InputRows:  32.512K  (32512)
                          -  LocalBytesSent:  0.00  
                          -  LocalSendTime:  0ns
                          -  LocalSentRows:  0
                          -  MemoryUsage:  
                              -  PeakMemoryUsage:  80.00  KB
                          -  MergeBlockTime:  0ns
                          -  OpenTime:  4.113ms
                          -  OverallThroughput:  0.0  /sec
                          -  PendingFinishDependency:  41.179ms
                          -  RowsProduced:  32.512K  (32512)
                          -  RpcAvgTime:  11.850ms
                          -  RpcCount:  10
                          -  RpcMaxTime:  86.891ms
                          -  RpcMinTime:  15.200ms
                          -  RpcSumTime:  118.503ms
                          -  SerializeBatchTime:  13.517ms
                          -  SplitBlockDistributeByChannelTime:  38.923ms
                          -  SplitBlockHashComputeTime:  2.659ms
                          -  UncompressedRowBatchSize:  135.19  KB
                          -  WaitForDependencyTime:  0ns
                              -  WaitForRpcBufferQueue:  0ns
                        RpcInstanceDetails:
                              -  Instance  85d4f75b72a9ea61:  Count:  4,  MaxTime:  36.238ms,  MinTime:  12.107ms,  AvgTime:  21.722ms,  SumTime:  86.891ms
                              -  Instance  85d4f75b72a9ea91:  Count:  3,  MaxTime:  11.107ms,  MinTime:  2.431ms,  AvgTime:  5.470ms,  SumTime:  16.412ms
                              -  Instance  85d4f75b72a9eac1:  Count:  3,  MaxTime:  7.554ms,  MinTime:  3.160ms,  AvgTime:  5.066ms,  SumTime:  15.200m
```
 2024-08-24 19:59:39 +08:00
Jerry Hu
5a810122a2 [debug](load) check the column type when string column is invalid (#39337) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-08-24 18:14:21 +08:00
Mingyu Chen
263746b04b [fix](paimon) fix crash when enable cache with paimon deletion vector(#39877) (#39875) 
bp #39877
 2024-08-24 17:58:20 +08:00
Mingyu Chen
14a2a66106 [fix](paimon) fix not able to read paimon data from hdfs with HA (#39806) (#39876) 
bp #39806
 2024-08-24 17:51:15 +08:00
zclllhhjj
460605ae3c [branch-2.1] pick some prs (#39860) 
## Proposed changes

Issue Number: close #xxx

https://github.com/apache/doris/pull/38385 optimize parsing datetime
https://github.com/apache/doris/pull/38978 make stream load failure
message more clear and disable some error's stacktrace by default
https://github.com/apache/doris/pull/39255 fix random function coredump
https://github.com/apache/doris/pull/39324 fix function corr
inconsistency with doc
https://github.com/apache/doris/pull/39449 check auto partitoin nullity
when creating partition
https://github.com/apache/doris/pull/39695 make
DynamicPartitionScheduler immediately know interval's change
https://github.com/apache/doris/pull/39754 Add some partition expr check
on creating table
 2024-08-24 17:26:42 +08:00
HappenLee
564d3cd647 [Performance](opt) opt the order by performance in permutation (#39089) 
## Proposed changes

Issue Number: cherry pick #38985

<!--Describe your changes.-->
 2024-08-24 16:05:46 +08:00
Mingyu Chen
9640e2de44 [fix](catalog) refactor location path and support default fs #39116 (#39203) 2024-08-24 16:05:13 +08:00
Kang
76596e5f73 [fix](thrift) fix wrong order of field 27 and 28 in TCreateTabletReq thrift (#39873) 
## Proposed changes

fix wrong order of field 27 and 28 in `TCreateTabletReq` thrift
introduced by #32418 and 0cde0cbf19011bc8d421add4734d7cd57308973f.

`TCreateTabletReq` is used for creating tablet, so this PR will fix
problem creating tablet when upgrading 2.0.x -> 2.1.4/2.1.5 or
2.1.4/2.1.5 -> 3.0.x, BUT will cause problem creating tablet when
upgrading 2.1.4/2.1.5 -> 2.1.6+.

master and branch-2.0 
```
    27: optional i64 time_series_compaction_level_threshold = 1
    28: optional TInvertedIndexStorageFormat inverted_index_storage_format = TInvertedIndexStorageFormat.V1
```

branch-2.1 (affects 2.1.4 and 2.1.5)
```
    27: optional TInvertedIndexStorageFormat inverted_index_storage_format = TInvertedIndexStorageFormat.V1
    28: optional i64 time_series_compaction_level_threshold = 1
```
 2024-08-24 16:02:08 +08:00
Mingyu Chen
5ed56770d4 [bugfix](external) Prevent multiple fs from being generated (#39663) (#39870) 
bp #39663

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
 2024-08-24 14:17:26 +08:00
Mingyu Chen
de2e8f0ae6 [fix](ctas) fix NPE when ctas with old planner and varchar issue (#39744) (#39871) 
bp #39744
 2024-08-24 09:24:47 +08:00
Mingyu Chen
b9da934b16 [fix](hive) report error with escape char and null format (#39700) (#39869) 
bp #39700

Co-authored-by: Socrates <suxiaogang223@icloud.com>
 2024-08-24 09:23:03 +08:00
lihangyu
a6f267c479 [pick](Variant) fix element_at should return nullable if result type is nullable (#39846) 
#39732
 2024-08-24 09:22:03 +08:00
Xinyi Zou
ae4d747c13 [branch-2.1](memory) Modify memory gc conf and add crash_in_alloc_large_memory_bytes (#39834) 
pick #39611
 2024-08-24 09:21:35 +08:00
Jerry Hu
8cf6c6a2b5 [fix](agg function) incorrect result of map agg(#39743) (#39854) 
## Proposed changes

pick #39743
 2024-08-24 09:19:52 +08:00
Mingyu Chen
cf698fb615 [fix](regression) fix some jdbc datasource docker health check (#39141) (#39872) 2024-08-24 03:29:18 +08:00
zy-kkk
4cb7122a89 [branch-2.1][fix](jdbc catalog) Fix Memory Leak by Enabling Weak References in HikariCP (#39833) 
pick(#39582)
 2024-08-24 00:45:12 +08:00
Siyang Tang
00e4f343b0 [enhancement](delete) Add a hint msg for forbidden delete when MV or rollup exists (#39505) (#39857) 
## Proposed changes

When MV or Rollup exists, delete is forbidden on the base table
currently. Add a hint msg to indicate it.
 2024-08-23 23:42:26 +08:00
morrySnow
9d597bde68 [fix](Nereids) producer to consumer should be multimap in cte (#39850) (#39867) 2024-08-23 23:25:11 +08:00
Mingyu Chen
508c7a7040 [fix](hive)Modify the Hive notification event processing method when using meta cache and add parameters to the Hive catalog. (#39239) (#39865) 
bp #39239

Co-authored-by: daidai <2017501503@qq.com>
 2024-08-23 23:21:02 +08:00
lihangyu
3103bb08dc [pick](Variant) casting to decimal type may lost precision (#39843) 
#39650
 2024-08-23 22:47:32 +08:00
Kaijie Chen
37443aa7e1 [improve](move-memtable) reuse connection in load_stream_stub (#39231) (#39762) 
backport #39231
 2024-08-23 22:46:28 +08:00
Siyang Tang
5b124a03ba [enhancement](err-msg) Add detailed column and schema info when failed to create a column iterator (#38689) (#39861) 
As title.
 2024-08-23 21:39:19 +08:00
zhangdong
19c51cc217 [fix](mtmv) use name instead of id in meta of MTMV (#39355) (#39748) 
pick from master #39355
 2024-08-23 20:40:54 +08:00
wangbo
6ceb574aa0 [branch-2.1]Pick IO limit/workload group usage table (#39839) 2024-08-23 18:51:47 +08:00
Xinyi Zou
e716658fba [branch-2.1](arrow-flight-sql) Fix exceed user property max connection cause Reach limit of connections (#39836) 
pick #39127
pick #39802
 2024-08-23 17:27:34 +08:00
xzj7019
424ad2384a [opt](nereids) refine left semi/anti cost under short-cut opt (#39636) 
## Proposed changes

pick from https://github.com/apache/doris/pull/37951

---------

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
 2024-08-23 17:26:56 +08:00
Xinyi Zou
baf5b71b39 [branch-2.1](memory) Modify thedefault JEMALLOC_CONF and support flush Jemalloc tcache (#39829) 
pick #38185
 2024-08-23 17:21:42 +08:00
wuwenchi
c40246efa9 [bugfix](iceberg)Fixed random core with writing iceberg partitioned table for 2.1 (#39808)(#39569) (#39832) 
## Proposed changes

bp: #39808 #39569
 2024-08-23 17:19:48 +08:00