Commit Graph

9742 Commits

Author SHA1 Message Date
ccb6c2610a branch-2.1: [fix](nereids)fix bug of PushDownFilterThroughSetOperation (#46370) (#46456)
pick from master #46370

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-01-06 21:34:40 +08:00
53889cbbb6 branch-2.1: [enhance](auth)Reduce the scope of permission locks #46115 (#46451)
Cherry-picked from #46115

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-06 18:51:09 +08:00
12350ca867 branch-2.1: [Improve](mtmv) skip the generation of invalid task for refresh mtmv #46280 (#46392)
Cherry-picked from #46280

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencent.com>
2025-01-05 08:59:11 +08:00
a4c328c4cd branch-2.1: [fix](compile) fix fe compile failed by code too large #46408 (#46420)
Cherry-picked from #46408

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
2025-01-05 05:07:49 +08:00
eb867f593a branch-2.1: [opt](iceberg) opt the error msg when create db which already exists #46334 (#46414)
Cherry-picked from #46334

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-04 19:05:52 +08:00
1405c48a77 [opt](coordinator) optimize parallel degree of shuffle when use nereids (#44754) (#46397)
cherry pick from #44754
2025-01-04 19:04:17 +08:00
2358dc4403 [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap (branch-2.1) (#46407)
### What problem does this PR solve?

pick #46406
Related PR: #40815
2025-01-04 11:54:08 +08:00
714b90a7fe branch-2.1: [fix](arrow-flight-sql) Fix arrow-flight-sql ConnectContext to use a unified ID #46284 (#46389)
Cherry-picked from #46284

Co-authored-by: Xinyi Zou <zouxinyi@selectdb.com>
2025-01-04 10:36:09 +08:00
8b45258ae6 [fix](external)using saved member variables for 2.1 (#46156) (#46400)
bp: #46156
2025-01-04 09:01:14 +08:00
e9921d7a07 branch-2.1: [fix](external catalog) Fixed non-master FE log replay during refresh (#46373)
cherry-pick #46369
2025-01-03 22:03:30 +08:00
384b78fa4e [Fix]delete internal group (#46351) 2025-01-03 21:59:36 +08:00
c2a17b8ad9 [fix](schema-change) Fix job replay failure when partitions added to table after job finish (#46166) (#46367)
Cherry-picked from #46166
2025-01-03 19:25:43 +08:00
ff68859546 branch-2.1: [improvement](http) improve table schema api for catalog table #45933 (#45982)
Cherry-picked from #45933

Co-authored-by: wudi <wudi@selectdb.com>
2025-01-03 16:40:12 +08:00
05292563ca branch-2.1: [fix](statistics)Fix sample all tablets scale factor incorrect bug. #46306 (#46329)
Cherry-picked from #46306

Co-authored-by: James <lijibing@selectdb.com>
2025-01-03 16:28:59 +08:00
98a8738a7d [fix](metaCache)fix bug that names cache can not invalidate.(#46287) (#46349)
bp #46287
2025-01-03 14:51:40 +08:00
86f2050af7 [fix](test) use different catalog name for test_hive_parquet_skip_page (#46314)
cherry-pick #46315
2025-01-03 09:13:06 +08:00
c385293ee9 branch-2.1: [opt](nereids) improve prune partition with lots of in (xxx) #46261 (#46286)
improve prune partition with lots of `in (xxx)`

this can reduce from 35s to 300ms when there have 4000 options
2025-01-02 18:52:31 +08:00
068ff8deae branch-2.1: [opt](show) let all types table support show index #45861 (#45898)
Cherry-picked from #45861

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-01-02 17:13:58 +08:00
2c7e5cae9a branch-2.1: [fix](catalog) Fix mark handling for failed tasks to maintain getLeftMarks #46205 (#46256)
cherry pick from #46205
2025-01-02 16:39:43 +08:00
f6d98a6061 branch-2.1: [Bug](auto-partition) fix auto partition could set storage_medium properties #45955 (#46260)
Cherry-picked from #45955

Co-authored-by: zhangstar333 <zhangsida@selectdb.com>
2025-01-02 16:38:55 +08:00
bfab6361f7 branch-2.1: [fix](report) report queue size based on the number of Bes and config #46190 (#46243)
Cherry-picked from #46190

Co-authored-by: deardeng <dengxin@selectdb.com>
2025-01-02 15:11:03 +08:00
fbc361ec7c branch-2.1: [opt](arrow-flight-sql) Support arrow-flight-sql protocol getStreamCatalogs, getStreamSchemas, getStreamTables #46217 (#46268)
Cherry-picked from #46217

Co-authored-by: Xinyi Zou <zouxinyi@selectdb.com>
2025-01-02 15:07:04 +08:00
b5beaf4eb1 branch-2.1: [fix](mtmv)The refresh method for MTMV is commit. If the status is PAUSED, no more tasks should be generated #46020 (#46265)
Cherry-picked from #46020

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-02 14:03:28 +08:00
9e45c70923 branch-2.1: [chore](restore) log the table name and partition info when partition type is different #46213 (#46234)
Cherry-picked from #46213

Co-authored-by: walter <maochuan@selectdb.com>
2025-01-01 08:09:11 +08:00
a6fc852ce7 branch-2.1: [opt](catalog) use table in db object return get db #46211 (#46229)
Cherry-picked from #46211

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
Co-authored-by: morningman <yunyou@selectdb.com>
2025-01-01 08:00:49 +08:00
9593a7bfc3 branch-2.1: [fix](split)Fixed the bug that batch mode split could not query data in multiple be scenarios. #46218 (#46227)
Cherry-picked from #46218

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-01-01 00:34:48 +08:00
28960b7b7d [Improve](nereids) use hash set replace three set in DiscreteValue to improve in predicate performance (#45181) (#46220)
### What problem does this PR solve?

cp #45181
2024-12-31 23:04:57 +08:00
1d8b3a0bcc branch-2.1: [fix](audit) fix the delay in loading audit log #46175 (#46193)
Cherry-picked from #46175

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
2024-12-31 22:56:12 +08:00
f59c285538 branch-2.1: [fix](index)Add duplicated indexes check in add index #46155 (#46210)
Cherry-picked from #46155

Co-authored-by: qiye <luen@selectdb.com>
2024-12-31 22:46:29 +08:00
9eb07e578e branch-2.1 [Fix](catalog)Remove the fs.disable.cache parameter to prevent excessive FS-associated objects and memory leaks (#46184) (#46191)
#46184
(cherry picked from commit 17667aeb92efc5580b8615e53a3ef4f514a27204)
2024-12-31 16:54:10 +08:00
fa20529dd9 [enhance](auth)Optimize the authentication logic of Ranger Doris(#41207) (#46179)
pick: https://github.com/apache/doris/pull/41207

only pick Performance optimization related
2024-12-31 11:54:27 +08:00
aa8c93cfe2 branch-2.1: [fix](sql) Wrong result for alter set property #46028 (#46141)
pick https://github.com/apache/doris/pull/46028
2024-12-31 09:52:43 +08:00
1af18d0211 [fix](audit-loader) Fix the bug that causes audit loader to fail. (#46119)
Fix the bug that causes audit loader to fail.
Related PR: #45167 #40493

The bug causes audit loader fail as following errors in audit.log.
```
2024-12-27 11:47:47,001 [stream_load] |Label=audit_log_20241227_114552_856_127_0_0_1_8030|Db=__internal_schema|Table=audit_log|User=|ClientIp=10.0.1.3|Status=Success|Message=OK|Url=http://10.0.1.4:8040/api/_load_error_log?file=__shard_7/error_log_insert_stmt_c24ed0d941f59867-ec08b8542bc2a4a1_c24ed0d941f59867_ec08b8542bc2a4a1|TotalRows=34|LoadedRows=0|FilteredRows=34|UnselectedRows=0|LoadBytes=6887|StartTime=2024-12-27 11:45:52.858|FinishTime=2024-12-27 11:45:52.888
```
The detail error is:
```
curl http://10.0.1.4:8040/api/_load_error_log?file=__shard_7/error_log_insert_stmt_c24ed0d941f59867-ec08b8542bc2a4a1_c24ed0d941f59867_ec08b8542bc2a4a1

Reason: actual column number in csv file is  more than  schema column number.actual number: 29, schema column number: 27; line delimiter: [
], column separator: [  ], result values:
```

Co-authored-by: derenli <derenli@tencent.com>
2024-12-31 07:30:00 +08:00
c8fb934bc1 branch-2.1: [chore](config) disable restore_reset_index_id by default #46104 (#46127)
Cherry-picked from #46104

Co-authored-by: walter <maochuan@selectdb.com>
2024-12-30 11:52:58 +08:00
5425e4686b branch-2.1: [fix](auth) fix use database stmt access unauthorized catalog #45720 (#45978)
Cherry-picked from #45720

Co-authored-by: Petrichor <xiaowenli@selectdb.com>
2024-12-30 10:43:28 +08:00
9b91a4a560 branch-2.1: [test](nereids) add test for normalize sort #45890 (#46112)
Cherry-picked from #45890

Co-authored-by: yujun <yujun@selectdb.com>
2024-12-30 10:19:40 +08:00
b085e94a5e branch-2.1: [fix](sql) Wrong result for partition item tosql #45918 (#45960)
Cherry-picked from #45918

Co-authored-by: Uniqueyou <wangyixuan@selectdb.com>
2024-12-30 10:09:58 +08:00
a490a366e4 [improvement](statistics)Support get oracle jdbc row count. (#45214) (#46133)
backport: https://github.com/apache/doris/pull/45214
2024-12-29 22:17:57 +08:00
6dd92be33d [feature](statistics)Support get row count for pg and sql server. (#42674) (#46131)
backport: https://github.com/apache/doris/pull/42674
2024-12-29 19:37:21 +08:00
a380f5d222 [enchement](utf8)import enable_text_validate_utf8 session var (#45537) (#46070)
bp #45537
2024-12-28 10:05:03 +08:00
829d9b3454 branch-2.1: [Fix](PreparedStatement) nondeterministic functions in prepared statement should not be short circuited #46003 (#46109)
Cherry-picked from #46003

Co-authored-by: lihangyu <lihangyu@selectdb.com>
2024-12-27 21:02:36 +08:00
75e9aa3fb4 branch-2.1: [fix](jdbc catalog) Get statistics using remote name #46040 (#46116)
Cherry-picked from https://github.com/apache/doris/pull/46040
2024-12-27 20:59:18 +08:00
8ac54b63c5 branch-2.1 [Fix](Catalog)Standardize the use of authentication attributes in the Catalog to avoid creating a large number of Authenticator objects. (#46106)
…utes in the Catalog to avoid creating a large number of Authenticator
objects.

#46052
2024-12-27 20:36:12 +08:00
fab89c199e branch-2.1: [chore](binlog) Add table type to getMeta method #45942 (#45972)
Cherry-picked from #45942

Co-authored-by: walter <maochuan@selectdb.com>
2024-12-27 20:03:53 +08:00
6cee95540d branch-2.1: [fix](variant) disable column name with dot character for variant type #45927 (#45992)
Cherry-picked from #45927

Co-authored-by: camby <cambyzhu@tencent.com>
2024-12-27 19:12:44 +08:00
69961a3c1b [opt](hms table)Some optimizations for hms external table for 2.1 (#44909) (#46084)
bp: #44909
2024-12-27 16:42:42 +08:00
ae68580866 branch-2.1: [test](nereids) add arthmetic comparison ut #45690 (#45893)
cherry pick from #45690
2024-12-27 14:54:52 +08:00
60d3c1e31d branch-2.1: [fix](nereids) fix compare with date like overflow #45868 (#45929)
cherry pick from #45868
2024-12-27 10:55:42 +08:00
8d3e8feb6e branch-2.1: [fix](show)show tables should be case insensitive when lowerCaseTableNames is 1 or 2. #46030 (#46058)
Cherry-picked from #46030

Co-authored-by: James <lijibing@selectdb.com>
2024-12-27 07:18:41 +08:00
4a07efe119 [improvement](statistics)Async drop table stats while doing truncate and schema change(#45923) (#46010)
backport: https://github.com/apache/doris/pull/45923
2024-12-26 21:57:18 +08:00