Commit Graph

21217 Commits

Author SHA1 Message Date
eda8a26ec8 [ci](fix) tmp exclude case test_routine_load_eof (#46586) 2025-01-08 11:12:53 +08:00
5d2930e783 [fix](shellcheck) fix hive-metastore and enable shellcheck in docker (#46496) (#46574)
cherry-pick (#46496)

Co-authored-by: Socrates <suyiteng@selectdb.com>
2025-01-08 11:10:34 +08:00
e90480037b [fix](index compaction)Fix construct index compaction columns core when reader close error (#46508)
fix #46507
2025-01-08 10:39:37 +08:00
f360ee23df [branch-2.1](metrics) Fix scan task related metrics (#46509)
### What problem does this PR solve?

Remove incorrect metrics
2025-01-08 10:10:03 +08:00
4b3422fc8e branch-2.1: [fix](bf index) add ngram bf index validation in nereids index definition check #45780 (#46216)
cherry pick from #45780
2025-01-08 10:04:21 +08:00
8aa2fafff4 [bugfix](asan core) should use weak ptr in rpc context to release rf context during query context deconstructed (#46565)
cherry-pick  #38653

Issue Number: close #xxx
F20240731 10:38:53.012670 20850 mem_tracker_limiter.cpp:135] mem tracker
label: Query#Id=7539c7d0087442b7-a1cda6392053548a, consumption: 52, peak
consumption: 1155332, mem tracker not equal to 0 when mem tracker
destruct, this usually means that memory tracking is inaccurate and
SCOPED_ATTACH_TASK and SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER are not
used correctly. 1. For query and load, memory leaks may have occurred,
it is expected that the query mem tracker will be bound to the thread
context using SCOPED_ATTACH_TASK and
SCOPED_SWITCH_THREAD_MEM_TRACKER_LIMITER before all memory alloc and
free. 2. If a memory alloc is recorded by this tracker, it is expected
that be recorded in this tracker when memory is freed. 3. Merge the
remaining memory tracking value by this tracker into Orphan, if you
observe that Orphan is not equal to 0 in the mem tracker web or log,
this indicates that there may be a memory leak. 4. If you need to
transfer memory tracking value between two trackers, can use
transfer_to..[Address Sanitizer]:
0x606002867d80, size 52, strack trace:
0# Allocator::alloc_impl(unsigned long, unsigned long) 1# void*
phmap::priv::Allocate<4ul, doris::vectorized::Allocator_
>(doris::vectorized::Allocator_*, unsigned long)
2# phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::initialize_slots(unsigned long) 3#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::resize(unsigned long) 4#
phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::prepare_insert(unsigned long) 5#
std::pair phmap::priv::raw_hash_set, phmap::Hash, phmap::EqualTo,
doris::vectorized::Allocator_ >::find_or_prepare_insert(int const&,
unsigned long)
6# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::emplace_decomposable(int const&, unsigned long, int const&)
7# std::pair, phmap::Hash, phmap::EqualTo, doris::vectorized::Allocator_
>::iterator, bool> phmap::priv::raw_hash_set, phmap::Hash,
phmap::EqualTo, doris::vectorized::Allocator_
>::EmplaceDecomposable::operator()(int const&, int const&) const
8# doris::HybridSet<(doris::PrimitiveType)5, doris::DynamicContainer,
doris::vectorized::ColumnVector >::insert(void const*) 9#
doris::HybridSetBase::insert(doris::HybridSetBase*) 10#
doris::RuntimePredicateWrapper::merge(doris::RuntimePredicateWrapper
const*)
11# doris::IRuntimeFilter::merge_from(doris::RuntimePredicateWrapper
const*)
12#

doris::RuntimeFilterMergeControllerEntity::merge(doris::PMergeFilterRequest
const*, butil::IOBufAsZeroCopyInputStream*)
13# doris::FragmentMgr::merge_filter(doris::PMergeFilterRequest const*,
butil::IOBufAsZeroCopyInputStream*)
14# std::_Function_handler::_M_invoke(std::_Any_data const&) 15#
doris::WorkThreadPool::work_thread(int)
16# execute_native_thread_routine
17# ?
18# ?
2025-01-08 09:30:29 +08:00
Pxl
2f98a6216e [Bug](join) fix columnstr64's offset overflow on serialize_value_into_arena #46461 (#46462)
pick from #46461
2025-01-07 19:59:35 +08:00
6bf31a2a0d branch-2.1: [fix](variable) variable version not persist in image #46463 (#46499)
Cherry-picked from #46463

Co-authored-by: morrySnow <zhangwenxin@selectdb.com>
2025-01-07 16:41:49 +08:00
45eaa3862d [regression-test](fix) fix test_audit_log_behavior.groovy bug (#46100) (#46524)
Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
In this case, select stmt from __internal_schema.audit_log where stmt
like '%3F6B9A_1%' order by time asc limit 1
will also be inserted into audit log table, and be selected, which will
cause assert fail in test case, which want a insert stmt.
2025-01-07 14:38:25 +08:00
c9ab3b9e3a [regression-test](fix) ignore E-2000 in compaction cases (#46522)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx
#46436
2025-01-07 14:26:54 +08:00
aa47a35384 [fix](mem) heap-buffer-overflow for function convert_to (#46405) (#46502)
pick #46405 to branch-2.1
2025-01-07 13:46:32 +08:00
abbf75e9fe branch-2.1: [fix](Nereids) insert into with cte cannot be explained (#46384) (#46501)
pick from master #46384
2025-01-07 13:44:32 +08:00
52455ed6d5 branch-2.1: [opt](cache) enhance cache key computation by removing comments and trimming SQL input #46099 (#46472)
Cherry-picked from #46099

Co-authored-by: York Cao <52438394+Baymine@users.noreply.github.com>
2025-01-07 13:39:48 +08:00
5962dceddb branch-2.1: [fix](hms)fix hive catalog follower not get event. #46485 (#46490)
Cherry-picked from #46485

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-01-07 12:19:06 +08:00
8f4b76a5f3 [fix](regression-test) adjust hudi cases (#46473) (#46488)
cherry-pick #46473

Co-authored-by: Socrates <suyiteng@selectdb.com>
2025-01-07 12:11:34 +08:00
5e9847a778 [regression-test](fix) fix global var bug in compaction/test_cumu_compaction_with_delete.groovy (#46514) 2025-01-07 11:41:05 +08:00
03a3f37cc4 [fix](index meta) make has_inverted_index function more robust #46364 (#46428)
cherry pick from #46364
2025-01-07 09:36:22 +08:00
ccb6c2610a branch-2.1: [fix](nereids)fix bug of PushDownFilterThroughSetOperation (#46370) (#46456)
pick from master #46370

Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
2025-01-06 21:34:40 +08:00
53889cbbb6 branch-2.1: [enhance](auth)Reduce the scope of permission locks #46115 (#46451)
Cherry-picked from #46115

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-06 18:51:09 +08:00
4b69673e4e [fix](fe.conf) use g1 by default for jdk17 (#46395) 2025-01-06 17:52:35 +08:00
dda058c433 branch-2.1: [fix](test)fix unstable case correctness_p0/test_mv_case/test_mv_case #46101 (#46431)
Cherry-picked from #46101

Co-authored-by: starocean999 <lichi@selectdb.com>
2025-01-06 17:42:40 +08:00
6958dec242 branch-2.1: [fix](regression)Fix schema change statistics case. Neredis not allow change MV schema. #46433 (#46448)
Cherry-picked from #46433

Co-authored-by: James <lijibing@selectdb.com>
2025-01-06 16:50:52 +08:00
7b8a93879c [test](mtmv)Test and check the differences between MTMV and OLAP (#44793)
pick from #36520 
pick from #44417 

Co-authored-by: zhangdong <zhangdong@selectdb.com>
2025-01-06 09:57:59 +08:00
12350ca867 branch-2.1: [Improve](mtmv) skip the generation of invalid task for refresh mtmv #46280 (#46392)
Cherry-picked from #46280

Co-authored-by: shee <13843187+qzsee@users.noreply.github.com>
Co-authored-by: garenshi <garenshi@tencent.com>
2025-01-05 08:59:11 +08:00
a4c328c4cd branch-2.1: [fix](compile) fix fe compile failed by code too large #46408 (#46420)
Cherry-picked from #46408

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
2025-01-05 05:07:49 +08:00
3d6ad47d08 branch-2.1: [fix](test) fix unstable lower case test #46410 (#46416)
Cherry-picked from #46410

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-04 22:28:29 +08:00
eb867f593a branch-2.1: [opt](iceberg) opt the error msg when create db which already exists #46334 (#46414)
Cherry-picked from #46334

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-04 19:05:52 +08:00
1405c48a77 [opt](coordinator) optimize parallel degree of shuffle when use nereids (#44754) (#46397)
cherry pick from #44754
2025-01-04 19:04:17 +08:00
1b1d7e2f29 branch-2.1: [fix](test) fix test_autoinc_broker_load.groovy #46320 (#46338)
Cherry-picked from #46320

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
2025-01-04 14:17:31 +08:00
2358dc4403 [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap (branch-2.1) (#46407)
### What problem does this PR solve?

pick #46406
Related PR: #40815
2025-01-04 11:54:08 +08:00
012848060b branch-2.1: [fix](test) Fix mysql catalog check db test #46263 (#46343)
Cherry-picked from #46263

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
2025-01-04 10:45:26 +08:00
714b90a7fe branch-2.1: [fix](arrow-flight-sql) Fix arrow-flight-sql ConnectContext to use a unified ID #46284 (#46389)
Cherry-picked from #46284

Co-authored-by: Xinyi Zou <zouxinyi@selectdb.com>
2025-01-04 10:36:09 +08:00
8b45258ae6 [fix](external)using saved member variables for 2.1 (#46156) (#46400)
bp: #46156
2025-01-04 09:01:14 +08:00
db224ba15f [fix](variant) fix schema change for variant from not null to null (#46403)
cherry-pick from #46279
2025-01-04 09:00:43 +08:00
d8c94d6392 branch-2.1: [fix](regression)fix hive translation unstable case. #46385 (#46409)
Cherry-picked from #46385

Co-authored-by: daidai <changyuwei@selectdb.com>
2025-01-04 08:59:56 +08:00
e9921d7a07 branch-2.1: [fix](external catalog) Fixed non-master FE log replay during refresh (#46373)
cherry-pick #46369
2025-01-03 22:03:30 +08:00
69de40ca27 branch-2.1: [fix](regression-test) fix injection would not be removed when exception #46357 (#46360)
Cherry-picked from #46357

Co-authored-by: hui lai <laihui@selectdb.com>
2025-01-03 22:02:50 +08:00
f93452dba7 branch-2.1: [improve](move-memtable) disable stack trace in load stream reply #46318 (#46332)
Cherry-picked from #46318

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
2025-01-03 22:01:21 +08:00
08cb06396b branch-2.1: [Fix](index build) should not append index to meta while column id is -1 #46307 (#46345)
cherry pick from #46307

Co-authored-by: airborne12 <jiangkai@selectdb.com>
2025-01-03 21:59:55 +08:00
384b78fa4e [Fix]delete internal group (#46351) 2025-01-03 21:59:36 +08:00
333b54eaba [fix](ip) fix ip nullable param without check (#44700) (#46252)
if we use ipv6_cidr_to_range function with nullable func which with
invalid ipv6 will make be core
```
mysql> select id, ipv6_cidr_to_range(nullable(''), 32) from fn_test_ip_nullable order by id;
```
2025-01-03 21:24:37 +08:00
26f570f40e [test](test) fix unstable paimon test cases (#46383)
cherry-pick #46381
2025-01-03 20:04:59 +08:00
c2a17b8ad9 [fix](schema-change) Fix job replay failure when partitions added to table after job finish (#46166) (#46367)
Cherry-picked from #46166
2025-01-03 19:25:43 +08:00
ff68859546 branch-2.1: [improvement](http) improve table schema api for catalog table #45933 (#45982)
Cherry-picked from #45933

Co-authored-by: wudi <wudi@selectdb.com>
2025-01-03 16:40:12 +08:00
05292563ca branch-2.1: [fix](statistics)Fix sample all tablets scale factor incorrect bug. #46306 (#46329)
Cherry-picked from #46306

Co-authored-by: James <lijibing@selectdb.com>
2025-01-03 16:28:59 +08:00
18846751d5 branch-2.1: [Bug](scan) do not release tablet_reader on NewOlapScanner::close #46296 (#46355)
Cherry-picked from #46296

Co-authored-by: Pxl <xl@selectdb.com>
2025-01-03 16:26:33 +08:00
98a8738a7d [fix](metaCache)fix bug that names cache can not invalidate.(#46287) (#46349)
bp #46287
2025-01-03 14:51:40 +08:00
025038410c [opt](test) add more debug info in test_jdbc_catalog_ddl.groovy (#46323)
cherry-pick #46322
2025-01-03 09:14:54 +08:00
34f9072ab6 [Improvement](local shuffle) Reduce locking scope in local exchanger … (#46294)
…(#46251)

Reduce lock scope from global level to data queue level.
2025-01-03 09:14:22 +08:00
79955e3d94 branch-2.1: [fix](regression-test) flush out file to fix hudi p2 #46258 (#46317)
Cherry-picked from #46258

Co-authored-by: Socrates <suyiteng@selectdb.com>
2025-01-03 09:13:48 +08:00