4e453dc1bb
Revert "[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. ( #40457 )" ( #40616 )
...
Reverts apache/doris#40540
2024-09-10 17:17:13 +08:00
7d80630757
[opt](scanner profile) More counter for scanner #40144 ( #40571 )
...
cherry pick from #40144
2024-09-10 16:08:55 +08:00
e43e6e2bba
[improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. ( #40457 ) ( #40540 )
...
backport: https://github.com/apache/doris/pull/40457
2024-09-10 12:55:53 +08:00
6961c95eca
[chore](error msg) More convenient error msg when function not found. #40296 ( #40570 )
...
cherry pick from #40296
2024-09-10 12:03:12 +08:00
185353e890
[Fix](inverted index) gc TEMP colum when next_batch in segment iterator ( #40563 )
...
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-10 09:56:03 +08:00
354967c09f
[branch-2.1](memory) pick reserve memory and workload group ( #40543 )
...
1. pick #38494
2. pick #39862
3. remove vdata_stream_test, master has been removed
2024-09-09 21:16:06 +08:00
f69063ea87
[Fix](Variant) use uinque id to access column reader ( #39841 ) ( #40269 )
...
#39841
#40295
2024-09-09 18:01:12 +08:00
8f37eccbf2
[Cherry-pick](branch-2.1) Pick "[Featrue](default value) Support bitmap_empty default value ( #40364 )" ( #40487 )
...
## Proposed changes
Pick #40364
<!--Describe your changes.-->
2024-09-09 16:57:38 +08:00
44a7efff4f
[branch-2.1] Picks "[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322 " ( #40383 )
...
picks https://github.com/apache/doris/pull/40322
2024-09-09 16:32:24 +08:00
314f6ae823
[fix](ES Catalog)Fix int parse error when querying by doc_values ( #40385 ) ( #40521 )
...
bp #40385
2024-09-09 14:29:21 +08:00
facce8b4d5
[fix](move-memtable) multi replica tables should tolerate minority failures ( #38003 ) ( #40477 )
...
backport #38003
2024-09-09 11:30:46 +08:00
e0b22b5104
[enhancement](schema-change) Log out end version before truncating new tablet data ( #39924 ) ( #40239 )
...
## Proposed changes
As title.
2024-09-09 10:46:41 +08:00
a963709fed
[opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 ( #40357 )
...
cherry pick from #39927
2024-09-09 10:42:19 +08:00
1c91fbc167
[fix](multi table) do not use strlen to calculate the length of msg ( #40367 ) ( #40511 )
...
pick #40367
Meet code dump when using single stream multi table load:
```
SUMMARY: AddressSanitizer: heap-buffer-overflow /root/doris/be/src/io/fs/multi_table_pipe.cpp:99:22 in doris::io::MultiTablePipe::dispatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, char const*, unsigned long, doris::Status (doris::io::KafkaConsumerPipe::*)(char const*, unsigned long))
```
1. It is hard to guaranteed that msg is a C-style string ending in '\0'
character. If not, it may cause the core dump to access memory out of
bounds.
2. It is not need to calculate the length of msg twice.
Therefore, deleting the logic that using strlen to calculate the length
of msg.
2024-09-09 10:35:59 +08:00
2023eab11e
[Fix](ShortCircuit) consider delete sign flag when hits row ( #40300 ) ( #40408 )
...
https://github.com/apache/doris/pull/40300
2024-09-09 10:04:05 +08:00
c1abaa4679
[Bug](map) fix wrong result on map_agg with streaming agg ( #40471 )
...
pick from #40454
2024-09-06 19:29:38 +08:00
0e057c49e8
[fix](table-func) fix explode-func with old pipeline ( #40482 )
...
## Proposed changes
if we use 2.0 fe and 2.1 be where pipeline use old logic may meet a core
like
```
22:44:04 F20240905 22:31:46.818060 25429 assert_cast.h:45] Bad cast from type:doris::vectorized::ColumnVector<int>* to doris::vectorized::ColumnVector<double>*
22:44:04 *** Check failure stack trace: ***
22:44:04 @ 0x560836b66586 google::LogMessage::SendToLog()
22:44:04 @ 0x560836b62fd0 google::LogMessage::Flush()
22:44:04 @ 0x560836b66dc9 google::LogMessageFatal::~LogMessageFatal()
22:44:04 @ 0x5608197f8013 assert_cast<>()
22:44:04 @ 0x5608220349af doris::vectorized::VExplodeJsonArrayTableFunction<>::_insert_values_into_column()
22:44:04 @ 0x5608220345d9 doris::vectorized::VExplodeJsonArrayTableFunction<>::get_value()
22:44:04 @ 0x560822007812 doris::vectorized::VTableFunctionNode::_get_expanded_block()
22:44:04 @ 0x560822009506 doris::vectorized::VTableFunctionNode::pull()
22:44:04 @ 0x5608365c4cc4 _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPNS5_18VTableFunctionNodeESt12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
22:44:04 @ 0x5608365c47b6 std::_Function_handler<>::_M_invoke()
22:44:04 @ 0x560810bcb5b0 doris::ExecNode::get_next_after_projects()
22:44:04 @ 0x5608365bf958 doris::pipeline::StatefulOperator<>::get_block()
22:44:04 @ 0x5608366bfe9d doris::pipeline::PipelineTask::execute()
22:44:04 @ 0x560836b3de7d doris::pipeline::TaskScheduler::_do_work()
22:44:04 @ 0x56081115a470 doris::ThreadPool::dispatch_thread()
22:44:04 @ 0x5608111399f9 doris::Thread::supervise_thread()
22:44:04 @ 0x7f43991edac3 (unknown)
```
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-06 19:29:09 +08:00
cb0613e249
[fix] (inverted index) fix error result in compound query ( #40425 )
...
## Proposed changes
`select count() from table where a + b > 0 or b > 0`
- When _execute_predicates_except_leafnode_of_andnode is executed, an
Expr tree is traversed from bottom to top. When it reaches the leaf node
b, the information of this column b is placed into new_predicate_info.
- However, this step is skipped directly at an ADD node, which leads to
the GT node at the upper level generating a sign equivalent to b > 0,
the same as the sign on the right side b > 0.
- This causes the compound OR calculation to assume that both GT
conditions below have been evaluated, thus prematurely computing this
EXPR, when in fact, the ADD node has not been evaluated.
- If the SQL is written as SELECT COUNT(*) FROM table WHERE b + a > 0 OR
b > 0, the calculation would be correct because the sign generated by
this > node would be equivalent to a > 0, which is different from b > 0
on the right side.
2024-09-06 10:27:59 +08:00
f64a728741
[enhance](variant) throw exception when field type is not supported in cast elimation ( #40448 )
...
#40388
2024-09-06 09:56:57 +08:00
7e27bb1ae6
[fix](window_funnel) fix wrong result of fixed mode ( #40460 )
...
BP #40459
2024-09-06 09:48:24 +08:00
87ac378c4a
[branch-2.1](be-ut) wait lazy open in ut ( #40453 )
...
## Proposed changes
LRUFileCache test need to wait lazy open done
2024-09-06 09:47:47 +08:00
0928c9c6ed
[fix](unary function) Fix wrong result of asin, acos and sqrt when processing invalid input #40267 ( #40358 )
...
cherry pick from #40267
2024-09-05 19:51:01 +08:00
d9fa59be4d
[Chore](runtime-filter) avoid dcheck fail when rf merge failed ( #39172 ) ( #40409 )
...
pick from #39172
2024-09-05 14:50:47 +08:00
26feaab711
[fix](delete_predicate) fix wrong data after upgrade from v2.0 version ( #40400 )
...
pick https://github.com/apache/doris/pull/40401 to branch-2.1
related issue:
https://github.com/apache/doris/issues/40390
related pr:
https://github.com/apache/doris/pull/22442
2024-09-05 14:46:03 +08:00
cc20ecd738
Revert "[fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets ( #38728 ) ( #39262 )" ( #40375 )
...
This reverts commit c9949f24e5c15e9529285f0e99b7ffdb1095558b.
This pr may increase the probability of full clone failure, so revert it
first.
2024-09-05 00:01:03 +08:00
653daeb8cb
Revert "[enhancement](index compaction) Enable index compaction by default ( #36812 )" ( #40351 )
...
Reverts apache/doris#38676
2024-09-04 14:16:09 +08:00
b4beec8ea8
[fix](OrcWriter) fix be core when upgrading BE without upgrading FE ( #40303 )
...
bp: #40282
2024-09-04 10:24:41 +08:00
0e9fa3dff7
[fix](decimaltype) handle exception with tablet init ( #40263 )
...
## Proposed changes
to avoid be core like
```
terminate called after throwing an instance of 'doris::Exception'
what(): [E6] meet invalid precision: real_precision=28, max_decimal_precision=27, min_decimal_precision=1
0# doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/exception
.cpp:0
1# doris::Exception::Exception<unsigned int const&, unsigned long>(int, std::basic_string_view<char, std::char_traits<char> > const&, unsigned int const&, unsigned long&
&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
2# doris::vectorized::DataTypeDecimal<doris::vectorized::Decimal<__int128> >::DataTypeDecimal(unsigned int, unsigned int, unsigned int, unsigned int) at /home/zcp/repo_c
enter/doris_branch-3.0/doris/be/src/vec/data_types/data_type_decimal.h:0
3# doris::vectorized::DataTypeFactory::create_data_type(doris::TypeDescriptor const&, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/data_types/data_ty
pe_factory.cpp:0
4# doris::vectorized::DataTypeFactory::create_data_type(doris::TypeDescriptor const&, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/data_types/data_ty
pe_factory.cpp:0
5# doris::vectorized::DataTypeFactory::create_data_type(doris::TypeDescriptor const&, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/data_types/data_ty
pe_factory.cpp:0
6# doris::SlotDescriptor::get_empty_mutable_column() const at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base
.h:1295
7# doris::VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<doris::OlapTableSchemaParam>&, doris::TOlapTablePartitionParam const&) at /home/zcp/repo_cen
ter/doris_branch-3.0/doris/be/src/vec/common/cow.h:154
8# doris::vectorized::VTabletWriter::_init(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/sink/writer/vtablet_w
riter.cpp:1177
9# doris::vectorized::VTabletWriter::open(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:488
10# doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status
.h:488
11# std::_Function_handler<void (), doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const
&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98
12# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/threadpool.cpp:0
13# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
14# ?
```
Issue Number: close #xxx
<!--Describe your changes.-->
2024-09-03 14:38:16 +08:00
67a2daf33c
[fix](execution) fix wrong check when blocking result sink ( #40289 )
...
Otherwise, when client stop fetch result data, the result sink will
still adding batch to the result queue,
which causing BE OOM.
2024-09-03 14:17:42 +08:00
41271ecba0
[fix](ES Catalog)Do not push down limit to ES when predicates can not be processed by ES. ( #40111 ) ( #40265 )
...
bp #40111
2024-09-03 11:17:24 +08:00
92752b90e7
[feature](metacache) add system table catalog_meta_cache_statistics #40155 ( #40210 )
...
bp #40155
2024-09-02 23:23:35 +08:00
4a91b2d46b
[pick](SchemaCache) remove redundant Schema cache ( #40257 )
...
#40091
2024-09-02 19:24:03 +08:00
70eeda9f50
[cherry-pick](branch-21) fix encryption_function return wrong result at const column ( #40201 ) ( #40247 )
...
## Proposed changes
cherry-pick from master #40201
the string object have error of order, should be src, key,iv,mode
2024-09-02 16:09:15 +08:00
c8d3202595
[regression-test](cases) optimize some cases ( #40240 )
...
#40174
2024-09-02 14:50:48 +08:00
e4ee0e8379
[fix](topn) Fix wrong rows returned by TOPN sorter ( #40243 )
...
## Proposed changes
pick #40241
<!--Describe your changes.-->
2024-09-02 14:34:53 +08:00
b104e933cd
[Bug](expr) execute expr should use local states instead of operators ( #40189 ) ( #40219 )
...
## Proposed changes
cherry-pick from master #40189
<!--Describe your changes.-->
2024-09-01 00:41:10 +08:00
1768169b9a
Revert "[Improvement](sort) Free sort blocks if this block is exhausted ( #39306 )" ( #40211 )
...
Reverts apache/doris#39956
2024-08-31 15:58:55 +08:00
d949df0c4f
[chore](cancel) Make enable_pipeline_task_leakage_detect false by default #40145 ( #40192 )
...
cherry pick from #40145
2024-08-30 22:48:00 +08:00
3ac8347e3d
[Improvement](sort) Free sort blocks if this block is exhausted ( #39306 ) ( #39956 )
...
## Proposed changes
pick #39306
<!--Describe your changes.-->
2024-08-30 13:35:43 +08:00
531fe54207
[fix](join) Fix partitioned hash join strategy ( #40164 )
...
## Proposed changes
pick #40163
<!--Describe your changes.-->
2024-08-30 12:44:42 +08:00
163193b1d4
[branch-2.1](function) fix random_bytes return same data for multi rows ( #39891 ) ( #40137 )
...
pick https://github.com/apache/doris/pull/39891
Issue Number: close #xxx
before:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a |
+------------------------+
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
| 0x7b4e5727024bc5b59e2c |
+------------------------+
```
now:
```sql
mysql [optest]>SELECT random_bytes(10) a FROM numbers("number" = "10");
+------------------------+
| a |
+------------------------+
| 0xd82cf60825b29ef2a0fd |
| 0x6f8c808415bdbaa6d257 |
| 0x7c26b5214297a151c25c |
| 0x43f02c77293063900437 |
| 0x5e5727569dec5e24f96b |
| 0x434f20bf74d7759640b7 |
| 0x087ed96b739750c733a6 |
| 0xdf05f6d7ede4972eb846 |
| 0xcefab471912264b5c54f |
| 0x1bddc019409d1926aa10 |
+------------------------+
```
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
2024-08-30 10:43:42 +08:00
70daa1f85d
[opt](inverted index) Controls whether the in_list can execute fast_execute. ( #40141 )
...
https://github.com/apache/doris/pull/40022
2024-08-30 10:32:43 +08:00
ca07a00c93
Revert "[branch-2.1](hive) support hive write text table ( #38549 ) (#4… ( #40157 )
...
…0063)"
This reverts commit c6df7c21a3c09ae1664deabacb88dfcea9d94b68.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
Co-authored-by: yiguolei <yiguolei@gmail.com >
2024-08-30 10:25:38 +08:00
0b9817ea4b
[cherry-pick](branch-2.1) Pick "[Enhancement](txn) Block new insert into if schema change happens during transaction ( #39483 )" ( #40115 )
...
## Proposed changes
Pick #39483
<!--Describe your changes.-->
2024-08-30 10:01:10 +08:00
b86e4e8498
[fix](profile) fix possible coredump of rpc verbose profile ( #40117 )
...
## Proposed changes
Issue Number: close #xxx
`_instance_to_rpc_stats_vec` may be updated when sorting in
`ExchangeSinkBuffer<Parent>::update_profile`, which may cause coredump.
2024-08-29 23:54:52 +08:00
fe881f8676
[fix](core) Fix BUCKET_HASH_SHUFFLE local exchanger ( #40149 )
...
## Proposed changes
pick #40148
<!--Describe your changes.-->
2024-08-29 23:23:42 +08:00
090474a815
[branch-2.1](load) Lower lock granularity to reduce loading CPU pressure ( #40135 )
...
pick https://github.com/apache/doris/pull/40134
2024-08-29 23:22:58 +08:00
a7156ee775
[fix](parquet)Fix the be core issue when reading parquet unsigned types. ( #39926 ) ( #40123 )
...
bp #39926
2024-08-29 21:52:52 +08:00
841565a40d
[fix](local shuffle) Fix hash shuffle local exchanger ( #40101 )
...
## Proposed changes
pick #40036
<!--Describe your changes.-->
2024-08-29 16:52:28 +08:00
c6df7c21a3
[branch-2.1](hive) support hive write text table ( #38549 ) ( #40063 )
...
1. Support write hive text table
2. Add SessionVariable `hive_text_compression` to write compressed hive
text table
3. Supported compression type: gzip, bzip2, snappy, lz4, zstd
pick from https://github.com/apache/doris/pull/38549
2024-08-29 16:50:40 +08:00
