database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,099 Commits 1 Branch 0 Tags
2235c1abd3a67ea3ae92fdd0986e101b4f7be08d
Commit Graph

20099 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Calvin Kirs
2235c1abd3 [Feat](OSS)Decouple COS and OBS dependencies with optional inclusion mechanism (#39831) 
…

improve the flexibility of the project by decoupling direct dependencies
on the hadoop-cos and hadoop-huaweicloud libraries. These changes allow
users to control whether COS and OBS dependencies are included in the
final build, enabling a more customizable setup.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-09 16:50:12 +08:00
bobhan1
44a7efff4f [branch-2.1] Picks "[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322" (#40383) 
picks https://github.com/apache/doris/pull/40322
 2024-09-09 16:32:24 +08:00
hui lai
9c9827b92e [chore](routine load) make error msg clear if routine load name illegal (#40037) (#40510) 
pick (#40037)
 2024-09-09 15:47:18 +08:00
qiye
314f6ae823 [fix](ES Catalog)Fix int parse error when querying by doc_values (#40385) (#40521) 
bp #40385
 2024-09-09 14:29:21 +08:00
qiye
5f4d1931ff [feature](ES Catalog)Add FE open API for ES Catalog (#40444) (#40526) 
bp #40444
 2024-09-09 14:25:43 +08:00
hui lai
4b72a4c04e [improve](routine load) delay schedule EOF tasks to avoid too many small transactions (#39975) (#40509) 
pick (#39975)

We encountered a scenario where a large number of small transactions
were generated, resulting in an impact on query performance: Kafka's
data comes in batches of very small data every very short time, which
leads to tasks being frequently scheduled and ending very quickly,
resulting in a large number of small transactions.

To solve this problem, we delay the scheduling of tasks that perceive
EOF, which would not delay data consumption, for perceiving EOF
indicates that the consumption speed is greater than the production
speed.
 2024-09-09 11:36:22 +08:00
zhangdong
01a651e573 [enhance](mtmv) During cache generation, no longer hold the write loc… (#40489) 
…k for mtmv (#40402)

pick: https://github.com/apache/doris/pull/40402
 2024-09-09 11:35:22 +08:00
Kaijie Chen
facce8b4d5 [fix](move-memtable) multi replica tables should tolerate minority failures (#38003) (#40477) 
backport #38003
 2024-09-09 11:30:46 +08:00
seawinde
c32d9a129a [test](mtmv) SSB mv rewrite test use little data set for test performance (#40188) (#40437) 
## Proposed changes

commitId: 0baa9366
pr: https://github.com/apache/doris/pull/40188
 2024-09-09 11:23:47 +08:00
seawinde
a67f20f073 [opt](mtmv) Support to contain select constant clause when create async materialized view (#40244) (#40435) 
## Proposed changes
commitId: 518a0fc0
pr: https://github.com/apache/doris/pull/40244
 2024-09-09 11:23:15 +08:00
zhangdong
2c588e2738 [fix](auth)fix case should grant after create view (#40108) (#40488) 
pick: https://github.com/apache/doris/pull/40108
 2024-09-09 11:21:52 +08:00
zhangdong
4084ff2c4a [enhance](mtmv)Optimize the speed of obtaining the last update time o… (#40421) 
…f Hive (#40169)

pick: https://github.com/apache/doris/pull/40169
 2024-09-09 11:20:40 +08:00
zhangdong
fc4c2aed64 [enhance](mtmv) Prohibit creating materialized views in external cata… (#40420) 
…logs (#39557)

pick: https://github.com/apache/doris/pull/39557
 2024-09-09 11:16:40 +08:00
zhangdong
ecb75c2e7d [fix](mtmv) Mtmv support set both immediate and starttime (#39573) (#40418) 
pick: https://github.com/apache/doris/pull/39573
 2024-09-09 11:13:51 +08:00
zhangdong
a3eba2aad5 [fix](auth) ordinary users can see the processes of other users (#39747) (#40415) 
pick: https://github.com/apache/doris/pull/39747
 2024-09-09 11:13:18 +08:00
zhangdong
e1e09badfa [chore](mtmv)add plan log for UpdateMvByPartitionCommand (#40072) (#40413) 
pick: https://github.com/apache/doris/pull/40072
 2024-09-09 11:09:58 +08:00
zhangdong
93a2518d12 [fix](auth)Fix some issues with incorrect permission verification (#3… (#40410) 
…9726)

pick: https://github.com/apache/doris/pull/39726
 2024-09-09 11:04:49 +08:00
Siyang Tang
d373ca7da1 [enhancement](schema-change) Support light schema change on hash columns and agg key columns with varchar type to change length (#39319) (#40236) 
## Proposed changes

1. Schema change should rebuild distribution info after modifying
columns, especially distribution columns. Or it may cause dynamic
partition failed when checking distribution columns' equality.
2. Support hash key columns to do light schema change. For unique key or
dup key columns, could not be enabled temporarily due to some historical
reasons. See #39798 .
 2024-09-09 10:55:32 +08:00
Siyang Tang
e0b22b5104 [enhancement](schema-change) Log out end version before truncating new tablet data (#39924) (#40239) 
## Proposed changes

As title.
 2024-09-09 10:46:41 +08:00
zhiqiang
a963709fed [opt](scanner) Control the degree of parallelism of scanner when only limit involved #39927 (#40357) 
cherry pick from #39927
 2024-09-09 10:42:19 +08:00
hui lai
1c91fbc167 [fix](multi table) do not use strlen to calculate the length of msg (#40367) (#40511) 
pick #40367

Meet code dump when using single stream multi table load:
```
SUMMARY: AddressSanitizer: heap-buffer-overflow /root/doris/be/src/io/fs/multi_table_pipe.cpp:99:22 in doris::io::MultiTablePipe::dispatch(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, char const*, unsigned long, doris::Status (doris::io::KafkaConsumerPipe::*)(char const*, unsigned long))
```

1. It is hard to guaranteed that msg is a C-style string ending in '\0'
character. If not, it may cause the core dump to access memory out of
bounds.
2. It is not need to calculate the length of msg twice.

Therefore, deleting the logic that using strlen to calculate the length
of msg.
 2024-09-09 10:35:59 +08:00
lihangyu
2023eab11e [Fix](ShortCircuit) consider delete sign flag when hits row (#40300) (#40408) 
https://github.com/apache/doris/pull/40300
 2024-09-09 10:04:05 +08:00
yiguolei
653e315ba5 2.1.6-rc04 (#40491) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-09-06 19:41:37 +08:00
zy-kkk
962c382077 [fix](jdbc catalog) Fix type recognition error when using query tvf to query doris (#40481) 
pick  (#40122)

Using string to match Doris type will not work with query tvf, so use
field matching instead
 2024-09-06 19:30:32 +08:00
Pxl
c1abaa4679 [Bug](map) fix wrong result on map_agg with streaming agg (#40471) 
pick from #40454
 2024-09-06 19:29:38 +08:00
amory
0e057c49e8 [fix](table-func) fix explode-func with old pipeline (#40482) 
## Proposed changes
if we use 2.0 fe and 2.1 be where pipeline use old logic may meet a core
like
```
22:44:04   F20240905 22:31:46.818060 25429 assert_cast.h:45] Bad cast from type:doris::vectorized::ColumnVector<int>* to doris::vectorized::ColumnVector<double>*
22:44:04   *** Check failure stack trace: ***
22:44:04       @     0x560836b66586  google::LogMessage::SendToLog()
22:44:04       @     0x560836b62fd0  google::LogMessage::Flush()
22:44:04       @     0x560836b66dc9  google::LogMessageFatal::~LogMessageFatal()
22:44:04       @     0x5608197f8013  assert_cast<>()
22:44:04       @     0x5608220349af  doris::vectorized::VExplodeJsonArrayTableFunction<>::_insert_values_into_column()
22:44:04       @     0x5608220345d9  doris::vectorized::VExplodeJsonArrayTableFunction<>::get_value()
22:44:04       @     0x560822007812  doris::vectorized::VTableFunctionNode::_get_expanded_block()
22:44:04       @     0x560822009506  doris::vectorized::VTableFunctionNode::pull()
22:44:04       @     0x5608365c4cc4  _ZNSt5_BindIFMN5doris8ExecNodeEFNS0_6StatusEPNS0_12RuntimeStateEPNS0_10vectorized5BlockEPbEPNS5_18VTableFunctionNodeESt12_PlaceholderILi1EESD_ILi2EESD_ILi3EEEE6__callIS2_JOS4_OS7_OS8_EJLm0ELm1ELm2ELm3EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
22:44:04       @     0x5608365c47b6  std::_Function_handler<>::_M_invoke()
22:44:04       @     0x560810bcb5b0  doris::ExecNode::get_next_after_projects()
22:44:04       @     0x5608365bf958  doris::pipeline::StatefulOperator<>::get_block()
22:44:04       @     0x5608366bfe9d  doris::pipeline::PipelineTask::execute()
22:44:04       @     0x560836b3de7d  doris::pipeline::TaskScheduler::_do_work()
22:44:04       @     0x56081115a470  doris::ThreadPool::dispatch_thread()
22:44:04       @     0x5608111399f9  doris::Thread::supervise_thread()
22:44:04       @     0x7f43991edac3  (unknown)
```
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-06 19:29:09 +08:00
abmdocrt
b3c0b72faf [cherry-pick](branch-2.1) Pick "[Fix](regression-test) Fix test_group_commit_stream_load_with_nonexist_db_and_table regression test case (#40109)" (#40475) 
## Proposed changes

Pick #40109 

<!--Describe your changes.-->
 2024-09-06 16:39:58 +08:00
qiye
8104b992d1 [fix](ES Catalog)Do not extract doc_values of field with ignore_above setting (#40314) (#40464) 
bp #40314
 2024-09-06 16:25:30 +08:00
Sun Chenyang
cb0613e249 [fix] (inverted index) fix error result in compound query (#40425) 
## Proposed changes

`select count() from table where a + b > 0 or b > 0`


![image](https://github.com/user-attachments/assets/df56bb36-660d-4b4f-8e38-4eebcaa09e51)



- When _execute_predicates_except_leafnode_of_andnode is executed, an
Expr tree is traversed from bottom to top. When it reaches the leaf node
b, the information of this column b is placed into new_predicate_info.

- However, this step is skipped directly at an ADD node, which leads to
the GT node at the upper level generating a sign equivalent to b > 0,
the same as the sign on the right side b > 0.

- This causes the compound OR calculation to assume that both GT
conditions below have been evaluated, thus prematurely computing this
EXPR, when in fact, the ADD node has not been evaluated.

- If the SQL is written as SELECT COUNT(*) FROM table WHERE b + a > 0 OR
b > 0, the calculation would be correct because the sign generated by
this > node would be equivalent to a > 0, which is different from b > 0
on the right side.
 2024-09-06 10:27:59 +08:00
lihangyu
f64a728741 [enhance](variant) throw exception when field type is not supported in cast elimation (#40448) 
#40388
 2024-09-06 09:56:57 +08:00
TengJianPing
7e27bb1ae6 [fix](window_funnel) fix wrong result of fixed mode (#40460) 
BP #40459
 2024-09-06 09:48:24 +08:00
Socrates
87ac378c4a [branch-2.1](be-ut) wait lazy open in ut (#40453) 
## Proposed changes

LRUFileCache test need to wait lazy open done
 2024-09-06 09:47:47 +08:00
abmdocrt
88750e7e5d [cherry-pick](branch-2.1) Pick "[Fix](group commit) Fix table not found fault when disable group commit (#39731)" (#40323) 
<!--Describe your changes.-->

## Proposed changes

Pick #39731 

<!--Describe your changes.-->
 2024-09-05 21:39:15 +08:00
Dongyang Li
4eaf65d887 [fix](tools) adjust download url (#40429) (#40445) 
pick from master #40429

Co-authored-by: stephen <hello-stephen@qq.com>
 2024-09-05 20:58:59 +08:00
zhiqiang
0928c9c6ed [fix](unary function) Fix wrong result of asin, acos and sqrt when processing invalid input #40267 (#40358) 
cherry pick from #40267
 2024-09-05 19:51:01 +08:00
Calvin Kirs
961d2c9af5 [feat](job)Internal job cancellation immediately and the strong association with the STARTS parameter (#36805) (#38110) 
…

## Proposed changes

For internal tasks, such as MTMV, the start time may already be set, or
the time may be adjusted immediately.

<!--Describe your changes.-->

(cherry picked from commit 904a6c0fc1a804520285533de874fe4d0ffff2c1)

## Proposed changes

Issue Number: close #36805

<!--Describe your changes.-->
 2024-09-05 16:28:35 +08:00
abmdocrt
52393f829d [cherry-pick](branch-2.1) Fix some group commit fault (#40319) 
## Proposed changes

Pick #39986 #40120

<!--Describe your changes.-->
 2024-09-05 15:26:59 +08:00
yiguolei
40d10bdb0a Revert "Revert "[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value"" (#40411) 
Reverts apache/doris#40359
 2024-09-05 14:59:01 +08:00
abmdocrt
d9172bed64 [cherry-pick](branch-2.1) Pick "[Enhancement](delete) Modify some delete cases to ajust enable_mow_light_delete config (#40387)" (#40389) 
## Proposed changes

pick #40387

<!--Describe your changes.-->
 2024-09-05 14:56:44 +08:00
Pxl
d9fa59be4d [Chore](runtime-filter) avoid dcheck fail when rf merge failed (#39172) (#40409) 
pick from #39172
 2024-09-05 14:50:47 +08:00
camby
26feaab711 [fix](delete_predicate) fix wrong data after upgrade from v2.0 version (#40400) 
pick https://github.com/apache/doris/pull/40401 to branch-2.1

related issue: 
https://github.com/apache/doris/issues/40390

related pr:
https://github.com/apache/doris/pull/22442
 2024-09-05 14:46:03 +08:00
minghong
c6771da240 [fix](nereids) fix bug for A>n, where A.max is infinity #39936 (#40368) 
## Proposed changes
pick #39936 
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-05 08:33:04 +08:00
Luwei
cc20ecd738 Revert "[fix](compaction) fix the longest continuous rowsets cannot be selected when missing rowsets (#38728) (#39262)" (#40375) 
This reverts commit c9949f24e5c15e9529285f0e99b7ffdb1095558b.

This pr may increase the probability of full clone failure, so revert it
first.
 2024-09-05 00:01:03 +08:00
yiguolei
08fb91f161 Revert "[cherry-pick](branch-2.1) Fix enable_mow_light_delete default value" (#40359) 
Reverts apache/doris#40283
 2024-09-04 17:55:25 +08:00
qiye
653daeb8cb Revert "[enhancement](index compaction) Enable index compaction by default (#36812)" (#40351) 
Reverts apache/doris#38676
 2024-09-04 14:16:09 +08:00
Tiewei Fang
b4beec8ea8 [fix](OrcWriter) fix be core when upgrading BE without upgrading FE (#40303) 
bp: #40282
 2024-09-04 10:24:41 +08:00
feiniaofeiafei
db06c678a3 [Fix](planner) legacy planner repeat group by has grouping report error (#40281) (#40305) 
cherry-pick #40281 to branch-2.1
 2024-09-04 08:50:21 +08:00
Jerry Hu
3cf5d15b49 [fix](fe) avoid setting the value of batch_size excessively high (#35941) (#40312) 
Issue Number: close #xxx

pick #35941

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-03 19:49:26 +08:00
Siyang Tang
67d46297f2 [fix](regression-test) Sync after insert to get stable results (#40309) 
## Proposed changes

As title.
 2024-09-03 16:20:39 +08:00
amory
0e9fa3dff7 [fix](decimaltype) handle exception with tablet init (#40263) 
## Proposed changes
to avoid be core like 
```
terminate called after throwing an instance of 'doris::Exception'
  what():  [E6] meet invalid precision: real_precision=28, max_decimal_precision=27, min_decimal_precision=1

        0#  doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/exception
.cpp:0
        1#  doris::Exception::Exception<unsigned int const&, unsigned long>(int, std::basic_string_view<char, std::char_traits<char> > const&, unsigned int const&, unsigned long&
&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
        2#  doris::vectorized::DataTypeDecimal<doris::vectorized::Decimal<__int128> >::DataTypeDecimal(unsigned int, unsigned int, unsigned int, unsigned int) at /home/zcp/repo_c
enter/doris_branch-3.0/doris/be/src/vec/data_types/data_type_decimal.h:0
        3#  doris::vectorized::DataTypeFactory::create_data_type(doris::TypeDescriptor const&, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/data_types/data_ty
pe_factory.cpp:0
        4#  doris::vectorized::DataTypeFactory::create_data_type(doris::TypeDescriptor const&, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/data_types/data_ty
pe_factory.cpp:0
        5#  doris::vectorized::DataTypeFactory::create_data_type(doris::TypeDescriptor const&, bool) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/data_types/data_ty
pe_factory.cpp:0
        6#  doris::SlotDescriptor::get_empty_mutable_column() const at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base
.h:1295
        7#  doris::VOlapTablePartitionParam::VOlapTablePartitionParam(std::shared_ptr<doris::OlapTableSchemaParam>&, doris::TOlapTablePartitionParam const&) at /home/zcp/repo_cen
ter/doris_branch-3.0/doris/be/src/vec/common/cow.h:154
        8#  doris::vectorized::VTabletWriter::_init(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/vec/sink/writer/vtablet_w
riter.cpp:1177
        9#  doris::vectorized::VTabletWriter::open(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status.h:488
        10# doris::vectorized::AsyncResultWriter::process_block(doris::RuntimeState*, doris::RuntimeProfile*) at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/common/status
.h:488
        11# std::_Function_handler<void (), doris::vectorized::AsyncResultWriter::start_writer(doris::RuntimeState*, doris::RuntimeProfile*)::$_0>::_M_invoke(std::_Any_data const
&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/atomicity.h:98
        12# doris::ThreadPool::dispatch_thread() at /home/zcp/repo_center/doris_branch-3.0/doris/be/src/util/threadpool.cpp:0
        13# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
        14# ?
```

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-03 14:38:16 +08:00