database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,269 Commits 1 Branch 0 Tags
386c55f00aa9cd2f178eefb1d41f6e082e0715cd
Commit Graph

8349 Commits

Author SHA1 Message Date
Mingyu Chen
5b3b2cec80 [feat](metatable) support table$partitions for hive table (#40774) (#41230) 
bp #40774
and pick part of #34552, add `isPartitionedTable()` interface in `TableIf`
 2024-09-25 09:52:07 +08:00
Sun Chenyang
8bb57bcc3e [fix] (inverted index) fix the error in the query result when using count on index (#41200) 
## Proposed changes

Introduced by  #39473

<!--Describe your changes.-->
 2024-09-24 19:47:18 +08:00
lihangyu
78adf64353 [compability](variant) fix nullable serde during upgrading to 3.0 (#41191) 
need to check nullable during serialization when upgrading to 3.0
 2024-09-24 19:44:54 +08:00
hui lai
d4c1b39d03 [fix](multi table) restrict the multi tables load memory under high concurrency with a large number of tables (#39992) (#41131) 
pick (#39992)

BE node was killed by OOM-killer when use multi table load under high
concurrency with a large number of tables(128 concurrency and every
concurrency load 200 tables).

This pr restricts the multi tables load memory under this issue. If
memory reaches hard limit, new task will be rejected and return
directly.
 2024-09-24 16:34:32 +08:00
hui lai
4237341e0d [fix](move-memtable) do not execute close if create rowset failed when loading MOW table (#40105) (#41132) 
pick (#40105)

Core dump happened when load to MOW table:
```
Check failure stack trace: ***
@ 0x55fae437d246 google::LogMessage::SendToLog()
@ 0x55fae4379c90 google::LogMessage::Flush()
@ 0x55fae437da89 google::LogMessageFatal::~LogMessageFatal()
@ 0x55faacf26bbf doris::BaseTablet::check_delete_bitmap_correctness()
@ 0x55fab05049ef doris::RowsetBuilder::commit_txn()
@ 0x55fab09026e8 doris::LoadStreamWriter::close()
@ 0x55fab089eff7 std::_Function_handler<>::_M_invoke()
@ 0x55fab0d14d7c doris::WorkThreadPool<>::work_thread()
@ 0x55fae76ae6f0 execute_native_thread_routine
@ 0x7fa32ea45ac3 (unknown)
@ 0x7fa32ead7850 (unknown)
@ (nil) (unknown)
Query id: a21981d5c8ef4113-84df9a5a8680e004 ***
is nereids: 0 ***
tablet id: 0 ***
Aborted at 1724668499 (unix time) try "date -d @1724668499" if you are using GNU date ***
Current BE git commitID: 2f848737c1 ***
SIGABRT unknown detail explain (@0x20db) received by PID 8411 (TID 9837 OR 0x7f9e42cfe640) from PID 8411; stack trace: ***
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /home/zcp/repo_center/doris_master/doris/be/src/common/signal_handler.h:421
1# 0x00007FA32E9F3520 in /lib/x86_64-linux-gnu/libc.so.6
2# pthread_kill at ./nptl/pthread_kill.c:89
3# raise at ../sysdeps/posix/raise.c:27
4# abort at ./stdlib/abort.c:81
5# 0x000055FAE4387B1D in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
6# 0x000055FAE437A15A in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
7# google::LogMessage::SendToLog() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
8# google::LogMessage::Flush() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
9# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
10# doris::BaseTablet::check_delete_bitmap_correctness(std::shared_ptr, long, long, std::unordered_set, std::equal_to, std::allocator > const&, std::vector, std::allocator > >*) at /home/zcp/repo_center/doris_master/doris/be/src/olap/base_tablet.cpp:1152
11# doris::RowsetBuilder::commit_txn() at /home/zcp/repo_center/doris_master/doris/be/src/olap/rowset_builder.cpp:316
12# doris::LoadStreamWriter::close() at /home/zcp/repo_center/doris_master/doris/be/src/runtime/load_stream_writer.cpp:311
13# std::_Function_handler::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
14# doris::WorkThreadPool::work_thread(int) at /home/zcp/repo_center/doris_master/doris/be/src/util/work_thread_pool.hpp:159
15# execute_native_thread_routine at ../../../../../libstdc+-v3/src/c+11/thread.cc:84
16# start_thread at ./nptl/pthread_create.c:442
17# 0x00007FA32EAD7850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```

If create rowset failed,` calc_delete_bitmap_task` still could be
executed:
```
add segment failed load_id=5649413b98976f0d-a105b42749f561b0, txn_id=2, tablet_id=10088, status=[INTERNAL_ERROR]create row
set failed
...
submit calc delete bitmap task to executor, tablet_id: 10088, txn_id: 2
```

This PR skips close to avoid `submit_calc_delete_bitmap_task` if create
rowset failed when loading MOW table to solve this problem.
 2024-09-24 16:02:22 +08:00
hui lai
d7659ff34d [fix](bytebuffer) fix allocate size improper in append_and_flush (#40613) (#41133) 
pick (#40613)

fix allocate size improper in append_and_flush introduced by
https://github.com/apache/doris/pull/38960
 2024-09-24 16:01:52 +08:00
hui lai
602275d1fe [improvement](stream load) display the time of receiving network data (#40735) (#41139) 
pick (#40735)

Display the time of receiving network data.

For example:
```
Stream load result: {
    "TxnId": 19,
    "Label": "c10ee032-c19c-484f-80b2-16f6c1d0c209",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 6,
    "NumberLoadedRows": 6,
    "NumberFilteredRows": 0,
    "NumberUnselectedRows": 0,
    "LoadBytes": 60,
    "LoadTimeMs": 32,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 4,
    "ReceiveDataTimeMs": 6,
    "CommitAndPublishTimeMs": 22
}
```
 2024-09-24 16:00:28 +08:00
walter
ddd17be874 [fix](migrate) Fix iterator returning too early #41168 (#41189) 
cherry pick from #41168
 2024-09-24 14:10:10 +08:00
daidai
2b427c316a [feature](functions) impl scalar functions normal_cdf,to_iso8601,from_iso8601_date (#40695) (#41049) 
bp #40695
 2024-09-24 09:52:39 +08:00
walter
0d38a9a36d [feature](restore) support atomic restore (#41107) 
Cherry-pick #40353, #40734, #40817, #40876, #40921, #41017, #41083
 2024-09-24 09:41:41 +08:00
Socrates
f6917acd6a [cherry-pick](branch2.1) Impl translate and url encode 2.1 (#41051) 
## Proposed changes

pick https://github.com/apache/doris/pull/40567

some code about const folding should wait the pr picked:
https://github.com/apache/doris/pull/40441
 2024-09-23 14:26:27 +08:00
yiguolei
011baeb2d2 [bugfix](allocatebytes) ignore null ptr column in Block (#41093) (#41099) 
## Proposed changes
Sometimes if expr failed, then there will be a nullptr column left in
the block.
We should ignore the nullptr column, or exception will be thrown and
some profile will not computed correctly.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-09-23 12:18:22 +08:00
Mryange
0e5c4281dc [fix](function) fix Substring/SubReplace error result with input utf8… (#40954) 
… string (#40929)
https://github.com/apache/doris/pull/40929
```

mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1)     |
+-------------------------------------+
| �a�好世界                             |
+-------------------------------------+

mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647)     |
+------------------------------------------+
| 中文测试                                 |
+------------------------------------------+
1 row in set (0.04 sec)

now
mysql [(none)]>select sub_replace("你好世界","a",1);
+-------------------------------------+
| sub_replace('你好世界', 'a', 1)     |
+-------------------------------------+
| 你a世界                             |
+-------------------------------------+
1 row in set (0.05 sec)

mysql [(none)]>select SUBSTRING('中文测试',5);
+------------------------------------------+
| substring('中文测试', 5, 2147483647)     |
+------------------------------------------+
|                                          |
+------------------------------------------+
1 row in set (0.13 sec)
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-23 10:44:03 +08:00
bobhan1
7d64c8cbc6 [branch-2.1] Picks "[opt](autoinc) Remove some restrictions on schema change on table that has auto-increment column #40280" (#41096) 
picks https://github.com/apache/doris/pull/40280
 2024-09-23 09:30:15 +08:00
Siyang Tang
d1d52ae68c [feature](compaction) Add an http action for visibility of compaction score on each tablet (#38489) (#40826) 
pick: #38489 

Usage:
1. `curl http://be_ip:be_host/api/compaction_score?top_n=10` Returns a
json object contains compaction score for top n, n=top_n.
```
[
    {
        "compaction_score": "5",
        "tablet_id": "42595"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42587"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42593"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42597"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42589"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42599"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42601"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42591"
    },
    {
        "compaction_score": "5",
        "tablet_id": "42585"
    },
    {
        "compaction_score": "4",
        "tablet_id": "10034"
    }
]
```
If top_n is not specified, return all compaction score for all tablets.
If top_n is illegal, raise an error.
```
invalid argument: top_n=wrong
```

2. `curl http://be_ip:be_host/api/compaction_score?sync_meta=true`
`sync_meta` is only available on cloud mode, will sync meta from meta
service. It can cooperate with top_n.
If add param `sync_meta` on non-cloud mode, will raise an error.
```
sync meta is only available for cloud mode
```

3. In the future, this endpoint may extend other utility, like fetching
tablet compaction score by table id, etc.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-21 20:35:55 +08:00
Mryange
9877a08834 [feature](function) support ngram_search function #38226 (#40893) 
https://github.com/apache/doris/pull/38226 
mysql [test]>select ngram_search('123456789' , '12345' , 3);
+---------------------------------------+
| ngram_search('123456789', '12345', 3) |
+---------------------------------------+
|                                   0.6 |
+---------------------------------------+
1 row in set (0.01 sec)

mysql [test]>select ngram_search("abababab","babababa",2);
+-----------------------------------------+
| ngram_search('abababab', 'babababa', 2) |
+-----------------------------------------+
|                                       1 |
+-----------------------------------------+
1 row in set (0.01 sec)
```

doc https://github.com/apache/doris-website/pull/899

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-21 20:34:44 +08:00
Qi Chen
36a45c5ebf [Fix](parquet-reader) Fix parquet reader crash in set_dict(). (#41074) 
## Proposed changes

Backport #40643
 2024-09-21 08:48:06 +08:00
lihangyu
c6a6adb3a4 [Fix](topn) avoid missmatched row count when upgrading (#40999) 
#41000
 2024-09-21 08:46:57 +08:00
daidai
c9d71cdc64 [Enhancement](brpc)Added enable_brpc_builtin_services parameter in be.conf (#40718) (#41047) 
bp #40718 

## Proposed changes

Since [brpc's built-in
services](https://brpc.apache.org/zh/docs/builtin-services/buildin_services/)
will expose some internal server status, for security reasons, add the
parameter `enable_brpc_builtin_services` in `be.conf`. When the
parameter is false, this built-in service can be disabled.
 2024-09-21 08:45:38 +08:00
Kaijie Chen
96b2fa1a24 [fix](move-memtable) fix bvar g_load_stream_file_writer_cnt (#39075) (#40985) 
backport #39075
 2024-09-20 20:20:49 +08:00
Qi Chen
7e0d498043 [Fix](orc-reader) Fix StringRef nullptr data in orc-reader. (#41073) 
## Proposed changes

backport #40857
 2024-09-20 20:06:50 +08:00
Siyang Tang
a8aebe6394 [fix](scanner-builder) Fix scan builder datarace beacause of picked from master's different codebase (#41019) 
## Proposed changes

As title. Master will hold header lock internally when capturing read
resource.
 2024-09-20 18:17:59 +08:00
Mingyu Chen
596cfc9b18 [opt](cpu-profile) enable cpu profile in BE webui (#40330) (#41044) 
bp #40330
 2024-09-20 17:18:16 +08:00
Mingyu Chen
8e860a26a7 [fix](systable) fix unstable case for partitions table (#40553) (#41043) 
bp #40553
 2024-09-20 17:13:30 +08:00
airborne12
68b7783e3a [Fix](scanner) lazy update block mem usage when status is not ok #40953 (#41034) 
cherry pick from #40953
 2024-09-20 15:09:26 +08:00
TengJianPing
6539c8fd35 [fix](decimal) throw overflow exception if result is NaN of Infinit when converting from decimal to float (#40290) (#41007) 
## Proposed changes

Issue Number: close #xxx

BP #40290
 2024-09-20 14:05:51 +08:00
Socrates
e0fac66223 [branch-2.1](fix) fix snappy decompressor bug (#40862) 
## Proposed changes
Hadoop snappycodec source :

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/codec/SnappyCodec.cc
Example:
OriginData(The original data will be divided into several large data
block.) :
     large data block1 | large data block2 | large data block3 | ....
The large data block will be divided into several small data block.
Suppose a large data block is divided into three small blocks:
large data block1: | small block1 | small block2 | small block3 |
CompressData: <A [B1 compress(small block1) ] [B2 compress(small block1)
] [B3 compress(small block1)]>

A : original length of the current block of large data block.
sizeof(A) = 4 bytes.
A = length(small block1) + length(small block2) + length(small block3)
Bx : length of  small data block bx.
sizeof(Bx) = 4 bytes.
Bx = length(compress(small blockx))
 2024-09-20 11:57:14 +08:00
Kaijie Chen
549bc3e288 [fix](pipelinex) fix fragment instance progress reports (#40325) (#40987) 
backport #40325
 2024-09-19 23:58:38 +08:00
bobhan1
49dec9f39d [branch-2.1] Picks "[opt](merge-on-write) Reduce the version not continuous logs for merge-on-write table #40946" (#40996) 
picks https://github.com/apache/doris/pull/40946
 2024-09-19 23:58:05 +08:00
Kaijie Chen
f32d540ac5 [test](move-memtable) add more fault injection cases for writer v2 (#39169) (#40986) 
backport #39169
 2024-09-19 22:31:12 +08:00
Kaijie Chen
b7ded71bfb [fix](stream-load) catch exception when parsing CONTENT_LENGTH (#40796) (#40988) 
backport #40796
 2024-09-19 22:15:08 +08:00
Jerry Hu
b8bc9b699c [fix](scan) Incorrect scan keys lead to wrong query results. (#40814) (#40971) 
## Proposed changes
pick #40814
```
mysql [doris_14555]>select * from table_9436528_3;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4                   | col6               | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 |    1 |  -82 |    1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 |    1 |   92 |    1 | 2024-02-16 04:37:37.00 |   23423423.0324234 | NULL |
| -100 |    0 |  -82 |    0 | 2023-11-11 10:49:43.00 |   840968969.872149 | NULL |
```
wrong result:
```
mysql [doris_14555]>select * from table_9436528_3 where col1 <= -100 and col2 in (true, false) and col3 = -82;
+------+------+------+------+------------------------+--------------------+------+
| col1 | col2 | col3 | col5 | col4                   | col6               | col7 |
+------+------+------+------+------------------------+--------------------+------+
| -100 |    1 |  -82 |    1 | 2024-02-16 04:37:37.00 | -1299962421.904282 | NULL |
| -100 |    1 |   92 |    1 | 2024-02-16 04:37:37.00 |   23423423.0324234 | NULL |
+------+------+------+------+------------------------+--------------------+------+
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-19 22:01:02 +08:00
Xin Liao
f2b93d5896 [Fix](load) Restrict the import of VARCHAR(0) data to avoid coredump #40940 (#40974) 
cherry pick from #40940
 2024-09-19 17:40:46 +08:00
amory
f483a7605c [fix](nestedtypes) fix nested type with is_exclusive (#40434) 
is_exclusive in column_array/map/struct has wrong semantic , we should
make sure it's nested column is also is_exclusive which can make
behavior right in some operator like join.

## Proposed changes
backport: https://github.com/apache/doris/pull/40398
Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-19 12:28:51 +08:00
Jerry Hu
fbed56639c [fix](compaction) catch exception in compaction (#40900) (#40930) 
## Proposed changes
pick #40900 
```

terminate called after throwing an instance of 'doris::Exception'
 {color:red} what():  [E6] Too large string size.{color}

	0#  doris::Exception::Exception(int, std::basic_string_view<char, std::char_traits<char> > const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/unique_ptr.h:173
	1#  doris::vectorized::read_string_binary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, doris::vectorized::BufferReadable&, unsigned long) at /root/doris_branch-2.1/doris/be/src/vec/io/io_helper.h:177
	2#  doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullUnaryInline<doris::vectorized::AggregateFunctionGroupConcat<doris::vectorized::AggregateFunctionGroupConcatImplStr>, true> >::deserialize_and_merge_from_column_range(char*, doris::vectorized::IColumn const&, unsigned long, unsigned long, doris::vectorized::Arena*) const at /root/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:0
	3#  doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateStateUnion>::add_batch_range(unsigned long, unsigned long, char*, doris::vectorized::IColumn const**, doris::vectorized::Arena*, bool) at /root/doris_branch-2.1/doris/be/src/vec/aggregate_functions/aggregate_function.h:0
	4#  doris::vectorized::VerticalBlockReader::_update_agg_value(std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, int, int, bool) at /root/doris_branch-2.1/doris/be/src/vec/olap/vertical_block_reader.cpp:326
	5#  doris::vectorized::VerticalBlockReader::_update_agg_data(std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&) at /root/doris_branch-2.1/doris/be/src/vec/olap/vertical_block_reader.cpp:308
	6#  doris::vectorized::VerticalBlockReader::_agg_key_next_block(doris::vectorized::Block*, bool*) at /root/doris_branch-2.1/doris/be/src/vec/olap/vertical_block_reader.cpp:0
	7#  doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /root/doris_branch-2.1/doris/be/src/common/status.h:491
	8#  doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::Tablet>, doris::ReaderType, std::shared_ptr<doris::TabletSchema>, bool, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::RowSourcesBuffer*, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector<unsigned int, std::allocator<unsigned int> >, long, doris::CompactionSampleInfo*) at /root/doris_branch-2.1/doris/be/src/olap/merger.cpp:0
	9#  doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::Tablet>, doris::ReaderType, std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /root/doris_branch-2.1/doris/be/src/olap/merger.cpp:445
	10# doris::Compaction::do_compaction_impl(long) at /root/doris_branch-2.1/doris/be/src/olap/compaction.cpp:385
	11# doris::Compaction::do_compaction(long) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1291
	12# doris::CumulativeCompaction::execute_compact_impl() at /root/doris_branch-2.1/doris/be/src/common/status.h:491
	13# doris::Compaction::execute_compact() at /root/doris_branch-2.1/doris/be/src/common/status.h:491
	14# doris::Tablet::execute_compaction(doris::Compaction&) at /root/doris_branch-2.1/doris/be/src/common/status.h:491
	15# std::_Function_handler<void (), doris::StorageEngine::_submit_compaction_task(std::shared_ptr<doris::Tablet>, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /root/doris_branch-2.1/doris/be/src/olap/olap_server.cpp:1018
	16# doris::ThreadPool::dispatch_thread() at /root/doris_branch-2.1/doris/be/src/util/threadpool.cpp:0
	17# doris::Thread::supervise_thread(void*) at /var/local/ldb-toolchain/bin/../usr/include/pthread.h:562
	18# ?
	19# ?

*** Query id: 0-0 ***
*** is nereids: 0 ***
*** tablet id: 38266341 ***
*** Aborted at 1726615633 (unix time) try "date -d @1726615633" if you are using GNU date ***
*** Current BE git commitID: db06c678a3 ***
*** SIGABRT unknown detail explain (@0x26af02) received by PID 2535170 (TID 2536168 OR 0x7f48c81cd640) from PID 2535170; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris_branch-2.1/doris/be/src/common/signal_handler.h:421
 1# 0x00007F4AB18F8520 in /lib/x86_64-linux-gnu/libc.so.6
 2# pthread_kill at ./nptl/pthread_kill.c:89
 3# raise at ../sysdeps/posix/raise.c:27
 4# abort at ./stdlib/abort.c:81
 5# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
 6# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
 7# 0x000055C401EF7811 in /mnt/disk1/STRESS_ENV/be/lib/doris_be
 8# 0x000055C401EF7964 in /mnt/disk1/STRESS_ENV/be/lib/doris_be
 9# doris::vectorized::read_string_binary(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, doris::vectorized::BufferReadable&, unsigned long) at /root/doris_branch-2.1/doris/be/src/vec/io/io_helper.h:180
10# doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateFunctionNullUnaryInline<doris::vectorized::AggregateFunctionGroupConcat<doris::vectorized::AggregateFunctionGroupConcatImplStr>, true> >::deserialize_and_merge_from_column_range(char*, doris::vectorized::IColumn const&, unsigned long, unsigned long, doris::vectorized::Arena*) const in /mnt/disk1/STRESS_ENV/be/lib/doris_be
11# doris::vectorized::IAggregateFunctionHelper<doris::vectorized::AggregateStateUnion>::add_batch_range(unsigned long, unsigned long, char*, doris::vectorized::IColumn const**, doris::vectorized::Arena*, bool) in /mnt/disk1/STRESS_ENV/be/lib/doris_be
12# doris::vectorized::VerticalBlockReader::_update_agg_value(std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&, int, int, bool) at /root/doris_branch-2.1/doris/be/src/vec/olap/vertical_block_reader.cpp:326
13# doris::vectorized::VerticalBlockReader::_update_agg_data(std::vector<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn>, std::allocator<COW<doris::vectorized::IColumn>::mutable_ptr<doris::vectorized::IColumn> > >&) at /root/doris_branch-2.1/doris/be/src/vec/olap/vertical_block_reader.cpp:308
14# doris::vectorized::VerticalBlockReader::_agg_key_next_block(doris::vectorized::Block*, bool*) in /mnt/disk1/STRESS_ENV/be/lib/doris_be
15# doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /root/doris_branch-2.1/doris/be/src/vec/olap/vertical_block_reader.cpp:58
16# doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::Tablet>, doris::ReaderType, std::shared_ptr<doris::TabletSchema>, bool, std::vector<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::RowSourcesBuffer*, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector<unsigned int, std::allocator<unsigned int> >, long, doris::CompactionSampleInfo*) in /mnt/disk1/STRESS_ENV/be/lib/doris_be
17# doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::Tablet>, doris::ReaderType, std::shared_ptr<doris::TabletSchema>, std::vector<std::shared_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics*) at /root/doris_branch-2.1/doris/be/src/olap/merger.cpp:445
18# doris::Compaction::do_compaction_impl(long) at /root/doris_branch-2.1/doris/be/src/olap/compaction.cpp:385
19# doris::Compaction::do_compaction(long) at /root/doris_branch-2.1/doris/be/src/olap/compaction.cpp:136
20# doris::CumulativeCompaction::execute_compact_impl() at /root/doris_branch-2.1/doris/be/src/olap/cumulative_compaction.cpp:79
21# doris::Compaction::execute_compact() at /root/doris_branch-2.1/doris/be/src/olap/compaction.cpp:118
22# doris::Tablet::execute_compaction(doris::Compaction&) at /root/doris_branch-2.1/doris/be/src/olap/tablet.cpp:2067
23# std::_Function_handler<void (), doris::StorageEngine::_submit_compaction_task(std::shared_ptr<doris::Tablet>, doris::CompactionType, bool)::$_0>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291
24# doris::ThreadPool::dispatch_thread() in /mnt/disk1/STRESS_ENV/be/lib/doris_be
25# doris::Thread::supervise_thread(void*) at /root/doris_branch-2.1/doris/be/src/util/thread.cpp:499
26# start_thread at ./nptl/pthread_create.c:442
27# 0x00007F4AB19DC850 at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:83
```

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-19 01:08:34 +08:00
yiguolei
76c7bf0785 [bugfix](loadmem) load memory is too small because flush memory is not right (#40696) (#40884) 2024-09-16 12:25:30 +08:00
yiguolei
a6e2364f56 [cherrypick](branch21) pick some exception handleing pr in page io (#40883) 
## Proposed changes

pick
https://github.com/apache/doris/pull/39536
https://github.com/apache/doris/pull/39913
https://github.com/apache/doris/pull/39964
https://github.com/apache/doris/pull/40020

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-09-16 08:56:43 +08:00
Xinyi Zou
b52b572ade [branch-2.1](memory) When Load ends, check memory tracker value returns is equal to 0 (#40850) 
pick
#38960
#39908
#40043
#40092
#40016
#40439

---------

Co-authored-by: hui lai <1353307710@qq.com>
Co-authored-by: yiguolei <676222867@qq.com>
 2024-09-15 23:47:53 +08:00
camby
f3b1f1c19b [fix](encrypt) wrong mode arg of encrypt and decrypt function make BE crash (#40726) (#40868) 
pick #40726 to branch-2.1
 2024-09-15 21:31:00 +08:00
bobhan1
ea4d166edb [branch-2.1] Picks "[opt](partial update) Allow to only specify key columns in partial update #40736" (#40863) 
picks https://github.com/apache/doris/pull/40736
 2024-09-15 18:57:43 +08:00
zclllhhjj
cecd214345 [branch-2.1](Column) refactor ColumnNullable to provide flags safety (#40769) (#40848) 
pick https://github.com/apache/doris/pull/40769

Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
 2024-09-14 16:27:43 +08:00
Jibing-Li
0b1d517caa [improvement](statistics)Return -1 to neredis if report olap table row count for new table is not done for all tablets. (#40457) (#40838) 
backport: https://github.com/apache/doris/pull/40457
 2024-09-14 13:19:35 +08:00
Xinyi Zou
f16615a1fc [branch-2.1](memory) Allocator support address sanitizers (#40836) 
pick
#33396
#33862
#33853
#33732
#33841
#33933
#34901
#35014

---------

Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
 2024-09-14 12:12:44 +08:00
Siyang Tang
341b2e0693 [enhancement](compaction) Abort compaction tasks when correspoding tablet states have been changed (#40271) (#40828) 
## Proposed changes

pick: #40271

1. Change the standard for cumu compaction capability. Tablets under
state `running` or `not ready` are capable to do cumulative compaction.
2. Abort a compaction task at the beginning when the tablet is no more
capable to do compaction.
 2024-09-14 11:19:31 +08:00
Siyang Tang
c79621fff1 [enhancement](schema-change) Make the schema change memory space adaptive (#40822) 
pick: #34350
 2024-09-14 11:17:41 +08:00
zhangstar333
9a79edca84 [cherry-pick](branch-21) fix partition_topn not reset output rows after do_partition_topn_sort (#40761) (#40792) 
## Proposed changes

cherry-pick from master https://github.com/apache/doris/pull/40761

<!--Describe your changes.-->
 2024-09-14 11:15:56 +08:00
Siyang Tang
523f0baa80 [fix](scan) Fix incorrect query results due to data race of compaction and parallel scanners building (#40552) (#40829) 
pick: #40552

Capture rowset splits and delete predicates atomicly in
`ParallelScannerBuilder::_load` as a single read source.

In this way, we could prevent reading stale rowsets with the delete
predicates eliminated by (base) compaction.
 2024-09-14 11:08:55 +08:00
Sun Chenyang
873f70c262 [fix] (compaction) fix compaction score in time series policy (#40242) (#40779) 
## Proposed changes

pick from master #40242

<!--Describe your changes.-->
 2024-09-13 14:35:16 +08:00
Kaijie Chen
7851563829 [fix](brpc_client_cache) resolve hostname in DNS cache before passing to brpc (#40074) (#40786) 
backport #40074
 2024-09-13 14:28:01 +08:00
Gabriel
3395cd5ce9 [PipelineX](improvement) Prepare tasks in parallel (#40270) 
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-09-13 13:34:29 +08:00