Commit Graph

7374 Commits

Author SHA1 Message Date
f6ec64c6ad [fix](exception) Fix Block noexcept method not throw exception (#34002) 2024-04-24 17:13:50 +08:00
00d773117d [fix](stream agg) fix coredump when close if open failed (#33978) 2024-04-24 17:13:50 +08:00
080c07ad87 [bug](random distribution) fix data loss and incorrect in random distribution table #33962 2024-04-24 17:13:50 +08:00
799c43686c [fix](jni-connector) avoid core dump if init connector failed (#34007)
_jni_scanner_cls may be null if connector init failed.
So need to check it before delete it.
2024-04-24 17:13:50 +08:00
8d98c71079 [FIX]fix cidr func with const param (#33968) 2024-04-24 17:13:50 +08:00
df96f76f78 [featrue](pipelineX) check output type in some node (#33716) 2024-04-24 17:13:49 +08:00
9bb149b3be [fix](stream-load) fix query id is zero in stream load log (#33954) 2024-04-22 22:33:24 +08:00
Pxl
5a5063be20 [bug](fix) heap use after free when json parse failed (#33955) 2024-04-22 22:33:24 +08:00
4d7ac82305 [profile](scanner) Fix wrong metrics (#33965) 2024-04-22 22:33:24 +08:00
299d069da9 Fix alter policy failed (#33910) 2024-04-22 22:33:24 +08:00
a050513c91 [Fix](clean trash) Fix clean trash use agent task (#33912) (#33972)
* [Fix](clean trash) Fix clean trash use agent task (#33912)

* add .h
2024-04-22 17:14:21 +08:00
e384b495e3 [fix](pipeline_x) The execution loop of a task should be broken if the task is cancelled (#33938) 2024-04-22 12:31:55 +08:00
c631f4f8a8 [fix](schema change) resolve the use count check of source logical column (#33932)
Fix error like:
```
8# google::LogMessageFatal::~LogMessageFatal() in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
 9# doris::vectorized::Block::clear_column_data(int) in /mnt/hdd01/ci/master-deploy/be/lib/doris_be
10# doris::vectorized::ParquetReader::get_next_block(doris::vectorized::Block*, unsigned long*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/format/parquet/vparquet_reader.cpp:514
11# doris::vectorized::VFileScanner::_get_block_impl(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vfile_scanner.cpp:333
12# doris::vectorized::VScanner::get_block(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vscanner.cpp:132
13# doris::vectorized::VScanner::get_block_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /home/zcp/repo_center/doris_master/doris/be/src/vec/exec/scan/vscanner.cpp:99
```

Because source logical column is the destination logical column if logical converter is consistent. Previously, the reference of column was reset after the conversion was completed, but if an EOF occurred, it was returned in advance, but EOF is not a true error.
```
if (_logical_converter->is_consistent()) {
            // If logical converter is consistent, _src_logical_column is the final destination column,
            // other components will check the use count
            _src_logical_column.reset();
}
```
2024-04-22 12:31:46 +08:00
7f61626c8d [fix](arrow_flight_sql) Fix ArrowSchema column alias (#33490)
run: select TABLE_SCHEMA as a, sum(TABLE_ROWS) as b  from tables group by TABLE_SCHEMA limit 2;
old output:

          TABLE_SCHEMA                        Nullable(Int64)_1
0  regression_test_mv_p0_sum_count           9
1  regression_test_query_p0_sql_functions_string_functions       70414
now output:

          a                        b
0  regression_test_mv_p0_sum_count            9
1  regression_test_query_p0_sql_functions_string_functions       70414
2024-04-22 11:28:22 +08:00
615765c1c0 [improvement](spill) improve spill directory and fix bugs (#33900)
* [improvement](spill) improve spill directory and fix bugs

* fix
2024-04-22 11:28:22 +08:00
00ff5f05d3 [chore](log) Avoid too many 'token parser result is empty' (#33921) 2024-04-21 13:22:26 +08:00
cb2598e814 [bugfix](memtracker) memtracker is attached duplicately (#33929)
fix:

F20240420 12:47:23.222411 31558 thread_context.h:164] Check failed: thread_mem_tracker()->label() == "Orphan" , thread mem tracker label: Load#Id=b43f342ae5564c23-b7b41daf24545f78, attach mem tracker label: Load#Id=4241cef180013366-1ba9f658007f339a
12:49:46   *** Check failure stack trace: ***
12:49:46       @     0x55584aae5d26  google::LogMessage::SendToLog()
12:49:46       @     0x55584aae2770  google::LogMessage::Flush()
12:49:46       @     0x55584aae6569  google::LogMessageFatal::~LogMessageFatal()
12:49:46       @     0x55581abce4ae  doris::ThreadContext::attach_task()
12:49:46       @     0x55581abc8e8e  doris::AttachTask::AttachTask()
12:49:46       @     0x5558170a055b  doris::MemTableWriter::flush_async()
12:49:46       @     0x5558170604ee  doris::MemTableMemoryLimiter::_flush_memtable()
12:49:46       @     0x55581705e8e6  doris::MemTableMemoryLimiter::_flush_active_memtables()
12:49:46       @     0x55581705d986  doris::MemTableMemoryLimiter::handle_memtable_flush()
12:49:46       @     0x555848c9a36d  doris::vectorized::VTabletWriterV2::_write_memtable()
12:49:46       @     0x555848c990c8  doris::vectorized::VTabletWriterV2::write()
2024-04-21 09:55:48 +08:00
687951202f [refactor](opt) move BE code of hll scalar functions together, optimize head files (#33757)
In this PR, we moved the BE code of hll scalar functions together to manage better, like bitmap functions file does.

Also, we optimized the head files by:
removing useless file "vec/aggregate_functions/aggregate_function.h" and "boost/iterator/iterator_facade.hpp",
using cstddef and cstdint instead of stddef.h and stdint.h.
2024-04-21 09:55:19 +08:00
36a70ba1e7 [Fix](Csv-Reader)Fix the issue of BE core dump caused by improper configuration of column_seperator and line_delimiter. (#33693) 2024-04-20 20:06:48 +08:00
03c3419265 [Refactor](executor)Add workload schedule policy table (#33729) 2024-04-20 20:06:34 +08:00
0e3ad5cd9d [fix](parquet) fix time zone error(isAdjustedToUTC=true) in parquet reader (#33675) (#33924)
bp (#33675)

Co-authored-by: Ashin Gau <AshinGau@users.noreply.github.com>
2024-04-20 19:06:54 +08:00
1ca96a1611 [fix](stream-load) fix stream load and http stream metric error #33899 2024-04-20 11:43:49 +08:00
365fcec473 Revert "[Improvementation](join) empty_block shall be set true when build block only one row (#33721)"
This reverts commit f17ac173b4e8052cb130119bdec649169f66ac4e.
2024-04-19 23:52:24 +08:00
Pxl
09b973db49 [Chore](runtime-filter) adjust need_local_merge setting conditions (#33886) 2024-04-19 23:50:04 +08:00
bec7c36c46 [fix](stacktrace) Fix dwarf_location_info_mode is passed as parameter to stack trace (#33863)
dwarf_location_info_mode is passed as parameter to stack trace
2024-04-19 23:41:46 +08:00
25358564ca [Fix](compile) Fix gcc compile on master (#33864)
This is imported by #33511. wrongly used

ColumnStr<T> ();

which violate C++20 standard(see https://wg21.cmeerw.net/cwg/issue2237) but still supported by clang up until now(see llvm/llvm-project#58112)
2024-04-19 23:41:37 +08:00
74590e4836 [refine](node) Remove the cse DCHECK from the constructor (#33856)
It's possible that a failure in the fe caused the check to fail, and at that moment, it may not be possible to retrieve the corresponding query ID from be.out.
2024-04-19 23:41:37 +08:00
7e91e69eb9 [fix](compaction) fix single compaction (#33907)
* [fix](compaction)Fix single compaction to get all local versions #33849

add test and comment

* remove single replica compaction prepare input rowsets

reviesd
2024-04-19 23:30:25 +08:00
439027119e [fix](schema change) fix schema change check does not calculate reader merged rows (#33825) (#33908) 2024-04-19 22:57:25 +08:00
0ac7849a9d [exec](table_fun) opt bitmap/split vexplode table func performance (#33876) 2024-04-19 15:22:14 +08:00
c747714c18 [fix](memory) Fix ExecEnv destroy memory tracking (#33781)
disable memory tracking when ExecEnv destroy.
fix memory tracker label convert to query id
2024-04-19 15:03:10 +08:00
f4704b3821 [improvement](storage) support glibc <2.21 for system call eventfd (#33218)
support glibc <2.21 for system call eventfd
2024-04-19 15:03:10 +08:00
Pxl
175e85d616 [Bug](runtime-filter) fix coredump on no null string type rf (#33869)
fix coredump on no null string type rf
2024-04-19 15:03:06 +08:00
8b061c7055 [Enhancement](group commit) Add fault injection case for group commit 2024-04-19 15:03:06 +08:00
e38d844d40 [fix](multi-table-load) fix single stream multi table load cannot finish (#33816) 2024-04-19 15:03:06 +08:00
659900040f [Fix](inverted index) fix wrong need read data opt when encounters columnA > columnB predicate (#33855) 2024-04-19 15:03:06 +08:00
ffd9da44a2 [fix](move-memtable) fix commit may fail due to duplicated reports (#32403) 2024-04-19 15:02:49 +08:00
56eb5ea00c [enhancement](partial-update) print more log while missed some rowsets (#33711) 2024-04-19 15:01:57 +08:00
315f6e44c2 [Branch-2.1](Outfile) Fixed the problem that the concurrent Outfile wrote multiple Success files (#33870)
backport: #33016
2024-04-19 12:09:53 +08:00
Pxl
ba05ef4405 [Chore](runtime-filter) add tmp debug info to investigate unknown filter error #33857 2024-04-18 21:03:09 +08:00
1300317723 [Exec](join) Support column string64 to avoid join failed in string size overflow the uint32 (#33511) (#33850) 2024-04-18 19:43:08 +08:00
8f6f4cf0eb [Pick](Variant) pick #33734 #33766 #33707 to branch-2.1 (#33848)
* [Fix](Variant Type) forbit distribution info contains variant columns (#33707)

* [Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size (#33734)

insert_range_from should start from `size` with `count` elements for null map

* [Fix](Variant) check column index validation for extracted columns (#33766)
2024-04-18 19:42:44 +08:00
657a29fd9e [refactor](partitioner) refine get channel id logics (#33765) 2024-04-18 19:05:24 +08:00
Pxl
f17ac173b4 [Improvementation](join) empty_block shall be set true when build block only one row (#33721)
empty_block shall be set true when build block only one row
2024-04-18 19:05:17 +08:00
ea19224d14 [exec](table_fun) opt numbers table func performance (#33804) 2024-04-18 19:04:03 +08:00
ad80a650e4 [fix][mow] segment_creator should not flush data when encouter abnormal exit (#33802) 2024-04-18 19:02:58 +08:00
04e30c91a0 [Fix](Variant) VariantRootColumnIterator::read_by_rowids with wrong null map size (#33734)
insert_range_from should start from `size` with `count` elements for null map
2024-04-18 19:02:58 +08:00
5a5b0c07d7 [fix](inverted index) fix incorrect case test_index_delete (#33609) 2024-04-18 19:02:49 +08:00
20b37e7a18 Add workload group id in workload policy's property (#33483) 2024-04-17 23:42:14 +08:00
ee3b6fdf58 [fix](conf) make be conf disable_storage_page_cache modifiable (#33773)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2024-04-17 23:42:14 +08:00