b244c41371
[Bug](regression-test) Fix grace stop be coredump in pipeline ( #18076 )
2023-03-24 17:44:06 +08:00
e8b9587fe6
[Improvement](dict) compute hash only if needed ( #18058 )
2023-03-24 11:45:58 +08:00
1999cccde9
[feature](array-type) Unique table support array value ( #17024 )
...
Unique table support array value
---------
Co-authored-by: huangqixiang.871 <huangqixiang.871@bytedance.com >
2023-03-24 10:18:59 +08:00
5445a86570
[Bug](array_product) Fix array_product for ARRAY<DECIMAL> ( #18014 )
2023-03-23 20:29:50 +08:00
b0948ea4cd
[Fix](SAP Hana External Table) fix that SAP Hana external table can not insert batch values ( #17957 )
...
In the batch insertion scenario, sap hana database does not support syntax insert into tables values (...),(...);
what it supports is:
```sql
INSERT INTO table(col1,col2)
SELECT c1v1, c2v1 FROM dummy
UNION ALL
SELECT c1v2, c2v2 FROM dummy;
```
2023-03-23 18:49:50 +08:00
cedd36c786
[improvement](compaction)Support segcompaction for inverted index ( #17874 )
...
Since Doris supports segcompaction #12866 during loading, inverted index support is also needed.
2023-03-23 14:41:30 +08:00
11936d85f9
[fix](inverted index) fix erroneous judgement for inverted index not read raw data ( #17992 )
...
when apply inverted index will use predicate_params() from ColumnPredicate, if comparison predicate be cloned, but the clone one not copy the predicate_params() together, that resulting when applying inverted index make the wrong choice.
2023-03-23 14:40:08 +08:00
4b626d260a
[Build] fix build fail when WITH_MYSQL=OFF ( #18021 )
2023-03-23 14:01:21 +08:00
0bb04c08aa
[improvement](coverage) build be with coverage enabled, which can get coverage data with llvm-cov-15 ( #17995 )
2023-03-23 12:07:19 +08:00
3870689cbb
[Fix](parquet-reader) Fix iceberg_schema_evolution regression test caused by slot col name different with parquet col name. ( #17988 )
2023-03-23 11:23:08 +08:00
089a91ecd5
[vectorized](function) support array_exists lambda function ( #17931 )
...
Co-authored-by: zhangyu209 <zhangyu209@meituan.com >
2023-03-23 11:11:39 +08:00
cfa0a8b136
[Improvement](DECIMALV3) multiply/plus DECIMAL32 and DECIMAL64 safely and not check overflow ( #18031 )
2023-03-23 10:10:03 +08:00
4be1b9e784
[enhancement](load) add slow log for memtable flush ( #17962 )
2023-03-22 20:21:39 +08:00
e2e806a5e7
[improve](clickhouse jdbc) support clickhouse array type ( #17993 )
...
In this PR, I match the array type of ClickHouse to the array type of Doris's jdbc external.
2023-03-22 19:42:32 +08:00
ebef0c038d
Revert "[fix](function) fix AES/SM3/SM4 encrypt/ decrypt algorithm initialization vector bug ( #17420 )" ( #17887 )
...
This reverts commit 397cc011c4f1ba5a25c770258c13f1cd3f28b47d.
2023-03-22 13:28:25 +08:00
34ead026d4
[Improvement](decimal) Improve cast function between decimal type ( #17996 )
2023-03-22 11:35:07 +08:00
40ca250678
[Feature](materialized-view) support where clause on create materialized view ( #17534 )
...
support where clause on create materialized view
2023-03-22 11:25:13 +08:00
401836f523
[Bug](planner) fix core dump when lateral view above union node and have predicate ( #17912 )
...
fix core dump when lateral view above union node and have predicate
2023-03-22 11:24:45 +08:00
6cbf393665
[enhance](meta action) remove useless pb field and refactor writer cooldown meta code ( #17652 )
2023-03-22 11:13:13 +08:00
7fd0ec7d17
[Bug](float) fix wrong value when enable fold constant by BE ( #17901 )
2023-03-22 09:51:03 +08:00
cb79e42e5c
[refactor](file-system)(step-1) refactor file sysmte on BE and remove storage_backend ( #17586 )
...
See #17764 for details
I have tested:
- Unit test for local/s3/hdfs/broker file system: be/test/io/fs/file_system_test.cpp
- Outfile to local/s3/hdfs/broker.
- Load from local/s3/hdfs/broker.
- Query file on local/s3/hdfs/broker file system, with table value function and catalog.
- Backup/Restore with local/s3/hdfs/broker file system
Not test:
- cold & host data separation case.
2023-03-21 21:08:38 +08:00
4193884a32
[feature](array_zip) Support array_zip function ( #17696 )
2023-03-21 18:44:30 +08:00
7754619e2b
[fix](quit) be can not quit cleanly due to deadlock ( #17971 )
2023-03-21 12:52:48 +08:00
656b01d191
[fix](agg) Avoid reusing a non-nullable column that has been converted to nullable within a block ( #17944 )
...
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:420
1# os::Linux::chained_handler(int, siginfo*, void*) in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
2# JVM_handle_linux_signal in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
3# signalHandler(int, siginfo*, void*) in /usr/local/java/jdk1.8.0_202/jre/lib/amd64/server/libjvm.so
4# 0x00007F4051C9F400 in /lib64/libc.so.6
5# memcpy at /root/doris/be/src/glibc-compatibility/memcpy/memcpy_x86_64.cpp:219
6# doris::vectorized::ColumnString::deserialize_and_insert_from_arena(char const*) at /root/doris/be/src/vec/columns/column_string.cpp:226
7# doris::vectorized::ColumnString::deserialize_vec_with_null_map(std::vector<StringRef, std::allocator<StringRef> >&, unsigned long, unsigned char const*) at /root/doris/be/src/vec/columns/column_string.cpp:283
8# void doris::vectorized::AggregationNode::_serialize_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*)::{lambda(auto:1&&)#1}::operator()<doris::vectorized::AggregationMethodSerialized<PHHashMap<StringRef, char*, DefaultHash<StringRef, void>, false> >&>(doris::vectorized::AggregationMethodSerialized<PHHashMap<StringRef, char*, DefaultHash<StringRef, void>, false> >&) const at /root/doris/be/src/vec/exec/vaggregation_node.cpp:1232
9# doris::vectorized::AggregationNode::_serialize_with_serialized_key_result(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/vaggregation_node.cpp:1294
10# std::_Function_handler<doris::Status (doris::RuntimeState*, doris::vectorized::Block*, bool*), std::_Bind_result<doris::Status, doris::Status (doris::vectorized::AggregationNode::*(doris::vectorized::AggregationNode*, std::_Placeholder<1>, std::_Placeholder<2>, std::_Placeholder<3>))(doris::RuntimeState*, doris::vectorized::Block*, bool*)> >::_M_invoke(std::_Any_data const&, doris::RuntimeState*&&, doris::vectorized::Block*&&, bool*&&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:293
11# doris::vectorized::AggregationNode::get_next(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/vec/exec/vaggregation_node.cpp:508
12# doris::ExecNode::get_next_after_projects(doris::RuntimeState*, doris::vectorized::Block*, bool*) at /root/doris/be/src/exec/exec_node.cpp:852
13# doris::PlanFragmentExecutor::get_vectorized_internal(doris::vectorized::Block**) at /root/doris/be/src/runtime/plan_fragment_executor.cpp:352
14# doris::PlanFragmentExecutor::open_vectorized_internal() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:300
15# doris::PlanFragmentExecutor::open() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:253
16# doris::FragmentExecState::execute() at /root/doris/be/src/runtime/fragment_mgr.cpp:251
17# doris::FragmentMgr::_exec_actual(std::shared_ptr<doris::FragmentExecState>, std::function<void (doris::PlanFragmentExecutor*)>) at /root/doris/be/src/runtime/fragment_mgr.cpp:498
18# std::_Function_handler<void (), doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::PlanFragmentExecutor*)>)::{lambda()#1}>::_M_invoke(std::_Any_data const&) at /var/local/ldb-toolchain/include/c++/11/bits/std_function.h:291
19# doris::ThreadPool::dispatch_thread() at /root/doris/be/src/util/threadpool.cpp:542
20# doris::Thread::supervise_thread(void*) at /root/doris/be/src/util/thread.cpp:455
21# start_thread in /lib64/libpthread.so.0
22# clone in /lib64/libc.so.6
2023-03-21 09:00:06 +08:00
7b93c17364
[Bug][Fix] regexp function core dump DCHECK failed and error result ( #17953 )
...
CREATE TABLE `test` (
`name` varchar(64) NULL,
`age` int(11) NULL
) ENGINE=OLAP
DUPLICATE KEY(`name`)
COMMENT 'OLAP'
DISTRIBUTED BY HASH(`name`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2",
"disable_auto_compaction" = "false"
);
insert into `test` values ("lemon",1),("tom",2);
select a.name regexp concat('^', a.name) from test a;
2023-03-21 08:56:19 +08:00
9021d2ae11
[fix](fragment mgr) fix the bug that g_fragmentmgr_prepare_latency update uncorrectly #17909
2023-03-21 08:50:38 +08:00
dc284b62d9
[vectorized](function) support array_filter function ( #17832 )
2023-03-20 23:18:10 +08:00
3661ca4510
[Bug](regression-test) Fix the collect_set/list size limit result failed ( #17963 )
...
fix regression test failed in fuzzy mode:collect_set
2023-03-20 21:00:41 +08:00
a92115f709
[Bug](materialized-view) fix select mv rollback fail on left join ( #17850 )
...
fix select mv rollback fail on left join
2023-03-20 19:14:17 +08:00
b4634342aa
[bug](txn) fix concurrent txns's status data race when aborting txn ( #17893 )
2023-03-20 17:55:03 +08:00
bd8e3e6405
[refactor](date) unify DateTimeValue and VecDateTimeValue ( #17670 )
2023-03-20 16:27:08 +08:00
378789ba8a
[Fix](parquet-reader) Fix dict_filter crashed caused by VDirectInPredicate checking expr result is not nullable. ( #17924 )
...
Be crashed in parquet dict_filter function caused by VDirectInPredicate checking expr result is not nullable.
2023-03-20 00:02:59 +08:00
1d26b4d6c2
[improvement](predicate) Cache the dict code in ComparisonPredicate ( #17684 )
2023-03-19 17:37:28 +08:00
dd53bc1c8d
[unify type system](remove unused type desc) remove some code ( #17921 )
...
There are many type definitions in BE. Should unify the type system and simplify the development.
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-03-19 14:05:02 +08:00
a993ac91d4
[bugfix](jsonb memory leak) there are memory leak in jsonb field ( #17922 )
...
* [bugfix](jsonb memory leak) there are memory leak in jsonb field
---------
Co-authored-by: yiguolei <yiguolei@gmail.com >
2023-03-19 14:04:14 +08:00
e359e412e1
[vectorized](udaf) fix java udaf meet error of std::bad_alloc ( #17848 )
...
Now if the user code of java udaf throws exception, because c++ code of agg function nobody could deal
with it, so maybe get error of std::bad_alloc
2023-03-19 11:52:15 +08:00
dfa2528b5e
[fix](bitmap) fix wrong result of bitmap count functions for null values ( #17849 )
...
bitmap count functions result is null when there are null values, which is not right:
2023-03-19 11:49:58 +08:00
e7e13bc338
[optimize](array function) array_apply fucntion vectorized compute column_filter loop ( #17687 )
2023-03-19 10:18:09 +08:00
d79da2f926
[Fix](parquet-reader) Fix dict filter not enabled. ( #17882 )
2023-03-18 22:16:37 +08:00
5c5dcfda78
Revert "[enhancement](memory) PODArray replaces MemPool in PredicateColumn ( #17800 )" ( #17910 )
...
This reverts commit 17d1c1bc7f6cc95eecd224eaa219c976b60fa17e.
2023-03-17 20:50:01 +08:00
46d88ede02
[Refactor](Metadata tvf) Reconstruct Metadata table-value function into a more general framework. ( #17590 )
2023-03-17 19:54:50 +08:00
043f77200f
[Bug](dynamic-table) Fix column alignment logic and support filtering null values when slot is not null ( #17842 )
...
Before this PR when encountering null values with some columns which is specified as `NOT NULL`, null values will not be filtered,thi behavior does not match with the original load behavior.
Second column alignment logic has bug :
```
template <typename ColumnInserterFn>
void align_variant_by_name_and_type(ColumnObject& dst, const ColumnObject& src, size_t row_cnt,
ColumnInserterFn inserter) {
CHECK(dst.is_finalized() && src.is_finalized());
// Use rows() here instead of size(), since size() will check_consistency
// but we could not check_consistency since num_rows will be upgraded even
// if src and dst is empty, we just increase the num_rows of dst and fill
// num_rows of default values when meet new data
size_t num_rows = dst.rows();
```
2023-03-17 16:53:30 +08:00
b95cd7eca2
[Refactor](function) Reconstruct default logic for const args. ( #17830 )
2023-03-17 11:13:13 +08:00
5d3de05976
[feature](map) basic functions for map datatype ( #16916 )
...
basic functions for map datatype:
- MAP<K, V> map(K k1, V v1, ...)
- BIGINT map_size(MAP<K, V> m)
- BOOL map_contains_key(MAP<K, V> m, K k1)
- BOOL map_contains_value(MAP<K, V> m, V v1)
- ARRAY< K> map_keys(MAP<K, V> m)
- ARRAY< V> map_values(MAP<K, V> m)
2023-03-17 10:28:17 +08:00
b4b126b817
[Feature](parquet-reader) Implements dict filter functionality parquet reader. ( #17594 )
...
Implements dict filter functionality parquet reader to improve performance.
2023-03-16 20:29:27 +08:00
c29582bd57
[pipeline](split by segment)support segment split by scanner ( #17738 )
...
* support segment split by scanner
* change code by cr
2023-03-16 15:25:52 +08:00
ea943415a0
[bugfix](compaction) remove useless check ( #17804 )
...
transient size may not equal to candidate_rowset size.
For example, one rowset has many segment, but size is smaller then
promotion size, this rowset will break pick rowset loop cause compaction
score is enough but will be filtered in level_size check, this will make
transient size not equal to candidate size.
2023-03-16 15:23:49 +08:00
caed2155f5
[test](fix) use vertorized interface in test ( #17649 )
2023-03-16 15:23:07 +08:00
ee7226348d
[FIX](Map) fix map compaction error ( #17795 )
...
When compaction case, memory map offsets coming to same olap convertor which is from 0 to 0+size
but it should be continue in different pages when in one segment writer .
eg :
last block with map offset : [3, 6, 8, ... 100]
this block with map offset : [5, 10, 15 ..., 100]
the same convertor should record last offset to make later coming offset followed last offset.
so after convertor :
the current offset should [105, 110, 115, ... 200], then column writer just call append_data() to make the right offset data append pages
2023-03-16 13:54:01 +08:00
17d1c1bc7f
[enhancement](memory) PODArray replaces MemPool in PredicateColumn ( #17800 )
...
MemPool is about to be removed, replaced by Arena and PODArray.
2023-03-16 09:01:28 +08:00
