5bf77c8f74
[Fix] (schema change) forbid adding time type column ( #28751 )
2023-12-21 10:27:50 +08:00
e4a58b7407
[Feature](materialized-view) support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg ( #28747 )
...
support match logicalAggregate(logicalProject(logicalFilter(logicalOlapScan())) without agg
2023-12-21 10:27:33 +08:00
007f152e5e
[Improve](compile) add __AVX2__ macro for JsonbParser ( #28754 )
...
* [Improve](compile) add `__AVX2__` macro for JsonbParser
* throw exception instead of CHECK
2023-12-21 10:25:26 +08:00
8759bce426
[fix](stmt):fix CreateTableStmt toSql placed comment in wrong place ( #27504 )
...
Issue Number: close #27474
Co-authored-by: tongyang.han <tongyang.han@jiduauto.com >
2023-12-21 09:57:20 +08:00
a443a39e2c
[enhance](blacklist) seperate blacklist conf from heartbeat ( #28638 )
...
There is a circuit breaker lasting for 2 minutes in grpc, then if a be is down and up again, send fragments to the be fails lasting for 2 minutes.
2023-12-21 00:17:45 +08:00
18ad8562f2
[refactor](broadcastbuffer) using a queue to remove ref and unref codes ( #28698 )
...
Co-authored-by: yiguolei <yiguolei@gmail.com >Add a new class broadcastbufferholderqueue to manage holders
Using shared ptr to manage holders, not use ref and unref, it is too difficult to maintain.
2023-12-20 21:23:25 +08:00
a8dcca98ec
[FIX](explode)fix explode array decimal ( #28744 )
...
* fix explode with array<decimal> has specific precision at old planner
2023-12-20 20:19:56 +08:00
280a01b815
[pipelineX](improvement) Support global runtime filter ( #28692 )
2023-12-20 20:06:26 +08:00
504693be7f
[bug](coredump) Fix coredump in aggregation node's destruction( #28684 )
...
fix coredump in aggregation node's destruction
2023-12-20 20:02:48 +08:00
36857006cd
[Fix](json reader) fix json reader crash due to fmt::format_to ( #28737 )
...
```
4# __gnu_cxx::__verbose_terminate_handler() [clone .cold] at ../../../../libstdc++-v3/libsupc++/vterminate.cc:75
5# __cxxabiv1::__terminate(void (*)()) at ../../../../libstdc++-v3/libsupc++/eh_terminate.cc:48
6# 0x00005622F33D22B1 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
7# 0x00005622F33D2404 in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
8# fmt::v7::detail::error_handler::on_error(char const*) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
9# char const* fmt::v7::detail::parse_replacement_field<char, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&>(char const*, char const*, fmt::v7::detail::format_handler<fmt::v7::detail::buffer_appender<char>, char, fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<char>, char> >&) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
10# void fmt::v7::detail::vformat_to<char>(fmt::v7::detail::buffer<char>&, fmt::v7::basic_string_view<char>, fmt::v7::basic_format_args<fmt::v7::basic_format_context<fmt::v7::detail::buffer_appender<fmt::v7::type_identity<char>::type>, fmt::v7::type_identity<char>::type> >, fmt::v7::detail::locale_ref) in /mnt/ssd01/pipline/OpenSourceDoris/clusterEnv/P0/Cluster0/be/lib/doris_be
11# doris::vectorized::NewJsonReader::_append_error_msg(rapidjson::GenericValue<rapidjson::UTF8<char>, rapidjson::MemoryPoolAllocator<rapidjson::CrtAllocator> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool*) at /root/doris/be/src/vec/exec/format/json/new_json_reader.cpp:924
12# doris::vectorized::NewJsonReader::_set_column_value
```
2023-12-20 19:58:30 +08:00
ea646c6682
[fix](mtmv)fix thread local reference to checkpoint's Env, causing Env to be unable to be reclaimed, resulting in excessive memory usage by FE ( #28723 )
...
when replay addTaskResult log,will create one ConnectContext,and set Env.getCurrentEnv,then store this ctx in ConnectContext.threadLocalInfo,threadLocalInfo is static,so this ctx can not be recycling，Env of replay thread also can not be recycling
2023-12-20 19:46:58 +08:00
7b96730e87
[fix](block) fix nullptr in MutableBlock::allocated_bytes ( #28738 )
2023-12-20 19:46:13 +08:00
49eed98c1e
[fix](tvf)Fixed the avro-scanner projection pushdown failing to query on multiple BEs ( #28709 )
2023-12-20 19:39:26 +08:00
40afee0097
[fix](Nereids) delete partition failed ( #28717 )
...
1. parser's partitionSpec changed unexpectly by PR #26492
2. delete without using should support un-equals expression
2023-12-20 19:31:53 +08:00
e8d0569d8b
[refine](pipelineX)Make the 'set ready' logic of SenderQueue in pipelineX the same as that in the pipeline ( #28488 )
2023-12-20 19:26:00 +08:00
d1cc6d9a85
[test](regression-test) order by decs should only make effect on its nearest column #28728
2023-12-20 19:06:41 +08:00
9a5ec43f05
[fix](nereids) Fix data wrong using mv rewrite and ignore case when getting mv related partition table ( #28699 )
...
1. Fix data wrong using mv rewrite
2. Ignore case when getting mv related partition table
3. Enable infer expression column name without alias when create mv
2023-12-20 17:59:46 +08:00
2b2d3d0eb1
[fix](meta_scanner) fix meta_scanner process ColumnNullable ( #28711 )
2023-12-20 17:41:38 +08:00
2683b1e8c3
[test](Nereids): add test for scalar agg ( #28712 )
2023-12-20 17:35:04 +08:00
b631c51e6a
[FIX](regresstest) fix case with double value #28668
...
double value in case has 27 figures after the point, which will make not standable output
2023-12-20 17:26:28 +08:00
08066a55f1
[Enhancement](auto-partition) change the behaviour when insert overwrite an auto partition table #28683
...
If we specific target partition(s) when inserting overwrite an auto partition table,
before:
could create new partition
now:
behalf just like non-auto partition table
2023-12-20 17:19:21 +08:00
552a5c1632
[fix](mtmv) fix insert overwrite getExecTimeout error ( #28700 )
...
should use InsertTimeout but QueryTimeout
2023-12-20 17:08:38 +08:00
9d5b9cc452
[fix](statistics)Fix drop stats fail silently bug. ( #28635 )
...
Drop stats use IN predicate to filter the column stats to delete. The default length of IN predicate is 1024, drop table stats with more than 1024 columns may fail.
This pr is to split the delete sql based on the IN predicate length.
2023-12-20 15:41:25 +08:00
2a81d238f5
[test](Nereids): remove shape plan project and distribute in eager test ( #28701 )
2023-12-20 14:41:49 +08:00
0c9c32c52d
[Feature](datatype) update be ut codes and fix bugs for IPv4/v6 ( #28670 )
2023-12-20 14:38:46 +08:00
bcc32b5b26
[feature](invert index) match_regexp feature added ( #28257 )
2023-12-20 14:30:35 +08:00
32964879f7
[opt](inverted index) Add null document interface to optimize empty string indexing ( #28661 )
2023-12-20 12:11:02 +08:00
afd55123ea
[fix](partial update) report error directly if missing rowsets during doing alignment when flushing memtable due to compaction ( #28677 )
2023-12-20 11:49:11 +08:00
fe184e322a
[code](pipelineX) refine some pipelineX code ( #28570 )
2023-12-20 11:45:06 +08:00
3e85797443
[fix](Nereids) should only do bind relation in view analyzer ( #28637 )
2023-12-20 10:47:51 +08:00
0ae87e705e
[fix](planner) ctas update datev1 to datev2 should use equals ( #28641 )
2023-12-20 10:31:49 +08:00
23784787b1
[fix](planner)should bind expr using no slot to correct tuple ( #28656 )
2023-12-20 10:29:59 +08:00
c26c0c31a5
[refactor](runtimefilter) do not use QueryContext in runtime filter ( #28559 )
2023-12-20 10:28:55 +08:00
4c0080e237
[feat](Nereids) support outer join and aggregate bitmap rewrite by mv ( #28596 )
...
- Support left outer join rewrite by materialized view
- Support bitmap_union roll up to imp count(distinct)
- Support partition materialized view rewrite
2023-12-20 10:23:30 +08:00
17268346d3
[Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction ( #28062 )" ( #28674 )
...
This reverts commit 485d7db5161efb011b43a10237df1ca83d5e5c3d.
2023-12-20 10:13:43 +08:00
e59c707022
[fix](stream_load)fix bug for stream ( #27752 )
...
1. forbid thed stream_load without content-length or chunked Transfer Encoding
2. forbid thed stream_load both with content-length and chunked Transfer Encoding
Co-authored-by: xingying01 <xingying01@corp.netease.com >
2023-12-20 09:47:52 +08:00
044d7830c9
[improvement](transaction) reduce publish txn log ( #28277 )
2023-12-20 09:31:21 +08:00
81643d78e7
[fix](regression) fix test_alter_colocate_table due to force_olap_table_replication_num=3 ( #28575 )
2023-12-20 09:28:44 +08:00
0c535f93d7
[fix](regression) restore reserve num replicas ( #28541 )
2023-12-20 09:28:24 +08:00
eb50db1f3f
[fix](regression) fix test_set_replica_status due to force_olap_table_replication_num=3 ( #28573 )
2023-12-20 09:27:18 +08:00
e33cf52dba
[fix](test) fix ccr test cases ( #28664 )
2023-12-20 09:26:55 +08:00
9aa878ea1f
[Fix](Job)Fixed job scheduling missing certain time window schedules ( #28659 )
...
Since scheduling itself consumes a certain amount of time, the start time of the time window should not be the current time, but the end time of the last schedule.
2023-12-20 09:21:15 +08:00
fb3b0afb85
[opt](task-assignment) use consistent hash as default task assigner and cache the consistent hash ring ( #28522 )
...
1. Use consistent hash algo as the default assigner for file query scan node
A consistent assignment can better utilize the page cache of BE node.
2. Cache the consistent hash ring
Init a consistent hash ring is time-consuming because there a thousands of virtual node need to be added.
So cache it for better performance
2023-12-19 22:29:35 +08:00
aab859be56
[enhance](partition_id) check partition id before store meta ( #28055 )
2023-12-19 21:31:41 +08:00
b2d16856b4
[Fix](memtable) fix shrink_memtable_by_agg without duplicated keys ( #28660 )
...
remove duplicated logic:
```
vectorized::Block in_block = _input_mutable_block.to_block();
_put_into_output(in_block);
```
`_input_mutable_block.to_block()` will move `_input_mutable_block`, and lead to `flush` with empty block
2023-12-19 20:45:16 +08:00
d2275e73df
[Fix](transactional-hive) Fix hive transactional table return empty result. ( #28518 )
2023-12-19 19:38:48 +08:00
7da86c37ec
[fix](hive) add support for quoteChar and seperatorChar for hive ( #28613 )
...
add support for quoteChar and seperatorChar .
2023-12-19 19:35:03 +08:00
0883d47832
[Enhance](broker) add inputstream expire scheduled checker to avoid memory leak for broker scan ( #28589 )
...
This pr introduces 2 broker conf:
1. enable_input_stream_expire_check: which indicates whether enable inputStream expire check.
2. input_stream_expire_seconds: which indicates the timeout seconds for inputStream since last update.
2023-12-19 19:24:29 +08:00
9c9249e911
[feature](inverted index) add ignore_above property to prevent long s… ( #28585 )
...
When string is too long, clucene will throw an error.
And the string is too long to analyze. So we ignore the string in index process when the string is longer than 256 bytes by default.
We add an poperty `ignore_above` for user to customize.
2023-12-19 18:54:36 +08:00
73a3d84c5e
[chore](user) Add user property parallel_fragment_exec_instance_num ( #28447 )
2023-12-19 18:33:01 +08:00
