Commit Graph

2544 Commits

Author SHA1 Message Date
d10a708fa2 [improve](jdbc catalog) add profile for jdbc scan (#27447) 2023-11-27 10:33:39 +08:00
baadc14e60 [Enhancement](function) support unix_timestamp with float (#26827)
---------

Co-authored-by: YangWithU <plzw8@outlook.com>
2023-11-27 09:58:53 +08:00
b6dd43bb9d [fix](load) return error instead of panic when slot and expr do not (#27423) 2023-11-26 18:53:37 +08:00
148f365da8 [fix](move-memtable) handle tablet location not found (#27561) 2023-11-26 18:42:15 +08:00
3d2b194985 [refactor](if_function) handle result status of function (#27491) 2023-11-25 10:35:55 +08:00
553e4a8903 [feature-wip](merge-on-write) MOW table support different primary keys and sort keys (#24788) 2023-11-24 16:37:30 +08:00
dfe3a2dd01 [feature](mtmv)(3)Implementing multi table materialized views (#26146)
Introduction to Main Classes:
- MTMVService:MTMV services for other modules to call
- MTMVHookService:All operations that affect the MTMV
  - MTMVJobManager:All operations that affect the MTMV job
  - MTMVCacheManager:All operations that affect the MTMV Cache
- MTMVTask&MTMVJob:Inherit from job framework
2023-11-24 12:34:38 +08:00
75c9f00137 [Bug](bitmap) Fix heap-use-after-free in the bitmap functions (#27411) 2023-11-24 10:09:31 +08:00
b580ee91ce [fix](compile) fix macOS compile and format code (#27494) 2023-11-23 23:24:10 +08:00
dd65cc1d14 [opt](MergedIO) no need to merge large columns (#27315)
1. Fix a profile bug of `MergeRangeFileReader`, and add a profile `ApplyBytes` to show the total bytes  of ranges.
2. There's no need to merge large columns, because `MergeRangeFileReader` will increase the copy time.
2023-11-23 19:15:47 +08:00
2ea33518b0 [Opt](load) use batching to optimize auto partition (#26915)
use batching to optimize auto partition
2023-11-23 19:12:28 +08:00
b457856bd2 [chore](be) remove bthread scanner related codes (#27417) 2023-11-23 15:18:49 +08:00
699798eaa7 [fix](function) make TIMESTAMP function DEPEND_ON_ARGUMENT (#27343)
* fix

* fix nullable

* remove null

* add case
2023-11-23 14:26:19 +08:00
7a75f8c380 [improve](move-memtable) set brpc streaming params in config (#27442) 2023-11-23 14:14:43 +08:00
b79f5d77f1 [improve](move-memtable) improve logging messages (#27443) 2023-11-23 11:46:29 +08:00
Pxl
301bfe4d5d [Bug](mark-join) fix mark join report error when probe block have column do not output (#27360)
fix mark join report error when probe block have column do not output
2023-11-23 11:16:02 +08:00
1b3512d942 [pipelineX](bug) Fix cancel timeout (#27396) 2023-11-22 22:31:34 +08:00
a68b62b421 [refactor](datatype) Introduce a toolkits for clarify datelike types usage (#26810) 2023-11-22 22:08:13 +08:00
5442e8d1fc [pipelineX](dependency) split different dependencies (#27366) 2023-11-22 12:50:39 +08:00
604842526b [improvement](expr) improve if expr performance (#27296) 2023-11-22 12:48:06 +08:00
Pxl
b541de7a03 do not push down agg on aggregate column (#27356)
do not push down agg on aggregate column
2023-11-22 10:53:29 +08:00
4fbcad9c7c [minor](show_export) make result of file url usable (#27209)
* [minor](show_export) make result of file url usable

* update regression-test

* update regression-test
2023-11-22 10:14:45 +08:00
1ebb54afdc [fix](null equal) fix coredump of pushing eq_for_null (#27341) 2023-11-21 18:36:33 +08:00
016dccb4b7 [fix](null_equal) fix wrong result and coredump of operator <=> (#27312)
* [fix](null_equal) fix wrong result and coredump of operator <=>

* fix
2023-11-21 14:32:03 +08:00
c0f22e8feb [FIX](complextype)fix struct nested complex collection type and and regresstest (#26973) 2023-11-20 22:29:12 +08:00
6ed0be8e3c [refactor](profilev2) unify the counter name in shuffle operator and normal operator (#27267)
using blocksproduced and rowsproduced to unify the counter name in DataStreamSender and other exec node, or exchange operator and other operators.
blocks produced and rows produced are more easy to understand.
---------

Co-authored-by: yiguolei <yiguolei@gmail.com>
2023-11-20 14:21:39 +08:00
459f75073f [pipelineX](dependency) remove OrDependency (#27242) 2023-11-20 13:05:34 +08:00
febd60c75f [fix](join) incorrect result of left join with other conjuncts (#27238) 2023-11-19 15:36:39 +08:00
836cda65d8 [refactor](profilev2) split merged profile to a single runtime profile to make the logic more clear (#27184) 2023-11-19 13:21:50 +08:00
b42828cf69 [fix](window_function) min/max/sum/avg should be always nullable (#27104)
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
2023-11-18 18:41:42 +08:00
b1eef30b49 [pipelineX](dependency) Wake up task by dependencies (#26879)
---------

Co-authored-by: Mryange <2319153948@qq.com>
2023-11-18 03:20:24 +08:00
0aec436ef8 [chore](be) format reader parameter settings (#22964) 2023-11-18 00:11:46 +08:00
5fb27eb652 [fix](compile) fix BE compile failure on Mac (#27206) 2023-11-17 23:52:51 +08:00
5d548935e0 [improvement](insert) support schema change and decommission for group commit (#26359) 2023-11-17 21:41:38 +08:00
c459408580 [fix](jni) avoid BE crash and NPE when close paimon reader (#27129)
1. Do not use FATAL log when jni encounter error, to avoid crash.
2. Fix NPE when closing PaimonReader, the reader may not be assigned if PaimonReader open failed.
2023-11-17 20:01:08 +08:00
52995c528e [fix](iceberg) iceberg use customer method to encode special characters of field name (#27108)
Fix two bugs:
1. Missing column is case sensitive, change the column name to lower case in FE for hive/iceberg/hudi
2. Iceberg use custom method to encode special characters in column name. Decode the column name to match the right column in parquet reader.
2023-11-17 18:38:55 +08:00
593e3662b0 [Fix](match) fix match null for no index (#26983)
This pull request addresses an issue observed with inverted index tables or tables without indices when querying null values using the MATCH function. 
Previously, executing a query like `SELECT * FROM table WHERE column MATCH null;` would yield incorrect results. 

The update introduces enhanced handling of nullable columns within the MATCH function, ensuring accurate query results when null values are involved.
2023-11-17 15:57:50 +08:00
a0661ed9d2 [Fix](multi-catalog) Fix complex type crash when using dict filter facility in the parquet-reader. (#27151)
- Fix complex type crash when using the dict filter facility in the parquet-reader by turning off the dict filter facility in this case.
- Add orc complex types regression test.
2023-11-17 13:43:58 +08:00
91af86bc78 [fix](function) fix error when use negative number in explode_numbers #27020 2023-11-17 12:02:14 +08:00
334260dff7 [feature](function) support ip function ipv4stringtonum(ordefault, ornull), inet_aton (#25510) 2023-11-17 10:27:07 +08:00
a4d78682ff [Optimize](point query) clear names to reduce mem consumption and cpu cost related to block column name (#26931) 2023-11-17 10:18:21 +08:00
afffcfd14c [fix](load) skip cancel already cancelled channels (#27111) 2023-11-16 18:38:40 +08:00
e29d8cb110 [feature](move-memtable) support pipelineX in sink v2 (#27067) 2023-11-16 15:00:55 +08:00
3ad865fef9 [refactor](storage) Expressing the types of computation layer and storage layer in PrimitiveTypeTraits (#26191) 2023-11-15 21:34:49 +08:00
035e593b26 remove useless hash function (#26955) 2023-11-15 20:37:21 +08:00
83edcdead9 [enhancement](random_sink) change tablet search algorithm from random to round-robin for random distribution table (#26611)
1. fix race condition problem when get tablet load index
2. change tablet search algorithm from random to round-robin for random distribution table when load_to_single_tablet set to false
2023-11-15 19:55:31 +08:00
0491437a86 [Opt](scanner-scheduler) Optimize BlockingQueue, BlockingPriorityQueue and change remote scan thread pool. (#26784)
## Proposed changes
- Optimize `BlockingQueue`, `BlockingPriorityQueue` by swapping `notify` and `unlock` to reduce lock competition. Ref: https://www.boost.org/doc/libs/1_54_0/boost/thread/sync_bounded_queue.hpp
- Change remote scan thread pool to `PriorityQueue`.

### Test result
Before:
```
mysql> select  sum(lo_partkey)  from  lineorder;
+-----------------+
| sum(lo_partkey) |
+-----------------+
| 300021444265405 |
+-----------------+
1 row in set (1.11 sec)
```

After:
```
mysql> select  sum(lo_partkey)  from  lineorder;
+-----------------+
| sum(lo_partkey) |
+-----------------+
| 300021444265405 |
+-----------------+
1 row in set (0.80 sec)
```
2023-11-15 18:24:36 +08:00
00896d8954 [fix](agg) fix coredump of multi distinct of decimal128I (#27014)
* [fix](agg) fix coredump of multi distinct of decimal128

* fix
2023-11-15 17:37:20 +08:00
6183b298e1 [refactor](data_type) remove some unused functions (#26966) 2023-11-15 09:23:53 +08:00
cdef768629 [fix](sink) crash caused by wild pointer of counter in VDataStreamSender (#26947)
If preparation fails, the counter _peak_memory_usage_counter will be a wild pointer.

*** SIGSEGV address not mapped to object (@0x454d49545f) received by PID 16992 (TID 18856 OR 0x7f4d05444700) from PID 1296651359; stack trace: ***
 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at /root/doris/be/src/common/signal_handler.h:417
 1# os::Linux::chained_handler(int, siginfo*, void*) in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so
 2# JVM_handle_linux_signal in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so
 3# signalHandler(int, siginfo*, void*) in /app/doris/Nexchip-doris-1.2.4.2-bin-x86_64/java8/jre/lib/amd64/server/libjvm.so
 4# 0x00007F55C85B9400 in /lib64/libc.so.6
 5# doris::vectorized::VDataStreamSender::close(doris::RuntimeState*, doris::Status) at /root/doris/be/src/vec/sink/vdata_stream_sender.cpp:734
 6# doris::PlanFragmentExecutor::close() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:543
 7# doris::PlanFragmentExecutor::~PlanFragmentExecutor() at /root/doris/be/src/runtime/plan_fragment_executor.cpp:95
 8# doris::FragmentExecState::~FragmentExecState() at /root/doris/be/src/runtime/fragment_mgr.cpp:112
 9# std::_Sp_counted_ptr<doris::FragmentExecState*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() at /root/ldb/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:348
10# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&, std::function<void (doris::RuntimeState*, doris::Status*)> const&) at /root/doris/be/src/runtime/fragment_mgr.cpp:855
11# doris::FragmentMgr::exec_plan_fragment(doris::TExecPlanFragmentParams const&) at /root/doris/be/src/runtime/fragment_mgr.cpp:592
12# doris::PInternalServiceImpl::_exec_plan_fragment_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::PFragmentRequestVersion, bool) at /root/doris/be/src/service/internal_service.cpp:463
13# doris::PInternalServiceImpl::_exec_plan_fragment_in_pthread(google::protobuf::RpcController*, doris::PExecPlanFragmentRequest const*, doris::PExecPlanFragmentResult*, google::protobuf::Closure*) at /root/doris/be/src/service/internal_service.cpp:305
14# doris::WorkThreadPool<false>::work_thread(int) at /root/doris/be/src/util/work_thread_pool.hpp:160
15# execute_native_thread_routine at ../../../../../libstdc++-v3/src/c++11/thread.cc:84
16# start_thread in /lib64/libpthread.so.0
17# clone in /lib64/libc.so.6
2023-11-14 19:05:49 +08:00