48b04752a5
[Chore](cases) set enable_nereids_planner on mv cases ( #25277 )
...
set enable_nereids_planner on mv cases
2023-10-11 15:58:06 +08:00
47578c0fc9
[fix](Nereids) fix toSql of date literal ( #25243 )
...
toSql should return '2023-2-1 ' for DateLiteral 2023-2-1
2023-10-11 13:04:05 +08:00
0d603dd4c3
[Bug](delete) Use date as common type for date comparison ( #25262 )
2023-10-11 11:51:43 +08:00
bb670118f5
[coverage](test) Delete unused function to improve test coverage ( #25233 )
2023-10-11 11:50:51 +08:00
cdf5f0fe68
[fix](pipelineX) mark join column should be nullable ( #25275 )
2023-10-11 11:35:43 +08:00
1e6d34d1d0
[Enhancement](sql-cache) Add partition update time for hms table and use it at sql-cache. ( #24491 )
...
Now FE does not record the update time of hms tbl's partitons, so the sql cache may be hit even the hive table's partitions have changed. This pr add a field to record the partition update time, and use it when enable sql-cache.
The cache will be missed if any partition has changed at hive side.
Use System.currentTimeMillis() but not the event time of hms event because we would better keep the same measurement with the schemaUpdateTime of external table. Add this value to ExternalObjectLog and let slave FEs replay it because it is better to keep the same value with all FEs, so the sql-cache can be hit by the querys through different FEs.
2023-10-11 11:05:16 +08:00
2f706cc84b
[compile](simdjson reader) use __AVX2__ macro to decide whether use simdjson to parse ( #25165 )
2023-10-11 10:50:13 +08:00
b91bce8a62
[feature](Nereids) add array distance functions ( #25196 )
...
- l1_distance
- l2_distance
- cosine_distance
- inner_product
2023-10-10 21:35:06 -05:00
8e66dbc4a8
[enhancement](log) add some decheck log to debug ( #25210 )
2023-10-11 10:33:13 +08:00
5be29f859a
[enhancement](node) add filter in partition sort node in BE #25188
...
add filter in partition sort node in BE
2023-10-11 10:30:15 +08:00
2ed5245014
[FIX](array_function) fix array_map function with array index function without checkout arg… #25226
2023-10-11 10:23:33 +08:00
7b22ae0c80
[pipelineX](feature) Support set operation operator ( #25251 )
...
---------
Co-authored-by: zhaochangle <zhaochangle@selectdb.com >
2023-10-11 10:22:45 +08:00
1fa8720164
[regression-test](merge-on-write) Fix partial update concurrency conflict case ( #25212 )
2023-10-11 10:17:01 +08:00
d4673ce28a
[Feature](Job)Jobs in the Finish state will be automatically deleted after three days. ( #25170 )
2023-10-11 10:04:19 +08:00
b7ac95a970
[enhancement](regression-test) open routine load regression test by default and add data check ( #25122 )
2023-10-11 10:03:16 +08:00
be11b48407
[fix](load) fix MemTableWriter::active_memtable_mem_consumption ( #25207 )
2023-10-10 22:33:50 +08:00
fb3b888ff1
[prune](partition)support prune partition when is auto partition with function call ( #24747 )
...
now create table use auto create partition:
AUTO PARTITION BY RANGE date_trunc(event_day, 'day')
so the value of event_day will be insert into partition of date_trunc(event_day, 'day'),
eg: select * from partition_range where date_trunc(event_day,"day")= "2023-08-07 11:00:00";
we can prune some partitions by invoke function of date_trunc("2023-08-07 11:00:00","day" );
2023-10-10 20:39:43 +08:00
913282b29b
[refactor](column) remove get_data_type in IColumn ( #25242 )
2023-10-10 20:27:15 +08:00
62a6b132be
[Fix](func numbers) Remove backend_nums argument of numbers function ( #25200 )
2023-10-10 20:25:58 +08:00
fc1bad9a6b
[feature](Nereids) support query MATERIALIZED_VIEW type table ( #25227 )
2023-10-10 06:44:29 -05:00
67ddfb1abc
[fix](httpserver) creating this cookie without the "secure" flag and enabling cross-origin resource safe ( #25107 )
2023-10-10 06:25:09 -05:00
8b56ca84c7
[fix](Nereids) support AnyDataType in function signature ( #25173 )
...
1. support AnyDataType in function signature
2. update histogram signature
2023-10-10 06:09:47 -05:00
ba1edcf2dc
[fix](stack trace) Optimize stack trace output ( #24933 )
...
Status prints the stack trace, the first four frame pointers are removed, it doesn't make sense.
Optimize stack trace field order.
example:
0# doris::PlanFragmentExecutor::cancel(doris::PPlanFragmentCancelReason const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at doris/core/be/src/common/status.h:0
1# doris::FragmentMgr::cancel_query_unlocked(doris::TUniqueId const&, doris::PPlanFragmentCancelReason const&, std::unique_lock<std::mutex> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at doris/cor
e/be/src/runtime/fragment_mgr.cpp:984
2# doris::FragmentMgr::cancel_query(doris::TUniqueId const&, doris::PPlanFragmentCancelReason const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at doris/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/
../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:778
3# long doris::MemTrackerLimiter::free_top_memory_query<doris::MemTrackerLimiter::TrackerLimiterGroup>(long, doris::MemTrackerLimiter::Type, std::vector<doris::MemTrackerLimiter::TrackerLimiterGroup, std::allocator<doris::MemTrackerLimiter::TrackerLimiterGroup> >&, s
td::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, doris::RuntimeProfile*) at doris/ldb_toolchain/bin/../
lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
4# doris::MemTrackerLimiter::free_top_memory_query(long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::RuntimeProfile*, doris::Mem
TrackerLimiter::Type) at doris/core/be/src/runtime/memory/mem_tracker_limiter.cpp:362
5# doris::MemInfo::process_full_gc() at doris/core/be/src/util/mem_info.cpp:198
6# doris::Daemon::memory_gc_thread() at doris/core/be/src/common/daemon.cpp:0
7# doris::Thread::supervise_thread(void*) at doris/ldb_toolchain/bin/../usr/include/pthread.h:562
8# start_thread
9# __clone
2023-10-10 18:23:07 +08:00
0435b286fb
[feature](Nereids) support metadata tvf and fix bugs in group_commit() ( #25224 )
...
metadata tvf list:
- backends
- catalogs
- frontends
- frontends_disks
- group_commit
- iceberg_meta
- workload_groups
fix group_commit bugs
- throw NPE when properties do not contain 'table_id'
- throw NPE when table_id's table do not exist
- throw class Cast failed when table_id's table's type is not OLAP
2023-10-10 05:20:19 -05:00
691889419f
[fix](dbt) fix dbt doris bug that appeared in annotated models and table column comment bug ( #24986 )
2023-10-10 05:10:46 -05:00
643f7cad0e
[typo](docs) Delete wrong schema change memory parameters ( #25234 )
2023-10-10 04:49:40 -05:00
5f95e97c56
[fix](function) array distance should return null when result is nan ( #25214 )
2023-10-10 04:41:51 -05:00
d702bc3c13
[typo](doc) hot and cold stratification increases FAQ ( #24974 )
2023-10-10 17:38:43 +08:00
6ca0f3fa5f
[Bug](writer) Fix ub in async writer ( #25218 )
2023-10-10 16:00:45 +08:00
7276665f1e
[enhancement](Nereids) avoiding broadcast join heuristically and pruning more in CostAndEnforceJob ( #25137 )
...
When the rowCount exceeds a certain threshold, refrain from generating a broadcast join.
Only enforce the best expression in CostAndEnforce Job, rather than enforcing every expression.
Remove lower bound group pruning
2023-10-10 13:38:10 +08:00
181c58c691
[fix](Nereids) count_by_enum signature is wrong ( #25167 )
2023-10-10 13:05:20 +08:00
771b8b5bec
[fix](case) Update analyze_stats.groovy ( #25146 )
2023-10-10 12:51:29 +08:00
7434f80300
[pipelineX](refactor) Refactor pending finish dependency ( #25181 )
2023-10-10 11:56:02 +08:00
880d0d7e70
[Bug](pipeline) Support the auto partition in pipeline load ( #25176 )
2023-10-10 11:51:12 +08:00
6ad22721cb
fix: ubsan compile bug ( #25199 )
2023-10-10 11:46:33 +08:00
39669c6df2
[feature](pipelineX) add runtimefliter in pipelineX multicast sink ( #25120 )
2023-10-10 10:41:08 +08:00
59dee6b235
[fix](Nereids) support string cast to complex type ( #25154 )
2023-10-10 10:26:33 +08:00
3a29bb4bc5
[fix](doc) spelling error for colocate join #25053 ( #25202 )
...
Issue: 25053
Translation text not cleaned up
2023-10-10 10:10:55 +08:00
f5b826b66d
[fix](mark join) mark join column should be nullable ( #24910 )
2023-10-10 10:10:36 +08:00
e2be5fafa9
[case](regresstest) update query for parquet/orc with array/map nested type and insert into ( #24746 )
2023-10-10 10:07:22 +08:00
b8621364d2
[FIX](serde)fix scale with decimalv2 in mysql writer which get real scale #25190
2023-10-10 09:09:57 +08:00
90ad48cdb7
[feature](pipelineX) add node id and profilev2 in pipelineX ( #25084 )
2023-10-10 09:09:26 +08:00
b58010c48e
[fix](export) BufferWritable must be committed before deconstruct ( #25185 )
...
F20231009 16:03:47.659968 3342535 string_buffer.hpp:48] Check failed: _now_offset == 0
*** Check failure stack trace: ***
@ 0x561a6f8e21e6 google::LogMessage::SendToLog()
@ 0x561a6f8de7b0 google::LogMessage::Flush()
@ 0x561a6f8e2a29 google::LogMessageFatal::~LogMessageFatal()
@ 0x561a4a409233 doris::vectorized::BufferWritable::~BufferWritable()
@ 0x561a6e202853 doris::vectorized::VCSVTransformer::write()
@ 0x561a6e1f19ba doris::vectorized::VFileResultWriter::_write_file()
@ 0x561a6e1f1522 doris::vectorized::VFileResultWriter::append_block()
@ 0x561a6e121bed
The error will occur in DEBUG mode, and doing export will invalid data.
It has been covered by baidu case.
2023-10-09 22:39:45 +08:00
5e8aef4756
[feature](Nereids) fold weeks_sub/add on fe ( #25155 )
...
support folding weeks_sub/add on fe
2023-10-09 21:52:44 +08:00
53b46b7e6c
[FIX](filter) update for filter_by_select logic ( #25007 )
...
this pr is aim to update for filter_by_select logic and change delete limit
only support scala type in delete statement where condition
only support column nullable and predict column support filter_by_select logic, because we can not push down non-scala type to storage layer to pack in predict column but do filter logic
2023-10-09 21:27:40 +08:00
37247ac449
[opt](Nereids) add two args signature to trim family functions ( #25169 )
2023-10-09 07:17:52 -05:00
08e7a7b932
[feat](optimizer) Scale sample stats with ratio to make it more precise ( #25079 )
...
Since Doris support query specific tablet only, so we don't depend on tableSample to do sample, instead use grammar: TABLET(id) to do so. In OlapAnalyzeTask, we calculate which tablets would be hit and set theirs id in it, so we could get how many rows actually queried and furthur we could get the scale up ratio here
2023-10-09 07:01:59 -05:00
400b9f2f97
[Enhancement](log) Improve Safety and Robustness of Log4j Configuration ( #24861 )
2023-10-09 06:44:37 -05:00
f8eb36158a
[fix](Nereids) alias function support arithmetic functions ( #25162 )
2023-10-09 19:04:47 +08:00
977d119545
[fix](Insert select tvf) fix NPE because tvf do not have catalog name ( #25149 )
2023-10-09 18:02:43 +08:00
