913282b29b
[refactor](column) remove get_data_type in IColumn ( #25242 )
2023-10-10 20:27:15 +08:00
62a6b132be
[Fix](func numbers) Remove backend_nums argument of numbers function ( #25200 )
2023-10-10 20:25:58 +08:00
fc1bad9a6b
[feature](Nereids) support query MATERIALIZED_VIEW type table ( #25227 )
2023-10-10 06:44:29 -05:00
67ddfb1abc
[fix](httpserver) creating this cookie without the "secure" flag and enabling cross-origin resource safe ( #25107 )
2023-10-10 06:25:09 -05:00
8b56ca84c7
[fix](Nereids) support AnyDataType in function signature ( #25173 )
...
1. support AnyDataType in function signature
2. update histogram signature
2023-10-10 06:09:47 -05:00
ba1edcf2dc
[fix](stack trace) Optimize stack trace output ( #24933 )
...
Status prints the stack trace, the first four frame pointers are removed, it doesn't make sense.
Optimize stack trace field order.
example:
0# doris::PlanFragmentExecutor::cancel(doris::PPlanFragmentCancelReason const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at doris/core/be/src/common/status.h:0
1# doris::FragmentMgr::cancel_query_unlocked(doris::TUniqueId const&, doris::PPlanFragmentCancelReason const&, std::unique_lock<std::mutex> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at doris/cor
e/be/src/runtime/fragment_mgr.cpp:984
2# doris::FragmentMgr::cancel_query(doris::TUniqueId const&, doris::PPlanFragmentCancelReason const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) at doris/ldb_toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/
../../../../include/x86_64-linux-gnu/c++/11/bits/gthr-default.h:778
3# long doris::MemTrackerLimiter::free_top_memory_query<doris::MemTrackerLimiter::TrackerLimiterGroup>(long, doris::MemTrackerLimiter::Type, std::vector<doris::MemTrackerLimiter::TrackerLimiterGroup, std::allocator<doris::MemTrackerLimiter::TrackerLimiterGroup> >&, s
td::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)> const&, doris::RuntimeProfile*) at doris/ldb_toolchain/bin/../
lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:187
4# doris::MemTrackerLimiter::free_top_memory_query(long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, doris::RuntimeProfile*, doris::Mem
TrackerLimiter::Type) at doris/core/be/src/runtime/memory/mem_tracker_limiter.cpp:362
5# doris::MemInfo::process_full_gc() at doris/core/be/src/util/mem_info.cpp:198
6# doris::Daemon::memory_gc_thread() at doris/core/be/src/common/daemon.cpp:0
7# doris::Thread::supervise_thread(void*) at doris/ldb_toolchain/bin/../usr/include/pthread.h:562
8# start_thread
9# __clone
2023-10-10 18:23:07 +08:00
0435b286fb
[feature](Nereids) support metadata tvf and fix bugs in group_commit() ( #25224 )
...
metadata tvf list:
- backends
- catalogs
- frontends
- frontends_disks
- group_commit
- iceberg_meta
- workload_groups
fix group_commit bugs
- throw NPE when properties do not contain 'table_id'
- throw NPE when table_id's table do not exist
- throw class Cast failed when table_id's table's type is not OLAP
2023-10-10 05:20:19 -05:00
691889419f
[fix](dbt) fix dbt doris bug that appeared in annotated models and table column comment bug ( #24986 )
2023-10-10 05:10:46 -05:00
643f7cad0e
[typo](docs) Delete wrong schema change memory parameters ( #25234 )
2023-10-10 04:49:40 -05:00
5f95e97c56
[fix](function) array distance should return null when result is nan ( #25214 )
2023-10-10 04:41:51 -05:00
d702bc3c13
[typo](doc) hot and cold stratification increases FAQ ( #24974 )
2023-10-10 17:38:43 +08:00
6ca0f3fa5f
[Bug](writer) Fix ub in async writer ( #25218 )
2023-10-10 16:00:45 +08:00
7276665f1e
[enhancement](Nereids) avoiding broadcast join heuristically and pruning more in CostAndEnforceJob ( #25137 )
...
When the rowCount exceeds a certain threshold, refrain from generating a broadcast join.
Only enforce the best expression in CostAndEnforce Job, rather than enforcing every expression.
Remove lower bound group pruning
2023-10-10 13:38:10 +08:00
181c58c691
[fix](Nereids) count_by_enum signature is wrong ( #25167 )
2023-10-10 13:05:20 +08:00
771b8b5bec
[fix](case) Update analyze_stats.groovy ( #25146 )
2023-10-10 12:51:29 +08:00
7434f80300
[pipelineX](refactor) Refactor pending finish dependency ( #25181 )
2023-10-10 11:56:02 +08:00
880d0d7e70
[Bug](pipeline) Support the auto partition in pipeline load ( #25176 )
2023-10-10 11:51:12 +08:00
6ad22721cb
fix: ubsan compile bug ( #25199 )
2023-10-10 11:46:33 +08:00
39669c6df2
[feature](pipelineX) add runtimefliter in pipelineX multicast sink ( #25120 )
2023-10-10 10:41:08 +08:00
59dee6b235
[fix](Nereids) support string cast to complex type ( #25154 )
2023-10-10 10:26:33 +08:00
3a29bb4bc5
[fix](doc) spelling error for colocate join #25053 ( #25202 )
...
Issue: 25053
Translation text not cleaned up
2023-10-10 10:10:55 +08:00
f5b826b66d
[fix](mark join) mark join column should be nullable ( #24910 )
2023-10-10 10:10:36 +08:00
e2be5fafa9
[case](regresstest) update query for parquet/orc with array/map nested type and insert into ( #24746 )
2023-10-10 10:07:22 +08:00
b8621364d2
[FIX](serde)fix scale with decimalv2 in mysql writer which get real scale #25190
2023-10-10 09:09:57 +08:00
90ad48cdb7
[feature](pipelineX) add node id and profilev2 in pipelineX ( #25084 )
2023-10-10 09:09:26 +08:00
b58010c48e
[fix](export) BufferWritable must be committed before deconstruct ( #25185 )
...
F20231009 16:03:47.659968 3342535 string_buffer.hpp:48] Check failed: _now_offset == 0
*** Check failure stack trace: ***
@ 0x561a6f8e21e6 google::LogMessage::SendToLog()
@ 0x561a6f8de7b0 google::LogMessage::Flush()
@ 0x561a6f8e2a29 google::LogMessageFatal::~LogMessageFatal()
@ 0x561a4a409233 doris::vectorized::BufferWritable::~BufferWritable()
@ 0x561a6e202853 doris::vectorized::VCSVTransformer::write()
@ 0x561a6e1f19ba doris::vectorized::VFileResultWriter::_write_file()
@ 0x561a6e1f1522 doris::vectorized::VFileResultWriter::append_block()
@ 0x561a6e121bed
The error will occur in DEBUG mode, and doing export will invalid data.
It has been covered by baidu case.
2023-10-09 22:39:45 +08:00
5e8aef4756
[feature](Nereids) fold weeks_sub/add on fe ( #25155 )
...
support folding weeks_sub/add on fe
2023-10-09 21:52:44 +08:00
53b46b7e6c
[FIX](filter) update for filter_by_select logic ( #25007 )
...
this pr is aim to update for filter_by_select logic and change delete limit
only support scala type in delete statement where condition
only support column nullable and predict column support filter_by_select logic, because we can not push down non-scala type to storage layer to pack in predict column but do filter logic
2023-10-09 21:27:40 +08:00
37247ac449
[opt](Nereids) add two args signature to trim family functions ( #25169 )
2023-10-09 07:17:52 -05:00
08e7a7b932
[feat](optimizer) Scale sample stats with ratio to make it more precise ( #25079 )
...
Since Doris support query specific tablet only, so we don't depend on tableSample to do sample, instead use grammar: TABLET(id) to do so. In OlapAnalyzeTask, we calculate which tablets would be hit and set theirs id in it, so we could get how many rows actually queried and furthur we could get the scale up ratio here
2023-10-09 07:01:59 -05:00
400b9f2f97
[Enhancement](log) Improve Safety and Robustness of Log4j Configuration ( #24861 )
2023-10-09 06:44:37 -05:00
f8eb36158a
[fix](Nereids) alias function support arithmetic functions ( #25162 )
2023-10-09 19:04:47 +08:00
977d119545
[fix](Insert select tvf) fix NPE because tvf do not have catalog name ( #25149 )
2023-10-09 18:02:43 +08:00
d02ef36631
[opt](Nereids) match predicate support array as first arg ( #25172 )
2023-10-09 04:17:27 -05:00
4de3df6a46
[refactor](column) remove unused method and column definitions ( #25152 )
...
remove unused method and column definitions
using primitive type in predicate column to check datev1 and datev2
2023-10-09 17:14:35 +08:00
263631e983
[improvement](meta) Infer the column name when create view if the column is expression ( #24990 )
...
## Proposed changes
Infer the column name when create view if the column is expression
## Further comments
expr column name infer strategy as following:
| expr | example | column name(before) | Inferred column name(if position is 2) |
| ------------- | --------------------------------------- | ------------------------------ | -------------------------------------- |
| function | dayofyear() | dayofyear() | __dayofyear_1 |
| cast | cast(1 as bigint) | CAST(1 AS BIGINT) | __cast_1 |
| anylyticExpr | min() | min() | __min_1 |
| predicate | 1 in (1,2,3,4) | 1 IN (1, 2, 3, 4) | __in_predicate_1 |
| literal | 1 or 'string_var_name' | 1 or 'string_var_name' | __literal_1 |
| arithmeticExpr | & | ... & ... | __arithmetic_expr_1 |
| identifier | a or b | a or b | a or b |
| case | CASE WHEN remark = 's' THEN 1 ELSE 2 END | CASE WHEN remark = 's' THEN 1 ELSE 2 END | __case_1 |
| window | min(timestamp) OVER (...) | min(timestamp) OVER(...) | __min_1 |
SQL for example:
```sql
CREATE VIEW v1 AS
SELECT
error_code,
1,
'string',
now(),
dayofyear(op_time),
cast (source AS BIGINT),
min(`timestamp`) OVER (
ORDER BY
op_time DESC ROWS BETWEEN UNBOUNDED PRECEDING
AND 1 FOLLOWING
),
1 > 2,
2 + 3,
1 IN (1, 2, 3, 4),
remark LIKE '%like',
CASE WHEN remark = 's' THEN 1 ELSE 2 END,
TRUE | FALSE
FROM
db_test.table_test1
```
the output column name is as following:
```
error_code
__literal_1
__literal_2
__now_3
__dayofyear_4
__cast_expr_5
__min_6
__binary_predicate_7
__arithmetic_expr_8
__in_predicate_9
__like_predicate_10
__case_expr_11
__arithmetic_expr_12
```
2023-10-09 04:14:01 -05:00
79fa1d1640
[enhancement](regression-test) add stream load json case ( #25168 )
2023-10-09 16:40:39 +08:00
af707e5244
[pipelineX](fix) fix external table scan operator ( #25166 )
2023-10-09 16:33:27 +08:00
e1b9854f90
[bugfix](thirdparty) Upgrade aws s3 sdk to prevent mem leak ( #25106 )
...
During the use of the AWS S3 SDK, we found that there is a memory leak. According to the official issue, upgrading the SDK should resolve the issue.
2023-10-09 16:08:50 +08:00
fbbaf929ee
[chore](workflow) 2/2, auto trigger "License Check" and "BE Code Formatter" instead of after approval ( #25101 )
...
Co-authored-by: stephen <hello-stephen@qq.com >
2023-10-09 15:57:30 +08:00
320709b9ff
[opt](Nereids) support like and regexp function ( #25148 )
2023-10-09 02:55:57 -05:00
7ceb029a17
[Fix](statistics)Fix alter column stats data size is always 0 bug ( #24891 )
...
Fix alter column stats data size is always 0 bug.
2023-10-09 15:48:11 +08:00
0bf954ba05
[fix](Nereids) unique table support bitmap column ( #25160 )
2023-10-09 02:39:11 -05:00
4f7fad5498
[fix](Nereids) properties parser should return map ( #25150 )
2023-10-09 02:32:56 -05:00
d7b6fe57df
[Bug](java-udf) fix java-udf memory leak ( #25151 )
2023-10-09 15:10:56 +08:00
cdba4c4775
[fix](Nereids) deep copier generate wrong slot for TVF ( #25156 )
2023-10-09 14:52:36 +08:00
d34ab7accc
[fix](Nereids) bind sink should use full base schema ( #25153 )
2023-10-09 01:40:57 -05:00
1a6c2a3e84
[fix](Nereids) miss BoolLiteral when convert literal to Nereids' one ( #25159 )
2023-10-09 01:39:09 -05:00
78878e2d56
[chore](workflow) 1/2, remove required checks ("License Check", "Clang Formatter") temporally ( #25134 )
...
Co-authored-by: stephen <hello-stephen@qq.com >
2023-10-09 14:35:38 +08:00
490563cfdb
[fix](Nereids) support empty array and map literal ( #25145 )
2023-10-09 01:23:51 -05:00
