52eeee347f
[opt](compound) Optimize by deleting the compound expr after obtaining the final result ( #28934 )
2023-12-26 14:10:53 +08:00
f7a624de38
[fix](case) Add sync to test case ( #29034 )
2023-12-26 14:08:41 +08:00
c8ed14f11c
[enhance](tablet) Reduce log in tablet meta ( #28719 )
2023-12-26 13:37:30 +08:00
509cfea99a
[feature](Load)(step2)support nereids load job schedule ( #26356 )
...
We will Integrate new load job manager into new job scheduling framework so that the insert into task can be scheduled after the broker load sql is converted to insert into TVF(table value function) sql.
issue: https://github.com/apache/doris/issues/24221
Now support:
1. load data by tvf insert into sql, but just for simple load（columns need to be defined in the table）
2. show load stmt
- job id, label name, job state, time info
- simple progress
3. cancel load from db
4. support that enable new load through Config.enable_nereids_load
5. can replay job after restarting doris
TODO:
- support partition insert job
- support show statistics from BE
- support multiple task and collect task statistic
- support transactional task
- need add ut case
2023-12-26 12:29:05 +08:00
b8fd55b0cf
[feature](Nereids) support table valued function http_stream ( #29004 )
2023-12-26 11:47:04 +08:00
065eb9a72b
[feature](nereids)support partition property in nereids ( #28982 )
2023-12-26 11:19:04 +08:00
92660bb1b2
[chore](config) modify variant_ratio_of_defaults_as_sparse_column from 0.95 to 1 ( #28984 )
...
since sparse column is not stable at present
2023-12-26 10:24:43 +08:00
f30e50676e
[opt](scanner) optimize the number of threads of scanners ( #28640 )
...
1. Remove `doris_max_remote_scanner_thread_pool_thread_num`, use `doris_scanner_thread_pool_thread_num` only.
2. Set the default value `doris_scanner_thread_pool_thread_num` as `std::max(48, CpuInfo::num_cores() * 4)`
2023-12-26 10:24:12 +08:00
75a45484b6
[chore](config) modify tablet_schema_cache_recycle_interval from 24h to 1h ( #28980 )
...
To prevent from too many tablet schema cache in memory and lead to performance issue when hold lock
to erase item
2023-12-26 00:34:58 +08:00
7434de9ed8
[improvement](nereids) Get partition related table disable nullable field and complete agg matched pattern mv rules. ( #28973 )
...
* [improvement] (nereids) Get partition related table disable nullable field and modify regression test, complete agg mv rules.
* make filed not null to create partition mv
2023-12-26 00:29:42 +08:00
2aea47c0a9
[fix](mtmv)add log for resolve pending task ( #28999 )
...
* add lock for resolve pending task
* add lock for resolve pending task
2023-12-26 00:29:28 +08:00
cefae3dc90
[bug](storage) Fix gc rowset bug ( #28979 )
2023-12-26 00:29:03 +08:00
17f3ca7349
[fix](planner)should save original select list item before analyze ( #28187 )
...
* [fix](planner)should save original select list item before analyze
* fix test case
* fix failed case
2023-12-25 23:06:45 +08:00
2411dbe0e6
[doc](multi-catalog)add krb and some ertificates FAQs ( #28858 )
...
add some security docs
2023-12-25 23:05:44 +08:00
137f785698
[fix](parquet_reader) misused bool pointer ( #28986 )
...
Signed-off-by: pengyu <pengyu@selectdb.com >
2023-12-25 22:58:08 +08:00
c2c5df9341
[opt](assert_num_rows) support filter in AssertNumRows operator and fix some explain ( #28935 )
...
* NEED
* Update pipeline x
* fix pipelinex compile
2023-12-25 22:47:23 +08:00
0af9371a96
[fix](hash join) fix column ref DCHECK failure of hash join node block mem reuse ( #28991 )
...
Introduced by #28851 , after evaluating build side expr, some columns in resulting block may be referenced more than once in the same block.
e.g. coalesce(col_a, 'string') if col_a is nullable but actually contains no null values, in this case funcition coalesce will insert a new nullable column which references the original col_a.
2023-12-25 22:19:01 +08:00
8921b313b4
[fix](doc) typo fix in dynamic-partition page ( #28511 )
2023-12-25 21:02:37 +08:00
7081139bdc
[fix](block) fix be core while mutable block merge may cause different row size between columns in origin block ( #27943 )
2023-12-25 20:35:22 +08:00
91e5b47439
[fix](hdfs) Fix HdfsFileSystem::exists_impl crash ( #28952 )
...
Calling hdfsGetLastExceptionRootCause without initializing ThreadLocalState
will crash. This PR modifies the condition for determining the existence of
a hdfs file, because hdfsExists will set errno to ENOENT when the file does
not exist, we can use this condition to check whether a file existence rather
than check the existence of the root cause.
2023-12-25 19:18:01 +08:00
2dc7d82aff
[Chore](Job)print log before task execute ( #28962 )
2023-12-25 16:55:59 +08:00
17917a0adb
[feature](load) enable memtable on sink node by default ( #28963 )
2023-12-25 16:27:16 +08:00
8e47fd28e2
[fix](doc) typo fix in auto-partition page ( #28512 )
2023-12-25 15:49:18 +08:00
29d3d5e433
[Test](Job)Add test case ( #28481 )
2023-12-25 15:12:14 +08:00
f2cdf15078
(enhance)(regression) Support force_olap_table_replication_num=3 run test_insert_random_distribution_table case ( #28903 )
2023-12-25 14:06:48 +08:00
c2eabbd441
[fix](load) fix nullptr when getting memtable flush running count ( #28942 )
...
* [fix](load) fix nullptr when getting memtable flush running count
* style
2023-12-25 13:49:18 +08:00
1d984e0ebb
return residual expr of join ( #28760 )
2023-12-25 12:53:14 +08:00
e9e1e2894b
[performance](variant) support topn 2phase read for variant column ( #28318 )
...
[performance](variant) support topn 2phase read for variant column
2023-12-25 11:50:41 +08:00
6de797844d
[doc](insert) Add group commit docs ( #25949 )
2023-12-25 11:34:11 +08:00
48f8f8f2ad
[fix](nereids)group by expr may be bound twice in bind agg slot ( #28771 )
2023-12-25 11:23:44 +08:00
c53611dcb3
[feature](nereids)support decimalv2 ( #28726 )
2023-12-25 11:23:27 +08:00
99755920e2
[fix](nereids)subquery unnest need handle subquery in Not expr correnctly ( #28713 )
2023-12-25 11:23:00 +08:00
6085a61f1e
[chore](test) correct create table statement ( #28863 )
2023-12-25 10:55:35 +08:00
f374beaa4e
[fix](log) regularise some BE error type and fix a load task check #28729
2023-12-25 10:45:19 +08:00
3273e0e635
[refactor](pipelineX)do not override dependency() function in pipelineX ( #28848 )
2023-12-25 10:36:31 +08:00
24b1b4d96b
[fix](pipelineX) fix use global rf when there no shared_scans ( #28869 )
2023-12-25 10:35:22 +08:00
e326ebb63e
[feature](pipelineX) control exchange sink by memory usage ( #28814 )
2023-12-25 10:31:50 +08:00
d42fd68d6b
[opt](invert index) Empty strings are not written to the index in the case of TOKENIZED ( #28822 )
2023-12-25 10:23:07 +08:00
b49671b31e
[feature](api) add profile text api ( #28697 )
2023-12-25 10:17:09 +08:00
ff365ca130
[docs] (DebugPoints) Update docs about Debug Points ( #28347 )
...
---------
Co-authored-by: qinhao <qinhao@newland.com.cn >
2023-12-25 09:33:47 +08:00
b7ae7a07c7
[fix](join) incorrect result of left semi/anti join with empty build side ( #28898 )
2023-12-25 09:07:38 +08:00
d8cb4da73d
[chore](prompt) Optimize tablet and replica prompt by pointing out what the numbers mean ( #28925 )
2023-12-24 22:29:24 +08:00
bade50db56
[chore](test) Add testing util sync point ( #28924 )
2023-12-24 21:59:11 +08:00
2f7deb3dea
[enhancement](bulk-load) cancel loading tasks directly without retrying when timeout exceeded ( #28666 )
2023-12-24 21:57:31 +08:00
1e44a4b145
[opt](nereids) convert or to inpredicate and optimize inpredicate partition prune ( #28316 )
2023-12-24 21:42:03 +08:00
145683ccdb
[improvement](group commit) make get column function more reliable when replaying wal ( #28900 )
2023-12-24 21:17:39 +08:00
7107415230
[fix](regression-test) test_partial_update_native_insert_stmt_complex is flaky ( #28927 )
2023-12-24 20:47:29 +08:00
b8de5cfc36
[fix] (nereids) Catch exception when mv fail and fix the npe ( #28932 )
2023-12-24 20:41:24 +08:00
1545c36d16
Revert "[bugfix](scannercore) scanner will core in deconstructor during collect profile ( #28727 )" ( #28931 )
...
This reverts commit 4066de375efe6ff8e156a61df4f9316b3d9eaa4e.
2023-12-24 20:37:33 +08:00
db1da161f5
[optimize](zonemap) skip zonemap if predicate does not support_zonemap ( #28595 )
...
* [optimize](zonemap) skip zonemap if predicate does not support_zonemap #27608 (#28506 )
2023-12-24 19:34:13 +08:00
