Commit Graph

15086 Commits

Author SHA1 Message Date
d0fea8db27 [chore][log] Opt log, revert some log introduced by #25739 (#26365) 2023-11-27 10:48:02 +08:00
550f3e801d [improve](routine_load) move log from write lock (#27576) 2023-11-27 10:47:31 +08:00
d10a708fa2 [improve](jdbc catalog) add profile for jdbc scan (#27447) 2023-11-27 10:33:39 +08:00
cd6c61347d [Feature](tvf)(avro-jni) avro-jni add projection push down (#26885) 2023-11-27 10:33:27 +08:00
baadc14e60 [Enhancement](function) support unix_timestamp with float (#26827)
---------

Co-authored-by: YangWithU <plzw8@outlook.com>
2023-11-27 09:58:53 +08:00
3791de3cfa [feature](mtmv)(6)implement cancel method (#27541)
1.implement cancel task method
2.fix `show create table ` not display `comment`
2023-11-27 09:49:46 +08:00
5700332c3c [enhance](S3) Print the error detail for every s3 operation (#27572) 2023-11-26 18:54:43 +08:00
b6dd43bb9d [fix](load) return error instead of panic when slot and expr do not (#27423) 2023-11-26 18:53:37 +08:00
148f365da8 [fix](move-memtable) handle tablet location not found (#27561) 2023-11-26 18:42:15 +08:00
04033dce01 [improve](routine_load) add db and table name in create routine load job log (#27500) 2023-11-25 23:41:34 +08:00
b08865982a [improvement](disk balance) Impr disk rebalancer sched when be load r… (#26412) 2023-11-25 23:35:16 +08:00
ff1a06abcf [test](regression) add routine load sequence and error test (#27519) 2023-11-25 23:30:20 +08:00
4232327a43 [fix](regression test) fix test_schema_change_cold_heat case (#27556) 2023-11-25 23:25:37 +08:00
7a1b4267d5 [improve](transaction) lightweight transaction lock logic (#27564) 2023-11-25 23:24:36 +08:00
d4f2db74f9 [fix](nereids) fix stats error when using dateTime type filter (#27571)
Currently doris doesn't support datetime type filter stats estimation, but only for date type.
It will cause the filter using datetime type column with the same date and different time computing out a inaccurate selectivity and estimate a wrong row count, such as :

where o.book_time >= '2020-03-01 00:00:00.0' and o.book_time <= '2020-03-01 23:59:59.0';

This pr adds the datetime type(only support hh:mm:ss scale) filter estimation and improve the row count estimation for the above case.
2023-11-25 11:04:51 +08:00
1b8d7da078 [chore](case) Use correct insert stmt for cold heat separation case #27546 2023-11-25 10:37:57 +08:00
7134da63ac [improvement](nereids) add config for nereids trace dir (#27495)
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-11-25 10:37:23 +08:00
3d2b194985 [refactor](if_function) handle result status of function (#27491) 2023-11-25 10:35:55 +08:00
cc395f5428 [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. (#27563) 2023-11-25 10:29:39 +08:00
6b1428dba1 Fix auto analyze doesn't filter unsupported type bug. (#27559)
Fix auto analyze doesn't filter unsupported type bug.
Catch throwable in auto analyze thread for each database, otherwise the thread will quit when one database failed to create jobs and all other databases will not get analyzed.
change FE config item full_auto_analyze_simultaneously_running_task_num to auto_analyze_simultaneously_running_task_num
2023-11-25 10:22:52 +08:00
6142a539f4 [deprecated](external) remove deprecated hudi and iceberg external table (#27456)
The creation of hudi and iceberg table is disallowed since v1.2.
All these features are covered by hudi/iceberg catalog.
We should remove the code in v2.1

The PR mainly changes:
1. remove the code of hudi/iceberg external table.
2. remove code of iceberg database.
3. disallowed hive external table's creation.
4. disabled odbc,mysql,broker external table by default, and add FE config `disable_odbc_mysql_broker_table` to control it
2023-11-24 23:18:07 +08:00
59efb1b238 [fix](planner)join node should output required slot from parent node #27526 2023-11-24 21:50:32 +08:00
eb250bca5c [fix](set) fix error message when set a session only variable (#27554) 2023-11-24 21:48:16 +08:00
f3204ec18c [fix](memory) Fix thread_context check failed: thread_mem_tracker()->label() == "Orphan" #27468 2023-11-24 20:57:41 +08:00
14527c4930 [chore](case) adjust timeout of broker load case (#27539)
when running p2, this case may fail because of timeout.
2023-11-24 20:55:26 +08:00
a0b1cb48a1 [Improve](regresscases) update cases for three-level nested types #27529 2023-11-24 20:53:28 +08:00
e4258dd868 [chore](fe plugin) Upgrade dependency to doris 2.0-SNAPSHOT #27522 2023-11-24 19:30:52 +08:00
d98d2d491f [chore](tablet scheduler) update sched ctx err (#27514) 2023-11-24 18:09:49 +08:00
d769fb69dd NEED (#27531) 2023-11-24 18:04:36 +08:00
70bbaa4e56 [test](regression) add cases about datev1/datatimev1 (#27543)
All cases' results are tested and passed with datetime/date v2
Cases about:
Calculation ( +, -
Kinds of predicates(<, >, =, <>, in, not in, is null, is not null)
Load test(from csv and select into)
Runtime filter
Delete conditions
Key columns(agg/duplicate/uniq model, distributed/partition, bitmap index...)
2023-11-24 17:58:32 +08:00
096ce9d75b [typo](doc)Add flink cdc partial column update column (#26402) 2023-11-24 17:10:33 +08:00
553e4a8903 [feature-wip](merge-on-write) MOW table support different primary keys and sort keys (#24788) 2023-11-24 16:37:30 +08:00
2152d5a5dd [regression test](control) add some regression case for control (#27346)
* [regression test](control) add some regression case for control

```
1. CANCEL DECOMMISSION BACKEND
2. SET REPLICA STATUS
3. CLEAN TRASH
4. SET REPLICA VERSION
5. BAD REPLICA
```
2023-11-24 16:13:12 +08:00
fdcb401835 [doc](fix) json load supported format (#27415)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-11-24 16:06:44 +08:00
89d5efac46 [pipelineX](dependency) Use dependency to replace ready_to_execute (#27528) 2023-11-24 16:03:14 +08:00
Pxl
33b540641e [Bug](materialized-view) add limitation for duplicate expr on materialized view (#27523)
add limitation for duplicate expr on materialized view
2023-11-24 15:47:14 +08:00
126714ace4 [fix](pipelineX) fix nullptr in loca exchange dependency (#27488)
fix nullptr in loca exchange dependency
2023-11-24 15:12:25 +08:00
ce99de3c7a [regression-test](fix)add ctas test cases. (#24278) 2023-11-24 14:44:11 +08:00
674dd98fb2 [fix](Nereids): CustomRewriteJob don't handle condition rule return null (#27516) 2023-11-24 14:38:37 +08:00
540132f656 [fix](profilev2) fix merge profile min is zero #27524 2023-11-24 14:11:11 +08:00
dbff9d7a89 [chore](fuzzy) topn_opt_limit_threshold (#27496) 2023-11-24 14:08:27 +08:00
dbbab63623 [fix](nereids)keep cast operator if cast a varchar to another longer varchar in LogicalSetOperator (#27393) 2023-11-24 14:07:04 +08:00
dfe3a2dd01 [feature](mtmv)(3)Implementing multi table materialized views (#26146)
Introduction to Main Classes:
- MTMVService:MTMV services for other modules to call
- MTMVHookService:All operations that affect the MTMV
  - MTMVJobManager:All operations that affect the MTMV job
  - MTMVCacheManager:All operations that affect the MTMV Cache
- MTMVTask&MTMVJob:Inherit from job framework
2023-11-24 12:34:38 +08:00
da71fde066 [fix](build index) Fix inverted index hardlink leak and missing problem (#26903) 2023-11-24 10:30:21 +08:00
d5908b0c8b [regression](compaction) Add case to test base_compaction with dup_key_max_file_size_limit (#27088) 2023-11-24 10:27:51 +08:00
c24a33c857 [enhancement](audit) hide password and other sensitive information in log and audit log (#27115)
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-11-24 10:27:30 +08:00
0a8e3d2199 [enhance](PrefetchReader) Make the prefetch timeout one config (#27371) 2023-11-24 10:24:15 +08:00
9a92323221 [fix](fs) prefetch buffer check status before verifing buf size (#27388) 2023-11-24 10:19:37 +08:00
160adbaa69 [regression test](routine test) add case for num_as_string (#27436) 2023-11-24 10:15:47 +08:00
75c9f00137 [Bug](bitmap) Fix heap-use-after-free in the bitmap functions (#27411) 2023-11-24 10:09:31 +08:00