Commit Graph

15109 Commits

Author SHA1 Message Date
4ea69ed390 [regression test](broker load) add case for num_as_string (#27588) 2023-11-27 21:25:59 +08:00
bb68900bed [fix](bdbje) Fix bdbje logging level not work (#27597)
* `EnvironmentConfig.FILE_LOGGING_LEVEL` only set FileHandlerLevel, we should
   set logger level firstly, otherwise it will not take effect.
2023-11-27 21:24:34 +08:00
646f1ea087 [performance](Nereids): avoid use getStringValue() in getTimeFormatter() (#27625)
Original `getTimeFormatter()` will convert `long` to `string`, and then parse `string` to `int`.
2023-11-27 21:08:32 +08:00
HB
c7b9a32e3e [improvement](show snapshot) show iceberg snapshot print summary info (#27448)
Iceberg's snapshot has summary information, but Doris did not display it. This patch fixes this issue.
2023-11-27 20:56:50 +08:00
3d7d166355 [feature](cmd) add UNSET_VARIABLE statement to set back variables (#27552) 2023-11-27 20:30:04 +08:00
HB
36a528b6bc [fix](judge-partition) Fix incorrect logic in determining whether it is a partitioned table (#27515)
The old logic used to determine whether it was a partition table based on the number of buckets, but if I had a partition table with only one partition and the number of buckets in that partition was 1, it would be mistakenly recognized as a non partition table.

```
Table[test_load_doris_to_hive_2] is not partitioned
```
2023-11-27 18:56:52 +08:00
50c442fc6c [DOC](sparkload)add spark load faq (#27455)
add spark load FAQ
2023-11-27 17:49:52 +08:00
d5a56dc7f4 [information_schema](tables)modify information_schema.tables rows column use cache rows. (#27028)
Use the cached information and estimated information of the table in the rows column under 
information_schema.tables. Avoid querying information_schema.tables that will cause rpc timeout when there are a 
large number of tables in the catalog.
2023-11-27 17:48:06 +08:00
66eeafcd48 [refactor](Nereids): unify one DateLiteral init() (#27618)
`fromDateStr` will parse `date string` into `dateLiteral`, but `init()` already handle it, so we can use `init()` replace it.
2023-11-27 17:09:45 +08:00
fde4bab048 [fix](Nereids) non-deterministic expression should not be constant (#27606) 2023-11-27 16:40:30 +08:00
3d0dc94b18 [fix](ci) fix bug that "run build\n" not trigger pipeline (#27617)
Co-authored-by: stephen <hello-stephen@qq.com>
2023-11-27 16:23:42 +08:00
cbdb886b6e [fix](Nereids): fill up miss slot of order having project (#27480)
fill up miss slot of order having project such as 
```
select a + 1 as c from t having by c > 2 order by a 
```
2023-11-27 16:00:29 +08:00
612347f650 [fix](planner)sort node should materialized required slots for itself (#27605)
this is a follow up pr for #27526 . The old pr didn't fix the problem correctly, this pr do.
2023-11-27 15:37:11 +08:00
dc1a31715b [doc](flink) Update doc index title (#27410) 2023-11-27 15:32:10 +08:00
13b26ee920 [Fix](core) Fix wal space back pressure core and add regression test (#27311) 2023-11-27 15:10:26 +08:00
234aff3e78 [feature](Nereids): Pushdown TopN through Union (#27535)
```
topn
-> Union All 
  -> child plan1
  -> child plan2
  -> child plan3

rewritten to

topn
-> Union All 
 -> topn
  -> child plan1
 -> topn
  -> child plan2
 -> topn
  -> child plan3
```
2023-11-27 14:13:18 +08:00
1b4cd24b36 [opt](Nereids) support where, group by, having, order by clause without from clause in query statement (#27006)
Support where, group by, having, order by clause without from clause in query statement.
For example as following:

SELECT 1 AS a, COUNT(), SUM(2), AVG(1), RANK() OVER() AS w_rank
WHERE 1 = 1
GROUP BY a, w_rank
HAVING COUNT() IN (1, 2) AND w_rank = 1
ORDER BY a;

this will return result:

| a  |count(*)|sum(2)|avg(1)|w_rank|
+----+--------+------+------+------+
| 1  |       1|     2|   1.0|     1|


For another example as following:

select 1 c1, 2 union (select "hell0", "") order by c1
the second column datatype will be varchar(65533), 65533 is the default varchar length.

this will return result:

|c1    | 2 |
+------+---+
|1     | 2 |
|hell0 |   |
2023-11-27 12:05:14 +08:00
331effdb20 [feature](Nereids): support merge graph in group (#27353) 2023-11-27 11:48:38 +08:00
0e1e4c8508 [opt](nereids) disable infer column name when query (#27450)
Disable infer column name when query, because it cause some errors when using BI tools
This feature is firstly developed by #26055
2023-11-27 11:26:17 +08:00
5cb5241a9e [feature](mtmv) materialized view rewrite framework (#27059)
materialized view rewrite framework, support to query rewrite by struct info.
The idea is from "Optimizing Queries Using Materialized Views- A Practical, Scalable Solution"
2023-11-27 11:15:54 +08:00
3838b6fbae [refine](pipelineX) refine some code in pipelineX (#27472) 2023-11-27 11:04:16 +08:00
82d15669bc [minor](fe) convert Chinese annotations into English (#27560) 2023-11-27 11:03:44 +08:00
9aafcf2e22 [Enhance](fe) Support setting initial root password when FE firstly launch (#27438) 2023-11-27 11:03:27 +08:00
d0fea8db27 [chore][log] Opt log, revert some log introduced by #25739 (#26365) 2023-11-27 10:48:02 +08:00
550f3e801d [improve](routine_load) move log from write lock (#27576) 2023-11-27 10:47:31 +08:00
d10a708fa2 [improve](jdbc catalog) add profile for jdbc scan (#27447) 2023-11-27 10:33:39 +08:00
cd6c61347d [Feature](tvf)(avro-jni) avro-jni add projection push down (#26885) 2023-11-27 10:33:27 +08:00
baadc14e60 [Enhancement](function) support unix_timestamp with float (#26827)
---------

Co-authored-by: YangWithU <plzw8@outlook.com>
2023-11-27 09:58:53 +08:00
3791de3cfa [feature](mtmv)(6)implement cancel method (#27541)
1.implement cancel task method
2.fix `show create table ` not display `comment`
2023-11-27 09:49:46 +08:00
5700332c3c [enhance](S3) Print the error detail for every s3 operation (#27572) 2023-11-26 18:54:43 +08:00
b6dd43bb9d [fix](load) return error instead of panic when slot and expr do not (#27423) 2023-11-26 18:53:37 +08:00
148f365da8 [fix](move-memtable) handle tablet location not found (#27561) 2023-11-26 18:42:15 +08:00
04033dce01 [improve](routine_load) add db and table name in create routine load job log (#27500) 2023-11-25 23:41:34 +08:00
b08865982a [improvement](disk balance) Impr disk rebalancer sched when be load r… (#26412) 2023-11-25 23:35:16 +08:00
ff1a06abcf [test](regression) add routine load sequence and error test (#27519) 2023-11-25 23:30:20 +08:00
4232327a43 [fix](regression test) fix test_schema_change_cold_heat case (#27556) 2023-11-25 23:25:37 +08:00
7a1b4267d5 [improve](transaction) lightweight transaction lock logic (#27564) 2023-11-25 23:24:36 +08:00
d4f2db74f9 [fix](nereids) fix stats error when using dateTime type filter (#27571)
Currently doris doesn't support datetime type filter stats estimation, but only for date type.
It will cause the filter using datetime type column with the same date and different time computing out a inaccurate selectivity and estimate a wrong row count, such as :

where o.book_time >= '2020-03-01 00:00:00.0' and o.book_time <= '2020-03-01 23:59:59.0';

This pr adds the datetime type(only support hh:mm:ss scale) filter estimation and improve the row count estimation for the above case.
2023-11-25 11:04:51 +08:00
1b8d7da078 [chore](case) Use correct insert stmt for cold heat separation case #27546 2023-11-25 10:37:57 +08:00
7134da63ac [improvement](nereids) add config for nereids trace dir (#27495)
Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-11-25 10:37:23 +08:00
3d2b194985 [refactor](if_function) handle result status of function (#27491) 2023-11-25 10:35:55 +08:00
cc395f5428 [Fix](hive-transactional-table) Fix NPE when query empty hive transactional table. (#27563) 2023-11-25 10:29:39 +08:00
6b1428dba1 Fix auto analyze doesn't filter unsupported type bug. (#27559)
Fix auto analyze doesn't filter unsupported type bug.
Catch throwable in auto analyze thread for each database, otherwise the thread will quit when one database failed to create jobs and all other databases will not get analyzed.
change FE config item full_auto_analyze_simultaneously_running_task_num to auto_analyze_simultaneously_running_task_num
2023-11-25 10:22:52 +08:00
6142a539f4 [deprecated](external) remove deprecated hudi and iceberg external table (#27456)
The creation of hudi and iceberg table is disallowed since v1.2.
All these features are covered by hudi/iceberg catalog.
We should remove the code in v2.1

The PR mainly changes:
1. remove the code of hudi/iceberg external table.
2. remove code of iceberg database.
3. disallowed hive external table's creation.
4. disabled odbc,mysql,broker external table by default, and add FE config `disable_odbc_mysql_broker_table` to control it
2023-11-24 23:18:07 +08:00
59efb1b238 [fix](planner)join node should output required slot from parent node #27526 2023-11-24 21:50:32 +08:00
eb250bca5c [fix](set) fix error message when set a session only variable (#27554) 2023-11-24 21:48:16 +08:00
f3204ec18c [fix](memory) Fix thread_context check failed: thread_mem_tracker()->label() == "Orphan" #27468 2023-11-24 20:57:41 +08:00
14527c4930 [chore](case) adjust timeout of broker load case (#27539)
when running p2, this case may fail because of timeout.
2023-11-24 20:55:26 +08:00
a0b1cb48a1 [Improve](regresscases) update cases for three-level nested types #27529 2023-11-24 20:53:28 +08:00
e4258dd868 [chore](fe plugin) Upgrade dependency to doris 2.0-SNAPSHOT #27522 2023-11-24 19:30:52 +08:00