Commit Graph

6413 Commits

Author SHA1 Message Date
d7a3ff1ddf [Fix](Outfile) Fix the column type mapping in the orc/parquet file format (#32281)
| Doris Type             | Orc Type                     |  Parquet Type                |
|---------------------|--------------------|------------------------|
| Date                            | Long (logical: DATE)                 |       int32 (Logical: Date)                                        |
| DateTime                    | TIMESTAMP (logical: TIMESTAMP)    |       int96                          |
2024-03-22 08:52:16 +08:00
ca05129fa1 [fix](auth)node priv can login web ui (#32521) 2024-03-22 08:52:16 +08:00
c1d42eaa86 [fix](mtmv)resolve the issue of table version updates in concurrent situations (#32487)
Move the logic for version+1 from `unprotectedCommitTransaction `to`FinishTransaction`, as the write lock for the table was obtained in `FinishTransaction`
2024-03-22 08:52:16 +08:00
fd0bc720e9 [opt](information_schema) Add DEFAULT_ENCRYPTION column to schemata table (#32501) 2024-03-22 08:52:16 +08:00
886aabda31 [Fix](executor) Fix insert select not close coordinator #32571 2024-03-22 08:52:16 +08:00
b727fd6594 (kerberos)fix hive keberos principal usage 2024-03-22 08:16:59 +08:00
0cde0cbf19 (invert index) modify of time series compaction policy 2024-03-22 08:16:30 +08:00
dea6859e0d [refactor](jdbc catalog) refactor jdbc catalog get databases logic (#32579) 2024-03-21 14:07:50 +08:00
d2968dcf99 [fix](jdbc catalog) Fixed the problem that the metadata_refresh_interval_sec properties cannot be set (#32516) 2024-03-21 14:07:49 +08:00
085696744d [Enhancement] when partition column is datetime, date can work in create table command (#32335) 2024-03-21 14:07:49 +08:00
8e3d28b93a [Feature] Support Array Type compare function for nereids planner (#31701)
Support Array Type compare function for nereids planner
2024-03-21 14:07:49 +08:00
09be4dc7ee [fix](random-bucket) tabletindex when there is no cached value in memory (#32336)
1. In cloud mode, get visible version is a rpc to metaservice, while
loads would get visible version for all partitions.
2. VunionNode should follow batch size.
2024-03-21 14:07:49 +08:00
95322e2ebe [opt](variable) user variable support expression rather than literal (#32492) 2024-03-21 14:07:49 +08:00
2a1d2ad647 [fix](Nereids) decimalv2 and int/date bit arithmetic need cast to bigint (#32311) 2024-03-21 14:07:49 +08:00
b92a764665 [feature](function) Support for aggregate function foreach combiner for some error function (#31913)
Support for aggregate function foreach combiner for some error function
2024-03-21 14:07:49 +08:00
22782fec92 [opt](routine-load) optimize allocate task to be algorithm for load balance (#32021) 2024-03-21 14:07:49 +08:00
6871c964af [fix](nereids)NullSafeEqualToEqual rule only change to equal if both children are not nullable (#32374)
NullSafeEqualToEqual rule only change to equal if both children are not nullable
2024-03-21 14:07:49 +08:00
1b0c4d3aa2 [chor](log) Change log4j rollover strategy to 'max' (#32116)
The log file names are in the format fe.log.${date}-${index}, e.g.
```
fe.log.20240311-1
fe.log.20240311-2
fe.log.20240311-3
...
```
In the previous, fe.log/fe.audit.log will be renamed to fe.log.xxx-1/fe.audit.log.xxx-1
with the minimum index `1` when they reach rotation size. e.g.
```
fe.log.20240311-1 -> fe.log.20240311-2
fe.log.20240311-2 -> fe.log.20240311-3
fe.log.20240311-3 -> fe.log.20240311-4
fe.log            -> fe.log.20240311-1
```

there are some drawbacks
1. log4j needs to rename all existing rotated log to spare the
   index `1`
2. it's hard to deal with the duplicated names, because every time the
   log rotates, it starts from index `1`

After this change, the log will rename to a file name with larger index
instead of `1`. e.g. when rotate
```
fe.log.20240311-1
fe.log.20240311-2
fe.log.20240311-3
fe.log -> fe.log.20240311-4
```
2024-03-21 14:07:49 +08:00
6d401451cc [fix](stats) sort partitions when do sample analyze (#32185) 2024-03-21 14:07:49 +08:00
4b21c5c1cf [Enhencement](Nereids) add eliminate distinct constant rule (#32307)
select distinct 1,2,3 from tbl
=>
select 1,2,3 from (select 1, 2, 3 from tbl limit 1) as tmp
2024-03-21 14:07:24 +08:00
e892774c9a [improvement](agg) streaming agg should not take too much memory when spilling enabled (#32426) 2024-03-21 14:07:24 +08:00
7484a7ba5f [fix](broker load) improve the checking of overlapping partitions of same table (#32254) 2024-03-21 14:07:24 +08:00
3c377a8957 [fix](group commit) Fix group commit connect to observer fe (#32222) 2024-03-21 14:07:24 +08:00
66fe61b591 [fix](nereids)support topn-filter for non pipeline engine #32397 2024-03-21 14:07:24 +08:00
e541ca9f11 [Fix](Job)When jobname is the do keyword, parsing errors will occur when executing SQL. (#32379) 2024-03-21 14:07:24 +08:00
73de61ed84 [opt](hive) skip hidden file and dir (#32412)
When query hive table, we should skip all hidden dirs and files, like:
```
/visible/.hidden/path
/visible/.hidden.txt
```
2024-03-21 14:07:24 +08:00
a0a3a2a2ce [Fix](Variant) fix variant with not null (#32248)
ignore null bitmap for not null and make subcolumn access slots always nullable
2024-03-21 14:07:23 +08:00
6aec479013 [fix](planner)decimalv3 literal's precision and scale is not correctly set (#32288) 2024-03-21 14:07:23 +08:00
a4151e022e [bug](fold) fix fold constant rule can't handle variable expr (#32313) 2024-03-21 14:07:23 +08:00
26ed4b69b1 [opt](jdbc catalog) filter jdbc datasource internal database (#32294) 2024-03-21 14:07:23 +08:00
4d4cd43458 [Fix](Nereids) fix leading syntax problems and data mismatched problem (#32286)
- fix syntax problems of only one table used in leading or mistake usage of brace
  example: leading(t1),leading(t1 {t2})
- fix cte used in subquery of using leading
  example: with cte as (select c1 from t1) select count(*) from t1 join (select /*+ leading(cte t2) */ c2 from t2 join cte on c2 = cte.c1) as alias on t1.c1 = alias.c2;
  which cte used in subquery and subquery also have leading
- fix data mismatched with original plan cause of on predicate push to nullable side
  example: select count(*) from t1 left join t2 on c1 > 500 and c2 > 500 can not change to select count(*) from t1 left join t2 on c2 > 500 where c1 > 500
2024-03-21 14:07:23 +08:00
fab48f54b1 [enhancement](nereids)simplify OneRowRelation scalar subquery (#32276)
select count() from t where dt > (select '2024-02-02 00:00:00');
-->
select count() from t where dt > '2024-02-02 00:00:00';
2024-03-21 14:07:23 +08:00
403820599d [bug](inverted index) fix npe of InvertedIndexStorageFormat in table property (#32357)
Fix problem that if fe upgrade from a older version, it has error like:

```
MySQL [test]> show full tables;
ERROR 1105 (HY000): NullPointerException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.thrift.TInvertedIndexStorageFormat.toString()" because the return value of "org.apache.doris.catalog.OlapTable.getInvertedIndexStorageFormat()" is null
```
2024-03-21 14:07:22 +08:00
279ea2f366 [feature](proxy-protocol) Support proxy protocol v1 (#32338)
Enable proxy protocol to support IP transparency.
See: `IP Transparency` in f57387b502/docs/en/docs/admin-manual/cluster-management/load-balancing.md
for details
2024-03-21 14:07:22 +08:00
85b2c42f76 [Enhancement](jdbc catalog) Add a property to test the connection when creating a Jdbc catalog (#32125) (#32531) 2024-03-21 14:05:59 +08:00
fc2588c786 [fix](insert)fix sink user name (#32465) 2024-03-19 16:04:09 +08:00
93fe9521bf [feature](insert)fix implement hive table sink plan (#32430)
introduced by #32386
2024-03-19 09:55:16 +08:00
ecadb60bcd [Pick 2.1](inverted index) support inverted index format v2 (#30145) (#32418) 2024-03-19 08:11:33 +08:00
711c0cd55c [feature](insert)implement hive table sink plan (#31765) (#32386)
from #31765
2024-03-18 22:49:30 +08:00
a444e84be6 [feature](hive)add 'HmsCommiter' to support inserting data into hive table (#32283) (#32362)
bp #32283
Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-03-18 10:59:32 +08:00
1645f2e0a7 [feature](insert)add hive table sink definition (#31662) (#32347)
bp #31662
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
2024-03-17 20:52:44 +08:00
4732aae628 [Refactor](insert) refactor insert command to support other type of table (#31610) (#32345)
bp #31610
2024-03-17 20:46:07 +08:00
47019133c0 [improvement](Nereids) Support to remove sort which is under table sink (#31751) (#32337) 2024-03-17 15:45:53 +08:00
c34f5045c8 fix compile 2024-03-16 21:37:02 +08:00
83ab61ad22 Add QUEUE_START_TIME/QUEUE_END_TIME/QUERY_STATUS column for active_queries (#32259) 2024-03-16 20:53:46 +08:00
a15bf3057f [Fix](nereids) remove duplicate expr in grouping set (#32290)
db reported a error " expression duplicate in grouping set" when there are duplicate expression in grouping set.
e.g.select a from mal_test1 group by grouping sets((a,a))
This pr removes duplicate expr in grouping set :
select a from mal_test1 group by grouping sets((a))
2024-03-16 20:53:46 +08:00
f64a9a33f8 [fix](Nereids): don't pushdown project when project contains both side of join (#32214) 2024-03-16 20:53:46 +08:00
a90a1a76f1 [bugfix](profile) support multi execution profile for brokerload (#32280)
The bug is introduced by #27184
Profile Format is :
Summary
MergedProfile
ExecutionProfile1
ExecutionProfile2
...

There maybe multiple execution profiles for broker load.
2024-03-16 20:53:43 +08:00
9ad196f189 Revert "[fix](cloud) ignore some case in cloud mode (#32261)"
This reverts commit c0776c7c0756d602204edba76642cafa92e67cd8.
2024-03-16 14:11:22 +08:00
258dcfca97 [Refactor](executor)Add information_schema.workload_groups (#32195) (#32314) 2024-03-15 20:46:54 +08:00