Commit Graph

7914 Commits

Author SHA1 Message Date
2e25b0079b fix compile 2024-03-22 09:17:40 +08:00
e5d7ff64aa [fix](backup) clear snapshot info for cancelled backup job to reduce log size (#32604) 2024-03-22 08:53:17 +08:00
6b54171778 [bugfix](deadlock) pipelinex map lock should only scope in map not about pipelinectx's cancel method (#32622)
both global lock in fragment mgr should only protect the map logic, could not use it to protect cancel method.
fragment ctx cancel method should be protected by a lock.
query ctx cancel --> pipelinex fragment cancel ---> query ctx cancel will dead lock.
2024-03-22 08:52:38 +08:00
06026b2f46 [opt](paimon) add split num and partition num (#32597)
1. And selected partition number indicator in PaimonScanNode in explain result.
2. Add raw file split and total split indicator in PaimonScanNode in explain result.
3. Opt the error msg of table valued function, print the error msg first to avoid the long msg be truncated.
2024-03-22 08:52:16 +08:00
8d6561b32d [opt](jdbc catalog) set jdbc test_connection default true (#32546) 2024-03-22 08:52:16 +08:00
60eeff8e18 [enhance](mtmv)refresh mtmv must add auto (#32522) 2024-03-22 08:52:16 +08:00
d7a3ff1ddf [Fix](Outfile) Fix the column type mapping in the orc/parquet file format (#32281)
| Doris Type             | Orc Type                     |  Parquet Type                |
|---------------------|--------------------|------------------------|
| Date                            | Long (logical: DATE)                 |       int32 (Logical: Date)                                        |
| DateTime                    | TIMESTAMP (logical: TIMESTAMP)    |       int96                          |
2024-03-22 08:52:16 +08:00
ca05129fa1 [fix](auth)node priv can login web ui (#32521) 2024-03-22 08:52:16 +08:00
c1d42eaa86 [fix](mtmv)resolve the issue of table version updates in concurrent situations (#32487)
Move the logic for version+1 from `unprotectedCommitTransaction `to`FinishTransaction`, as the write lock for the table was obtained in `FinishTransaction`
2024-03-22 08:52:16 +08:00
fd0bc720e9 [opt](information_schema) Add DEFAULT_ENCRYPTION column to schemata table (#32501) 2024-03-22 08:52:16 +08:00
886aabda31 [Fix](executor) Fix insert select not close coordinator #32571 2024-03-22 08:52:16 +08:00
b727fd6594 (kerberos)fix hive keberos principal usage 2024-03-22 08:16:59 +08:00
0cde0cbf19 (invert index) modify of time series compaction policy 2024-03-22 08:16:30 +08:00
4c8aaa156a [fix](jni) remove 'push_down_predicates' and fix BE crash with decimal predicate (#32253) (#32599) 2024-03-21 14:07:50 +08:00
dea6859e0d [refactor](jdbc catalog) refactor jdbc catalog get databases logic (#32579) 2024-03-21 14:07:50 +08:00
d2968dcf99 [fix](jdbc catalog) Fixed the problem that the metadata_refresh_interval_sec properties cannot be set (#32516) 2024-03-21 14:07:49 +08:00
085696744d [Enhancement] when partition column is datetime, date can work in create table command (#32335) 2024-03-21 14:07:49 +08:00
8e3d28b93a [Feature] Support Array Type compare function for nereids planner (#31701)
Support Array Type compare function for nereids planner
2024-03-21 14:07:49 +08:00
09be4dc7ee [fix](random-bucket) tabletindex when there is no cached value in memory (#32336)
1. In cloud mode, get visible version is a rpc to metaservice, while
loads would get visible version for all partitions.
2. VunionNode should follow batch size.
2024-03-21 14:07:49 +08:00
95322e2ebe [opt](variable) user variable support expression rather than literal (#32492) 2024-03-21 14:07:49 +08:00
2a1d2ad647 [fix](Nereids) decimalv2 and int/date bit arithmetic need cast to bigint (#32311) 2024-03-21 14:07:49 +08:00
b92a764665 [feature](function) Support for aggregate function foreach combiner for some error function (#31913)
Support for aggregate function foreach combiner for some error function
2024-03-21 14:07:49 +08:00
22782fec92 [opt](routine-load) optimize allocate task to be algorithm for load balance (#32021) 2024-03-21 14:07:49 +08:00
6871c964af [fix](nereids)NullSafeEqualToEqual rule only change to equal if both children are not nullable (#32374)
NullSafeEqualToEqual rule only change to equal if both children are not nullable
2024-03-21 14:07:49 +08:00
1b0c4d3aa2 [chor](log) Change log4j rollover strategy to 'max' (#32116)
The log file names are in the format fe.log.${date}-${index}, e.g.
```
fe.log.20240311-1
fe.log.20240311-2
fe.log.20240311-3
...
```
In the previous, fe.log/fe.audit.log will be renamed to fe.log.xxx-1/fe.audit.log.xxx-1
with the minimum index `1` when they reach rotation size. e.g.
```
fe.log.20240311-1 -> fe.log.20240311-2
fe.log.20240311-2 -> fe.log.20240311-3
fe.log.20240311-3 -> fe.log.20240311-4
fe.log            -> fe.log.20240311-1
```

there are some drawbacks
1. log4j needs to rename all existing rotated log to spare the
   index `1`
2. it's hard to deal with the duplicated names, because every time the
   log rotates, it starts from index `1`

After this change, the log will rename to a file name with larger index
instead of `1`. e.g. when rotate
```
fe.log.20240311-1
fe.log.20240311-2
fe.log.20240311-3
fe.log -> fe.log.20240311-4
```
2024-03-21 14:07:49 +08:00
6d401451cc [fix](stats) sort partitions when do sample analyze (#32185) 2024-03-21 14:07:49 +08:00
4b21c5c1cf [Enhencement](Nereids) add eliminate distinct constant rule (#32307)
select distinct 1,2,3 from tbl
=>
select 1,2,3 from (select 1, 2, 3 from tbl limit 1) as tmp
2024-03-21 14:07:24 +08:00
e892774c9a [improvement](agg) streaming agg should not take too much memory when spilling enabled (#32426) 2024-03-21 14:07:24 +08:00
7484a7ba5f [fix](broker load) improve the checking of overlapping partitions of same table (#32254) 2024-03-21 14:07:24 +08:00
3c377a8957 [fix](group commit) Fix group commit connect to observer fe (#32222) 2024-03-21 14:07:24 +08:00
66fe61b591 [fix](nereids)support topn-filter for non pipeline engine #32397 2024-03-21 14:07:24 +08:00
e541ca9f11 [Fix](Job)When jobname is the do keyword, parsing errors will occur when executing SQL. (#32379) 2024-03-21 14:07:24 +08:00
73de61ed84 [opt](hive) skip hidden file and dir (#32412)
When query hive table, we should skip all hidden dirs and files, like:
```
/visible/.hidden/path
/visible/.hidden.txt
```
2024-03-21 14:07:24 +08:00
a0a3a2a2ce [Fix](Variant) fix variant with not null (#32248)
ignore null bitmap for not null and make subcolumn access slots always nullable
2024-03-21 14:07:23 +08:00
6aec479013 [fix](planner)decimalv3 literal's precision and scale is not correctly set (#32288) 2024-03-21 14:07:23 +08:00
6c8b5bb26f [fix](feut) comment out doc gen execution (#32413)
Followup #32384
After doc is removed, the doc generator should be skipped, or FE ut can not run
2024-03-21 14:07:23 +08:00
a4151e022e [bug](fold) fix fold constant rule can't handle variable expr (#32313) 2024-03-21 14:07:23 +08:00
26ed4b69b1 [opt](jdbc catalog) filter jdbc datasource internal database (#32294) 2024-03-21 14:07:23 +08:00
e952b5ef5b [opt](jdbc catalog) Refine the jdbc_connector close logic and actively clear the jvm occupied by jdbcexecutor (#32300) 2024-03-21 14:07:23 +08:00
f132c9b2c6 [Improve](spark-load)update spark version for spark load to resolve cve problem (#30368) 2024-03-21 14:07:23 +08:00
4d4cd43458 [Fix](Nereids) fix leading syntax problems and data mismatched problem (#32286)
- fix syntax problems of only one table used in leading or mistake usage of brace
  example: leading(t1),leading(t1 {t2})
- fix cte used in subquery of using leading
  example: with cte as (select c1 from t1) select count(*) from t1 join (select /*+ leading(cte t2) */ c2 from t2 join cte on c2 = cte.c1) as alias on t1.c1 = alias.c2;
  which cte used in subquery and subquery also have leading
- fix data mismatched with original plan cause of on predicate push to nullable side
  example: select count(*) from t1 left join t2 on c1 > 500 and c2 > 500 can not change to select count(*) from t1 left join t2 on c2 > 500 where c1 > 500
2024-03-21 14:07:23 +08:00
fab48f54b1 [enhancement](nereids)simplify OneRowRelation scalar subquery (#32276)
select count() from t where dt > (select '2024-02-02 00:00:00');
-->
select count() from t where dt > '2024-02-02 00:00:00';
2024-03-21 14:07:23 +08:00
403820599d [bug](inverted index) fix npe of InvertedIndexStorageFormat in table property (#32357)
Fix problem that if fe upgrade from a older version, it has error like:

```
MySQL [test]> show full tables;
ERROR 1105 (HY000): NullPointerException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.thrift.TInvertedIndexStorageFormat.toString()" because the return value of "org.apache.doris.catalog.OlapTable.getInvertedIndexStorageFormat()" is null
```
2024-03-21 14:07:22 +08:00
279ea2f366 [feature](proxy-protocol) Support proxy protocol v1 (#32338)
Enable proxy protocol to support IP transparency.
See: `IP Transparency` in f57387b502/docs/en/docs/admin-manual/cluster-management/load-balancing.md
for details
2024-03-21 14:07:22 +08:00
85b2c42f76 [Enhancement](jdbc catalog) Add a property to test the connection when creating a Jdbc catalog (#32125) (#32531) 2024-03-21 14:05:59 +08:00
fc2588c786 [fix](insert)fix sink user name (#32465) 2024-03-19 16:04:09 +08:00
93fe9521bf [feature](insert)fix implement hive table sink plan (#32430)
introduced by #32386
2024-03-19 09:55:16 +08:00
ecadb60bcd [Pick 2.1](inverted index) support inverted index format v2 (#30145) (#32418) 2024-03-19 08:11:33 +08:00
711c0cd55c [feature](insert)implement hive table sink plan (#31765) (#32386)
from #31765
2024-03-18 22:49:30 +08:00
a444e84be6 [feature](hive)add 'HmsCommiter' to support inserting data into hive table (#32283) (#32362)
bp #32283
Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-03-18 10:59:32 +08:00