database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
10,557 Commits 1 Branch 0 Tags
88ca4f3e6bc9558b447b1f0e0707547053eba05f
Commit Graph

3363 Commits

Author SHA1 Message Date
jakevin
098dac20c2 [log](Nereids): add more debug info when check logicalproperties. (#19763) 2023-05-18 08:37:10 +08:00
lvshaokang
c80c4477cf [Enhancement](broker-load) broker load show stmt support display cluster name if specified (#19392) 2023-05-18 00:10:15 +08:00
xueweizhang
97d4778ecf [enhancement](schema) dynamic_partition.time_unit support year (#19551) 
dynamic_partition.time_unit support year
 2023-05-17 23:49:15 +08:00
yongkang.zhong
8aa7f0e188 [fix](catalog) fix the include_database_list not in effect (#19589) 2023-05-17 22:56:21 +08:00
luozenglin
60d5c82f44 [fix](tvf) fix the inconsistency between tvf backends function and show backends result (#19697) 2023-05-17 22:55:46 +08:00
Jibing-Li
be47a27013 [Fix](multi catalog, nereids)Fix FileQueryScanNode couldn't filter partition in nereids planner bug (#19564) 
Nereids planner add conjuncts to ScanNode after call finalize, this may cause external table scan node fail to filter 
useless partition, because external table do the partition prune in the finalize method.

This pr is to fix this bug. In the rewrite stage, pass the conjuncts to LogicalFileScan object, and eventually pass to 
ScanNode while creating it. So that the ScanNode could use the conjuncts while doing finalize. 

Why not doing the partition prune in the LogicalFileScan like LogicalOlapScan doing? 

Because Iceberg api doesn't have the partition concept, it just accept a list of Conjuncts, 
so it's easier to pass the conjuncts to ScanNode (Hive, Icegerg, Hudi...) and doing the partition prune in there.
 2023-05-17 21:39:59 +08:00
zhangdong
b5f60bde42 [fix](checkpoint)fix Checkpoint error when http server is not ready #19699 2023-05-17 21:33:56 +08:00
slothever
2993cdb36e [fix](multi-catalog)fix iceberg catalog display type #19728 2023-05-17 21:33:18 +08:00
mch_ucchi
1d05feea1b [Feature](Nereids) add executable function to support fold constant for functions (#18209) 
1. Add date-time functions for fold constant for Nereids.
This is the list of executable date-time function nereids supports up to now:
- now()
- now(int)
- current_timestamp()
- current_timestamp(int)
- localtime()
- localtimestamp()
- curdate()
- current_date()
- curtime()
- current_time()
- date_{add/sub}(),{years/months/days/hours/minutes/seconds}_{add/sub}()
- datediff()
- {date/datev2}()
- {year/quarter/month/day/hour/minute/second}()
- dayof{year/month/week}()
- date_format()
- date_trunc()
- from_days()
- last_day()
- to_monday()
- from_unixtime()
- unix_timestamp()
- utc_timestamp()
- to_date()
- to_days()
- str_to_date()
- makedate()

2. solved problem:
- enable datev2/datetimev2 default.
- refactor Nereids foldConstantOnFE and support fold nested expression.
- separate the executable into multi-files for easily-reading and adding new functions
 2023-05-17 21:26:31 +08:00
Tiewei Fang
1eb929e1ca [Bugfix](Jdbc Catalog) fix data type mapping of SQLServer Catalog (#19525) 
We map `money/smallmoney` types of SQLSERVER into decimal type of doris.
 2023-05-17 21:02:42 +08:00
Ashin Gau
30c4f25cb3 [fix](multi-catalog) verify the precision of datetime types for each data source (#19544) 
Fix threes bugs of timestampv2 precision:
1. Hive catalog doesn't set the precision of timestampv2, and can't get the precision from hive metastore, so set the largest precision for timestampv2;
2. Jdbc catalog use datetimev1 to parse timestamp, and convert to timestampv2, so the precision is lost.
3. TVF doesn't use the precision from meta data of file format.
 2023-05-17 20:50:15 +08:00
Mingyu Chen
73be97f8d8 [fix](meta) fix upgrade failed on FE meta from 1.2 (#19674) 
Introduced from #19355.
We need to keep OP_CREATE_CLUSTER so that Doris can be upgraded from 1.2.x.
This OP type should be removed after 3.0
 2023-05-17 20:48:58 +08:00
LiBinfeng
131e77a816 [Fix](Nereids) fix minidump parameter name and double not a number serialize bug (#19635) 
Change nereids minidump switch from "Dump_nereids" to "enable_minidump" which is more exactly and neat. Also fix bug of Double.NaN (not a number) serialize bug when doing column statistic serialization.
 2023-05-17 20:16:50 +08:00
mch_ucchi
5d5db157d0 [Fix](planner) fix literal type incompatible after fold constant by be. (#19190) 2023-05-17 19:54:29 +08:00
Chengpeng Yan
05d47d43bd [Fix](Nereids) check the tableName in catalog (#19695) 
# Proposed changes
In the nereids. Before this PR: when we access some unexists tables. It will report the exception as follows:
```
mysql> select * from tt;
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: null
```

After this PR, it will get the following results:
```
mysql> select * from tt;
ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: Table [tt] does not exist in database [default_cluster:test].
```

## Problem summary
It is because in this [function](f5af07f7b2/fe/fe-core/src/main/java/org/apache/doris/nereids/CascadesContext.java (L328)), we ignore the exception. So the size of `tables` in `CascadesContext` is zero not null. So we can only get null after `table = cascadesContext.getTableByName(tableName);`.
 2023-05-17 19:48:30 +08:00
Zhang Wenxin
bee2e2964f [refactor](Nereids) refactor adjust nullable rule as a custom rewriter (#19702) 
use custom rewriter to do adjust nullable to avoid nullable changed in expression but not changed in output
 2023-05-17 19:24:42 +08:00
Kang
ce12cf404c [bugfix](inverted index) Fix mv inheriting unexpectedly inverted index of base table (#19722) 2023-05-17 17:18:07 +08:00
Zhang Wenxin
6ba2f681af [fix](Nereids) result error when do agg with distinct under pipeline (#19735) 2023-05-17 17:08:42 +08:00
Pxl
800de168db [Chore](function) clean some unused function symbols (#19649) 
clean some unused function symbols
 2023-05-17 15:31:51 +08:00
LiBinfeng
cc9d340400 [Fix](Nereids) Fix minidump connect context loading and concurrency bug (#19578) 
There are two problems of mini dump:
1、minidump do not load connect context to ThreadInfo, so it can not be get easily
2、minidump write maps with not concurrent protection, so the map size with change when we iterating map iterator

Solution:
1、loading connect context to minidump threading
2、use immutable map copy a new map before we actually doing iteration
 2023-05-17 15:09:00 +08:00
starocean999
3e661a30c2 [fix](planner)just return non-empty side of ExprSubstitutionMap if one of ExprSubstitutionMap is empty (#19600) 2023-05-17 15:06:43 +08:00
Zhang Wenxin
d9950a6422 [fix](Nereids) not fallback correctly when do forward (#19675) 2023-05-17 14:22:40 +08:00
yongjinhou
802e55114b http interfaces between FEs are not redirected (#19590) 2023-05-17 14:21:53 +08:00
jakevin
4607a3408e [minor](Nereids): unify name about Transpose. (#19662) 2023-05-17 11:33:02 +08:00
bigben0204
f95c1d7cb6 [feat](profile) Add a new rest api to query instance host and ip information for query profile action in branch master(#18668) (#19643) 2023-05-17 10:52:47 +08:00
Zhang Wenxin
c98147375d [fix](Nereids) decimal compare float should use double as common type (#19710) 2023-05-17 10:36:04 +08:00
Pxl
d784c99360 [Bug](planner) fix unassigned conjunct assigned on wrong node (#19672) 
* fix unassigned conjunct assigned on wrong node
 2023-05-17 10:28:22 +08:00
zhangdong
54507bb058 [fix](FQDN)fix Checkpoint error (#19678) 
Must use Env.getServingEnv() instead of getCurrentEnv(),because here we need to obtain selfNode through the official service catalog.
 2023-05-17 08:47:11 +08:00
Pxl
7f73749b88 [Bug](pipeline) fix distributionColumnIds not updated correct when outputColumnUnique… (#19704) 
fix distributionColumnIds not updated correct when outputColumnUnique
 2023-05-17 00:13:10 +08:00
yongkang.zhong
a1b1aff0ee [improvement](jdbc catalog) Adapt to hana's special view & Optimize jdbc name format (#19696) 2023-05-16 23:29:30 +08:00
Ziyu Wang
325a1d4b28 [vectorized](function) support array_count function (#18557) 
support array_count function.
array_count：Returns the number of non-zero and non-null elements in the given array.
 2023-05-16 17:00:01 +08:00
lihangyu
e22f5891d2 [WIP](row store) two phase opt read row store (#18654) 2023-05-16 13:21:58 +08:00
slothever
3f2d1ae9a4 [feature-wip](multi-catalog)(step1)support connect to max compute (#19606) 
Issue Number: #19679

support connect to max compute metadata by odps sdk
 2023-05-16 11:30:27 +08:00
Stalary
9cede6d763 [fix](row-policy) row policy supports external catalog (#19570) 
Row policy support external catalog
 2023-05-16 08:54:06 +08:00
Weijie Guo
9535ed01aa [feature](tvf) Support compress file for tvf hdfs() and s3() (#19530) 
We can support this by add a new properties for tvf, like :

`select * from hdfs("uri" = "xxx", ..., "compress_type" = "lz4", ...)`

User can:

Specify compression explicitly by setting `"compression" = "xxx"`.
Doris can infer the compression type by the suffix of file name(e.g. `file1.gz`)
Currently, we only support reading compress file in `csv` format, and on BE side, we already support.
All need to do is to analyze the `"compress_type"` on FE side and pass it to BE.
 2023-05-16 08:50:43 +08:00
Xiangyu Wang
8284c342cb [Fix](multi-catalog) Fix query hms tbl with compressed data files. (#19557) 
If a hms table's file format is csv, uncompressed data files may be coexists with compressed data files, so we need to set compressType separately.
 2023-05-16 08:49:45 +08:00
zhangdong
8ec18660fe [improvement](FQDN)Remove unused code (#19638) 2023-05-16 08:48:20 +08:00
luozenglin
6c9c9e9765 [feature-wip](resource-group) Supports memory hard isolation of resource group (#19526) 2023-05-15 22:45:46 +08:00
Yongqiang YANG
276e631e9c [chore](ddlExecutor) log class of unknown stmt in DdlExecutor (#19631) 
* [chore](ddlExecutor) log class of unknown stmt in DdlExecutor
 2023-05-15 21:59:44 +08:00
LiBinfeng
052c7cff89 [Fix](Planner) fix cast from decimal to boolean (#19585) 2023-05-15 15:13:16 +08:00
jakevin
69243b3a57 [fix](Nereids): SemiJoinLogicalJoinTranspose shouldn't throw error when eliminate outer failed. (#19566) 2023-05-15 12:31:54 +08:00
Pxl
4eb2604789 [Bug](function) fix function define of Retention inconsist and change some static_cast to assert cast (#19455) 
1. fix function define of `Retention` inconsist, this function return tinyint on `FE` and return uint8 on `BE`
2. make assert_cast support cast to derived
3. change some static cast to assert cast
4. support sum(bool)/avg(bool)
 2023-05-15 11:50:02 +08:00
Zhang Wenxin
5df5c77d39 [fix](Nereids) should not colocate agg when scan data partition is random (#19598) 2023-05-15 11:22:41 +08:00
Zhengguo Yang
6748ae4a57 [Feature] Collect the information statistics of the query hit (#18805) 
1. Show the query hit statistics for `baseall`

   ```sql
    MySQL [test_query_db]> show query stats from baseall;
    +-------+------------+-------------+
    | Field | QueryCount | FilterCount |
    +-------+------------+-------------+
    | k0    | 0          | 0           |
    | k1    | 0          | 0           |
    | k2    | 0          | 0           |
    | k3    | 0          | 0           |
    | k4    | 0          | 0           |
    | k5    | 0          | 0           |
    | k6    | 0          | 0           |
    | k10   | 0          | 0           |
    | k11   | 0          | 0           |
    | k7    | 0          | 0           |
    | k8    | 0          | 0           |
    | k9    | 0          | 0           |
    | k12   | 0          | 0           |
    | k13   | 0          | 0           |
    +-------+------------+-------------+
    14 rows in set (0.002 sec)

    MySQL [test_query_db]> select k0, k1,k2, sum(k3) from baseall  where k9 > 1 group by k0,k1,k2;
    +------+------+--------+-------------+
    | k0   | k1   | k2     | sum(`k3`)   |
    +------+------+--------+-------------+
    |    0 |    6 |  32767 |        3021 |
    |    1 |   12 |  32767 | -2147483647 |
    |    0 |    3 |   1989 |        1002 |
    |    0 |    7 | -32767 |        1002 |
    |    1 |    8 |    255 |  2147483647 |
    |    1 |    9 |   1991 | -2147483647 |
    |    1 |   11 |   1989 |       25699 |
    |    1 |   13 | -32767 |  2147483647 |
    |    1 |   14 |    255 |         103 |
    |    0 |    1 |   1989 |        1001 |
    |    0 |    2 |   1986 |        1001 |
    |    1 |   15 |   1992 |        3021 |
    +------+------+--------+-------------+
    12 rows in set (0.050 sec)

    MySQL [test_query_db]> show query stats from baseall;
    +-------+------------+-------------+
    | Field | QueryCount | FilterCount |
    +-------+------------+-------------+
    | k0    | 1          | 0           |
    | k1    | 1          | 0           |
    | k2    | 1          | 0           |
    | k3    | 1          | 0           |
    | k4    | 0          | 0           |
    | k5    | 0          | 0           |
    | k6    | 0          | 0           |
    | k10   | 0          | 0           |
    | k11   | 0          | 0           |
    | k7    | 0          | 0           |
    | k8    | 0          | 0           |
    | k9    | 1          | 1           |
    | k12   | 0          | 0           |
    | k13   | 0          | 0           |
    +-------+------------+-------------+
    14 rows in set (0.001 sec)
   ```

2. Show the query hit statistics summary for all the mv in a table

   ```sql
   MySQL [test_query_db]> show query stats from baseall all;
    +-----------+------------+
    | IndexName | QueryCount |
    +-----------+------------+
    | baseall   | 1          |
    +-----------+------------+
    1 row in set (0.005 sec)
   ```

3. Show the query hit statistics detail info for all the mv in a table

   ```sql
    MySQL [test_query_db]> show query stats from baseall all verbose;
    +-----------+-------+------------+-------------+
    | IndexName | Field | QueryCount | FilterCount |
    +-----------+-------+------------+-------------+
    | baseall   | k0    | 1          | 0           |
    |           | k1    | 1          | 0           |
    |           | k2    | 1          | 0           |
    |           | k3    | 1          | 0           |
    |           | k4    | 0          | 0           |
    |           | k5    | 0          | 0           |
    |           | k6    | 0          | 0           |
    |           | k10   | 0          | 0           |
    |           | k11   | 0          | 0           |
    |           | k7    | 0          | 0           |
    |           | k8    | 0          | 0           |
    |           | k9    | 1          | 1           |
    |           | k12   | 0          | 0           |
    |           | k13   | 0          | 0           |
    +-----------+-------+------------+-------------+
    14 rows in set (0.017 sec)
   ```

4. Show the query hit for a database

   ```sql
    MySQL [test_query_db]> show query stats for test_query_db;
    +----------------------------+------------+
    | TableName                  | QueryCount |
    +----------------------------+------------+
    | compaction_tbl             | 0          |
    | bigtable                   | 0          |
    | empty                      | 0          |
    | tempbaseall                | 0          |
    | test                       | 0          |
    | test_data_type             | 0          |
    | test_string_function_field | 0          |
    | baseall                    | 1          |
    | nullable                   | 0          |
    +----------------------------+------------+
    9 rows in set (0.005 sec)
   ```

5. Show query hit statistics for all the databases

   ```sql
    MySQL [(none)]> show query stats;
    +-----------------+------------+
    | Database        | QueryCount |
    +-----------------+------------+
    | test_query_db   | 1          |
    +-----------------+------------+
    1 rows in set (0.005 sec)
   ```
 2023-05-15 10:56:34 +08:00
nanfeng
0068828a94 [Feature](insert) support insert overwrite stmt (#19616) 2023-05-14 20:01:30 +08:00
Tiewei Fang
91cdb79d89 [Bugfix](Outfile) fix that export data to parquet and orc file format (#19436) 
1. support export `LARGEINT` data type to parquet/orc file format.
2. Export the DORIS `DATE/DATETIME` type to the `Date/Timestamp` logic type of parquet file format.
3. Fix that the data is not correct when the DATE type data is exported to ORC.
 2023-05-13 22:39:24 +08:00
LiBinfeng
316223ef34 [fix](planner) forbidden query in insert value list (#19493) 2023-05-12 19:46:19 +08:00
Mingyu Chen
4142cc0e8c [fix](merge conflict) fix FE compile error (#19586) 2023-05-12 18:18:22 +08:00
ElvinWei
c37d781942 [enchancement](statistics) manually inject table level statistics (#19495) 
supports users to manually inject table level statistics.

table stats type:
- row_count

Modify table or partition statistics:
```SQL
ALTER TABLE table_name SET STATS ('k1' = 'v1', ...) 
```

TODO：
- support other table stats type if necessary
- update statistics cache if necessary
 2023-05-12 17:03:12 +08:00
zhangdong
26a7f86b66 [improvement](auth)only GRANT_PRIV and USAGE_PRIV can GRANT for RESOURCE (#19547) 
only GRANT_PRIV and USAGE_PRIV can GRANT for RESOURCE
 2023-05-12 15:47:04 +08:00