database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
17,265 Commits 1 Branch 0 Tags
0aa7108ee252a39fb9407f9852ae20b7b243c8db
Commit Graph

2574 Commits

Author SHA1 Message Date
Xin Liao
0aa7108ee2 [fix](merge-on-write) incorrect result caused by key range filter with pk (#31456) 2024-02-29 19:51:47 +08:00
zzzxl
92e3b31f50 [feature](invert index) match_phrase_edge feature added (#31142) 2024-02-29 19:51:18 +08:00
feiniaofeiafei
686938f5db [fix](nereids) window function with grouping sets work not well (#31475) 
```sql        
select a, c, sum(sum(b)) over(partition by c order by c rows between unbounded preceding and current row) 
from test_window_table2 group by grouping sets((a),( c)) having a > 1 order by 1,2,3;
```
for this kind of case:
sum(sum(col)) over, nereids has cannot find slot problem.
the output slot of repeat and aggregate is computed wrongly.
Only collecting the trival-agg  in NormalizeRepeat can fix this problem.


Co-authored-by: feiniaofeiafei <moailing@selectdb.com>
 2024-02-29 16:44:40 +08:00
Guangdong Liu
9c4708ee74 [function](random_bytes)add random_bytes function (#31547) 
SELECT random_bytes(10);

random_bytes(10) |
----------------------+
0x9b8ea00b7d1084bc5b26|
 2024-02-29 16:44:39 +08:00
Tiewei Fang
4636b6195b [Fix](JNI) fix BE core when using JNI to query the empty map type value (#31502) 2024-02-29 14:03:38 +08:00
HappenLee
ff3c7765d6 [Bug](Func) Fix negative number in bitmap return error result (#31563) 2024-02-29 12:38:03 +08:00
Pxl
6af6997f1d [Improvement](materialized-view) add approx_count_distinct/ndv to FunctionAlias rule (#31535) 
add approx_count_distinct/ndv to FunctionAlias rule
 2024-02-29 12:38:03 +08:00
zy-kkk
3c37fb085c [refactor](jdbc catalog) split jdbc executor for different data sources (step-1) (#31406) 2024-02-29 12:38:03 +08:00
xzj7019
0fcdab468d [nereids] config global partition topn (#31476) 
* [nereids] config global partition topn

* [nereids] config global partition topn

---------

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
 2024-02-29 08:42:35 +08:00
谢健
4de25ede85 [fix](Nereids): other cond should be kept for each anti join when expanding anti join such as (#31521) 2024-02-29 08:42:35 +08:00
morrySnow
d88caca44a [fix](Nereids) push down topn distinct through join by mistake (#31396) 
should not push down topn distinct through join when the output
columns of the corresponding child of join is more than
aggregate distinct columns.

for example for LEFT_OUTER_JOIN:

left child of join's output is: c1, c2, c3.
distinct columns is: c1, c2
topn: limit 2

if we push down topn distinct, we could get result of join like this:

```
c1    c2    c3, ...
1     2     1
1     2     2
```

and the final result we get is:

```
c1    c2
1     2
```

this is wrong, because we need 2 lines, but only return 1.
 2024-02-28 17:51:32 +08:00
LiBinfeng
5824f8a4bc [Feat](nereids) support multi-leading (#30379) 
support multi-leading in each query block
 2024-02-28 13:07:47 +08:00
Guangdong Liu
47d112e4e6 [fix](MySQL) implement SHOW CHARSET statement. (#31389) 2024-02-28 13:07:23 +08:00
feiniaofeiafei
481d94c3fc [feature](nereids) deal the slots that appear both in agg func and grouping sets (#31318) 
this PR support slot appearing both in agg func and grouping sets.
sql like below:
select sum(a) from t group by grouping sets ((a)); 

Before this PR, Nereids throw exception like below:
col_int_undef_signed cannot both in select list and aggregate functions when using GROUPING SETS/CUBE/ROLLUP, please use union instead.

This PR removes the restriction and supports this situation.
 2024-02-27 10:12:33 +08:00
wangbo
1127b0065a [Improment](executor)Add scanbytes/scanrows condition (#31364) 
* Add scanbytes/scanrows condition

* fix reg
 2024-02-27 10:12:33 +08:00
Chester
f163d56a98 [feature](function) support sequence function(alias of array_range), enhance both to handle datetimev2 (#30823) 2024-02-27 10:12:19 +08:00
starocean999
5d4a2d93a6 [feature](nereids) support join with joinRelation (#30909) 2024-02-26 19:07:11 +08:00
Pxl
3acffaa205 [Feature](agg-state) support write_column_to_pb from DataTypeFixedLengthObjectSerDe (#31171) 2024-02-26 19:07:10 +08:00
Pxl
3b7261abe4 [Mv] check delete from column exists on mv (#31321) 2024-02-26 19:07:10 +08:00
zhiqiang
3451cd6c23 [fix](datetime) fix hour 24 on be (#31304) 2024-02-26 19:07:10 +08:00
zhangdong
7a9fe5d275 [enhance](mtmv)MTMV supports Hive multi-level partitioning (#31060) 
Issue Number: close #xxx

For example, the hive table is partitioned by `date` and `region`, with the following 6 partitions
```
20200101
        beijing
        shanghai
20200102
        beijing
        shanghai
20200103
        beijing
        shanghai
```

If the MTMV is partitioned by `date`, then the MTMV will have three partitions: 20200101, 202000102, 20200103

If the MTMV is partitioned by `region`, then the MTMV will have two partitions: beijing, shanghai
 2024-02-25 18:08:19 +08:00
kkop
93cbc96141 [regression test]Opt Test UniqueModel Schema Key Change (#31262) 
* Opt Test UniqueModel Schema Key Change

* Opt Test UniqueModel Schema key Change
 2024-02-24 16:26:44 +08:00
Kang
db58104bc3 [fix](inverted index) Fix inverted index for MOR unique table #31051 (#31354) 2024-02-23 23:10:36 +08:00
kkop
3576304e3e [regression test]Test UniqueModel Schema Value Change (#31260) 
* Test UniqueModel Schema Value Change

* Opt Test UniqueModel Schema value Change
 2024-02-23 19:03:28 +08:00
yangshijie
8f77e6363a [Feature](function) Support xxhash function like murmur hash function (#31193) 2024-02-23 19:03:28 +08:00
Jerry Hu
1456785aa1 [fix](join) incorrect result of mark join in nested loop join (#31280) 2024-02-23 19:03:28 +08:00
amory
f65876d803 [Feature](explode) support explode map type (#30151) 2024-02-22 20:08:44 +08:00
morrySnow
22efb1cb7a [fix](Nereids) not equals and hashCode should contains generate flag (#31123) 2024-02-22 19:51:20 +08:00
Ashin Gau
260568db17 [update](hudi) update hudi version to 0.14.1 and compatible with flink hive catalog (#31181) 
1. Update hudi version from 0.13.1 to .14.1
2. Compatible with the hudi table created by flink hive catalog
 2024-02-22 19:51:20 +08:00
morrySnow
98c3cb825f [fix](Nereids) simplify airthmetic should not change return type (#31237) 2024-02-22 19:50:47 +08:00
meiyi
ef22bd3318 [fix](txn insert) Txn insert can not write to table with mv (#31167) 2024-02-22 13:01:49 +08:00
amory
ad07dec0ed [Improve](InPredict) enhance in predict with struct type (#30840) 2024-02-22 13:01:49 +08:00
kkop
8a496bf554 [regression test]Optimize testUniqueModelSchemaKeyChange (#31150) 2024-02-22 13:01:48 +08:00
meiyi
ca89ac0c23 [fix](regression) insert_group_commit_into_unique is unstable when cluster has multi BE (#31188) 2024-02-21 19:18:45 +08:00
zhiqiang
49dd411f87 [fix](datetime) fix datetime round on BE (#31205) 
with tmp as (
            select CONCAT(
                YEAR('2024-02-06 03:37:07.157'), '-', 
                LPAD(MONTH('2024-02-06 03:37:07.157'), 2, '0'), '-',
                LPAD(DAY('2024-02-06 03:37:07.157'), 2, '0'), ' ',
                LPAD(HOUR('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(MINUTE('2024-02-06 03:37:07.157'), 2, '0'), ':',
                LPAD(SECOND('2024-02-06 03:37:07.157'), 2, '0'), '.', "123456789" )
            AS generated_string)
            select generated_string, cast(generated_string as DateTime(6)) from tmp
before (incorrect round)

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123456              |
+-------------------------------+-----------------------------------------+
after (round up, keep consistent with mysql):

+-------------------------------+-----------------------------------------+
| generated_string              | cast(generated_string as DATETIMEV2(6)) |
+-------------------------------+-----------------------------------------+
| 2024-02-06 03:37:07.123456789 | 2024-02-06 03:37:07.123457              |
+-------------------------------+-----------------------------------------+
1 row in set (0.03 sec)
same work with #30744 but implemented on BE
 2024-02-21 19:18:45 +08:00
starocean999
eb5e6a829a [fix](nereids)should normalize window expression by bottom project's output (#31115) 2024-02-21 19:18:45 +08:00
Kang
8d889e434b [fix](topn) Fix key topn block reverse is missed in some cases (#31199) 
* move reverse block row order operation after _next_batch_internal

* add testcase
 2024-02-21 19:18:45 +08:00
morrySnow
8fc9d80479 [compatibility](MySQL) update charset to utf8mb4, collation to utf8mb4_0900_bin (#31046) 
Doris's behaviour is more like utf8mb4 and utf8mb4_0900_bin than utf8 and utf8_general_ci
 2024-02-21 17:01:39 +08:00
minghong
f1fbfeba2f [fix](nereids) removeRuntimeFilter() removes more than one RFs if there are different types of RF from the buildNode to the target (#31197) 
this bug can be produced on tpcds 95, when set runtime filter type: min-max + bloom
 2024-02-21 17:01:39 +08:00
starocean999
98106ad60f [fix](nereids) fix bug of unnest subuqery with having clause (#31152) 
1. run PushDownFilterThroughAggregation, PushDownFilterThroughProject and MergeFilters before subquery unnesting
2. should keep plan unchanged even left semi join's condition is true (because the right table may be empty())
3. PushDownFilterThroughProject need check if filter's input slots are all coming from project's output
 2024-02-21 17:01:30 +08:00
Mingyu Chen
a8d8c6a271 [fix](file-writer) opt s3 file writer and fix empty file related issue #28983 #30703 #31169 (#31213) 
* (feature)(cloud) Use dynamic allocator instead of static buffer pool for better elasticity. (#28983)

* [fix](outfile) Fix unable to export empty data (#30703)

Issue Number: close #30600
Fix unable to export empty data to hdfs / S3, this behavior is inconsistent with version 1.2.7,
version 1.2.7 can export empty data to hdfs/ S3, and there will be exported files on S3/HDFS.

* [fix](file-writer) avoid empty file for segment writer (#31169)

---------

Co-authored-by: AlexYue <yj976240184@gmail.com>
Co-authored-by: zxealous <zhouchangyue@baidu.com>
 2024-02-21 16:48:54 +08:00
lihangyu
278b232e76 [Bug](json reader) object should stop processing when encounter error (#31159) 
If DATA_QUALITY_ERROR encountered we should stop processing this document any more.Otherwise there will be UB in simdjson.
 2024-02-21 13:53:32 +08:00
minghong
cd7230885f [fix](nereids)push more than one runtime filters into cte (#30901) 
* push rf into cte, used by tpcds95
 2024-02-21 13:53:32 +08:00
huanghaibin
1e3968fe7e [fix](group_commit) Need to wait wal to be deleted when creating MaterializedView (#30956) 2024-02-21 13:53:19 +08:00
meiyi
1d6dc9a5f0 [fix](catalog recycle bin) Forbid recover partition if table schame is changed #31146 2024-02-21 13:53:18 +08:00
wangbo
97c9d75af3 [Feature](executor)Add scan_thread_num property for workload group (#31106) 2024-02-20 16:24:05 +08:00
Pxl
493385c2c7 [Bug](AggState) fix not match function when agg combinator function has alias (#31101) 2024-02-20 16:23:53 +08:00
Hong Liu
a7a530189e [regression-test](decimalv3) Adding numerical precision exceeding the data type will error case (#31068) 
* Adding numerical precision exceeding the data type will result in an error case

* add order by

---------

Co-authored-by: smallhibiscus <844981280>
 2024-02-20 16:23:53 +08:00
starocean999
180fc13f6b [fix](nereids) disable PushDownJoinOtherCondition rule for mark join (#31084) 2024-02-19 17:48:29 +08:00
zy-kkk
b3ac2128dd [Refactor](catalog) Refactor Jdbc Catalog external name case mapping rules (#28414) 2024-02-19 17:22:03 +08:00