database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
11,475 Commits 1 Branch 0 Tags
c19e35116bbd2223efd57f02b612ebea08bb3667
Commit Graph

1927 Commits

Author SHA1 Message Date
lihangyu
50c1d55769 [Improve](dynamic schema) support filtering invalid data (#21160) 
* [Improve](dynamic schema) support filtering invalid data

1. Support dynamic schema to filter illegal data.
2. Expand the regular expression for ColumnName to support more column names.
3. Be compatible with PropertyAnalyzer and support legacy tables.
4. Default disable parse multi dimenssion array, since some bug unresolved
 2023-06-26 19:32:43 +08:00
starocean999
102b7f8873 remove useless case (#21166) 2023-06-26 16:27:32 +08:00
Armando Zhu
5d2b69b06d [Enhancement](regression) let test case fail fast when job is cancelled (#20578) (#21103) 
In doris regression-test/suites, a lot of test cases quit immediately only if "FINISHED", otherwise they will wait till timeout. For example:

while (max_try_secs--) {
        String res = getJobState(tbName1)
        if (res == "FINISHED") {
            sleep(3000)
            break
        } else {
            Thread.sleep(1000)
            if (max_try_secs < 1) {
                println "test timeout," + "state:" + res
                assertEquals("FINISHED", res)
            }
        }
   }
This PR added checks so that these test cases can quit immediately also if "CANCELLED", which is the only unchanging status other than "FINISHED".
 2023-06-26 12:58:51 +08:00
ZhangYu0123
66005570c9 [fix](regression) fix p1 test_backup_restore fail caused by http download 401 invalid token error #21107 2023-06-26 12:56:46 +08:00
TengJianPing
baf9a2107b [fix](regression) fix case failure by adding sync after stream load (#21155) 2023-06-26 10:38:46 +08:00
starocean999
58b3e5ebdb [fix](nereids)scan node's smap should use materiazlied slots and project list as left and right expr list (#21142) 2023-06-25 22:34:43 +08:00
airborne12
1ac8cdec7e [Fix](inverted index) fix inverted query cache for chinese tokenizer (#21106) 
1. query cache for chinese tokenizer is confusing when just converting w_char to char.
2. seperate query_type from inverted_index_reader to clean code.
 2023-06-25 22:04:02 +08:00
minghong
2d1163c4d8 [refactor](nereids) update Agg stats derive method #21036 
This pr has no effect on tpch queries.
Some tpcds queries are impacted.
They are 4/11/23/24/47/51/57/65/74, in which 4 and 51 are improved
 2023-06-25 21:47:32 +08:00
AKIRA
638aa41988 [fix](planner) fix push filter through agg #21080 
In the previous implementation, the check for groupby exprs was ignored. Add this necessary check to make sure it would work

You could reproduce it by runnning belowing sql:

CREATE TABLE t_push_filter_through_agg (col1 varchar(11451) not null, col2 int not null, col3 int not null)
UNIQUE KEY(col1)
DISTRIBUTED BY HASH(col1)
BUCKETS 3
PROPERTIES(
    "replication_num"="1"
);

CREATE VIEW `view_i` AS 
SELECT 
    `b`.`col1` AS `col1`, 
    `b`.`col2` AS `col2`
FROM 
(
    SELECT 
        `col1` AS `col1`, 
        sum(`cost`) AS `col2`
    FROM 
    (
        SELECT 
            `col1` AS `col1`, 
            sum(CAST(`col3` AS INT)) AS `cost` 
        FROM 
            `t_push_filter_through_agg` 
        GROUP BY 
            `col1`
    ) a 
    GROUP BY 
        `col1`
) b;

SELECT SUM(`total_cost`) FROM view_a WHERE `dt` BETWEEN '2023-06-12' AND '2023-06-18' LIMIT 1;
 2023-06-25 19:14:20 +08:00
shuke
55e7af1e31 [fix](test) fix two case bug #21124 2023-06-25 18:53:20 +08:00
Siyang Tang
46f0295b78 [feature](load-refactor-with-tvf) S3 load with S3 tvf and native insert (#19937) 2023-06-25 17:45:31 +08:00
Mryange
6896776034 [test](regression) update some case in p2 (#21094) 
update some case in p2
 2023-06-25 11:16:56 +08:00
shuke
6a2226867c [fix](test) load case bug in tpch_sf10_unique_p2 #21093 2023-06-22 15:19:46 +08:00
shuke
d5455dd04b [fix](regression-test) test case variable scope bug (#21071) 2023-06-22 09:07:44 +08:00
starocean999
8b561cfb03 [fix](nereids)create datev2 and datetimev2 literal if enable_date_conversion is true (#21065) 2023-06-21 20:29:36 +08:00
airborne12
6ac0bfeceb [Feature](inverted index) add unicode parser for inverted index (#21035) 2023-06-21 20:14:06 +08:00
zhannngchen
cc53391c9a Revert "[feature](merge-on-write) enable merge on write by default (#… (#21041) 2023-06-21 18:36:46 +08:00
HHoflittlefish777
2beed11256 [Bug](streamload) fix inconsistent load result of be and fe (#20950) 2023-06-21 18:12:51 +08:00
Mryange
8bcd42d3f6 [test](regression) update some case in brown_p2 #21037 2023-06-21 16:25:07 +08:00
shuke
4766e75d4e [test](regression) add analyze after stream load (#21066) 2023-06-21 16:03:46 +08:00
DeadlineFen
4d84cd8ca1 Revert "Revert "[Test](regression) CCR syncer thrift interface regression test (#20935)" (#20990)" (#21022) 
This reverts commit 2a294801f1324a999570158eea3224239eefbb29.
 2023-06-21 15:20:21 +08:00
Qi Chen
bad22dd4e2 [Fix](orc-reader) Fix orc dict filter null value issue in _convert_dict_cols_to_string_cols which caused incorrect result. (#21047) 
Query results should not have empty values.
```
use regresssion.multi_catalog;
select commit_id from github_events_orc WHERE (event_type = 'CommitCommentEvent') AND commit_id != "" limit 10;
```
```
+------------------------------------------+
| commit_id                                |
+------------------------------------------+
| 685c1fd8dbbdc10c042932f9a9f88be00ff96c75 |
| 685c1fd8dbbdc10c042932f9a9f88be00ff96c75 |
| 4e3ab2ff2d2474f5d51334b9b0fdf17e9845a166 |
|                                          |
|                                          |
|                                          |
|                                          |
|                                          |
|                                          |
| 7191c20cb49da07a7fc16aa32dc0de4faff528b2 |
+------------------------------------------+
10 rows in set (0.54 sec) 
```
 2023-06-21 14:54:01 +08:00
lihangyu
62fb0e642e [chore](dynamic schema) deprecated create dynamic schema table (#21058) 2023-06-21 14:44:57 +08:00
Pxl
5f0bb49d46 [Feature](materialized-view) support create mv contain aggstate column (#20812) 
support create mv contain aggstate column
 2023-06-21 13:06:52 +08:00
amory
18beb822a3 [FIX](array-type) fix array string output with fe const expr (#21042) 
fe foldconstRule make array() function expr with const literal , and would not pass this array literal to be . but we should make fe array string output format is same with be array string output
 2023-06-21 11:52:02 +08:00
Pxl
b4773e1195 [Chore](materialized-view) enable nereids planner on regression test mv_p0 (#21023) 
enable nereids planner on regression test mv_p0
 2023-06-21 10:01:27 +08:00
dujl
0cf9de8cef [fix](decimalv3) fix result error when cast a round decimalv3 to double (#20678) 2023-06-21 00:02:48 +08:00
Kang
2c11ce0a02 [bugfix](topn) fix key topn merge block conflict with index predicate result columns (#20820) 2023-06-20 21:23:00 +08:00
LiBinfeng
f10258577b [Fix](Planner) Fix group concat with multi distinct and segs (#20912) 
Problem:
when use select group_concat(distinct a, 'seg1'), group_concat(distinct b, 'seg2') ... Error would rised
Reason:
Group_concat function regard 'seg' as arguments also, so multi distinct column error would rised
Solved:
let Multi Distinct group_concat function only get first argument as real argument
 2023-06-20 21:00:18 +08:00
TengJianPing
55a6649da9 [fix](testcase) fix test case failure of insert null value into not null column (#20963) 2023-06-20 20:46:07 +08:00
zy-kkk
7e01f074e2 [improvement](jdbc mysql) support auto calculate the precision of timestamp/datetime (#20788) 2023-06-20 10:39:34 +08:00
zzzzzzzs
824bc02603 [Function] Support date function: microsecond() (#20044) 2023-06-20 10:32:54 +08:00
jakevin
d02ecef406 [fix](Nereids): revert push down alias into union (#20991) 
revert #20543 to tmp avoid problem
 2023-06-20 09:32:26 +08:00
Mryange
5a28b6f9fc [fix](datetime) Fix the error in date calculation that includes constants (#20863) 
before

```
mysql> select hours_add('2023-03-30 22:23:45.23452',8);
+-------------------------------------+
| hours_add('2023-03-30 22:23:45', 8) |
+-------------------------------------+
| 2023-03-31 06:23:45                 |
+-------------------------------------+

mysql> select date_add('2023-03-30 22:23:45.23452',8);
+------------------------------------+
| date_add('2023-03-30 22:23:45', 8) |
+------------------------------------+
| 2023-04-07 22:23:45                |
+------------------------------------+

mysql [test]>select hours_add('2023-03-30 22:23:45.23452',8);
+-------------------------------------------+
| hours_add('2023-03-30 22:23:45.23452', 8) |
+-------------------------------------------+
| 2023-03-31 06:23:45.000234                |
+-------------------------------------------+
```

after

```
mysql [test]>select hours_add('2023-03-30 22:23:45.23452',8);
+-------------------------------------------+
| hours_add('2023-03-30 22:23:45.23452', 8) |
+-------------------------------------------+
| 2023-03-31 06:23:45.23452                 |
+-------------------------------------------+
1 row in set (0.01 sec)

mysql [test]>select date_add('2023-03-30 22:23:45.23452',8);
+------------------------------------------+
| date_add('2023-03-30 22:23:45.23452', 8) |
+------------------------------------------+
| 2023-04-07 22:23:45.23452                |
+------------------------------------------+
1 row in set (0.00 sec)

mysql [test]>set enable_nereids_planner=true;
Query OK, 0 rows affected (0.00 sec)

mysql [test]>set enable_fallback_to_original_planner=false;
Query OK, 0 rows affected (0.00 sec)

mysql [test]>select hours_add('2023-03-30 22:23:45.23452',8);
+-------------------------------------------+
| hours_add('2023-03-30 22:23:45.23452', 8) |
+-------------------------------------------+
| 2023-03-31 06:23:45.23452                 |
+-------------------------------------------+
1 row in set (0.03 sec)

mysql [test]>select date_add('2023-03-30 22:23:45.23452',8);
+------------------------------------------+
| days_add('2023-03-30 22:23:45.23452', 8) |
+------------------------------------------+
| 2023-04-07 22:23:45.23452                |
+------------------------------------------+
1 row in set (0.00 sec)
```
 2023-06-19 23:44:30 +08:00
starocean999
e6f50c04f1 [fix](nereids)SubqueryToApply rule lost is null condition (#20971) 
* [fix](nereids)SubqueryToApply rule lost is null condition
 2023-06-19 23:43:40 +08:00
minghong
f20ef165fe [opt](Nereids) update join stats derive (#20895) 
in hash join condition, some equals are trustable, some are not.
an equal is trustable if one side is almost unique, like primary key. for such equal condition we could estimate more accurate.
the problem is in rewriten q20, the are 2 equal condition, one is trustable, another is not. But we treat both of them as trustable.

Test result:
on tpch100, from 2.2 sec to 0.44 sec
no impact on tpch other queries
no performance impact on tpcds queries
 2023-06-19 23:40:44 +08:00
DeadlineFen
2a294801f1 Revert "[Test](regression) CCR syncer thrift interface regression test (#20935)" (#20990) 
This reverts commit dd482b74c849b022862e7cfb1f1d0b933a84e3d2.
 2023-06-19 21:38:03 +08:00
Yongqiang YANG
dd5ecea36a [fix](compress) snappy does not work right (#20934) 2023-06-19 14:11:10 +08:00
yongjinhou
26cca5e00a [Enhancement](tvf) Add frontends table-valued-function (#20857) 2023-06-19 13:57:40 +08:00
TengJianPing
fb9fcf460a [fix](leftjoin) fix bug of left and full join with other conjuncts (#20946) 
Fix bug of left and full outer join with other conjuncts. When equal matched row count of a probe row exceed batch_size, some times the _join_node->_is_any_probe_match_row_output flag is not set correcty, which result in outputing extra rows for the probe row.
 2023-06-19 12:27:06 +08:00
Pxl
85c5d7c6a9 [Chore](materialized-view) add ssb_flat mv test case (#20869) 
add ssb_flat mv test case
 2023-06-19 10:51:50 +08:00
Zhiyu Hu
1efd345963 [Enhancement](table) adding information_schema.parameters table (#20259) 
this is a virtual table for compatibility information_schema parameters table
 2023-06-19 09:05:46 +08:00
Siyang Tang
8366ce7a81 [enhancement](insert-stmt) Make insert into tbl values(); compatible with mysql (#20694) 2023-06-18 19:56:07 +08:00
YueW
d6b7640cf0 [fix](inverted index) fix check failed for block erase temp column (#20924) 2023-06-18 19:27:48 +08:00
jakevin
ac3290021d [fix](Nereids): MergeSetOperations can merge SetOperation ALL. (#20902) 2023-06-18 17:49:03 +08:00
mch_ucchi
5ae14549d1 [Feature](Nereids) support delete using syntax to delete data from unique key table (#20452) 2023-06-18 16:22:21 +08:00
DeadlineFen
dd482b74c8 [Test](regression) CCR syncer thrift interface regression test (#20935) 2023-06-18 00:13:09 +08:00
shuke
72d8e41760 [fix]( regression-test ) mv_p0/test_drop_partition_from_index/test_drop_partition_from_index.groovy #20915 2023-06-17 10:39:30 +08:00
TengJianPing
82af1507e7 [fix](testcase) fix case failure of strict insert (#20930) 2023-06-17 09:41:15 +08:00
zy-kkk
fe18cfa2fb [improvement](pg jdbc)Support for automatically obtaining the precision of the postgresql timestamp type (#20909) 2023-06-16 23:41:09 +08:00