Commit Graph

8276 Commits

Author SHA1 Message Date
c18bfdc93e [test][regression cases][external]add external table p2 regression cases according doris1.2 docs 20230105 (#15651) 2023-01-06 20:19:39 +08:00
cad47dd9d9 [test](Nereids) add two regression test cases for Nereids (#15598)
1. test predicates infer could work well with push down predicates through join
2. test count with subquery containing constant literal
2023-01-06 16:29:50 +08:00
53559e2bdc [fix](decimalv2) fix loss of precision when cast to decimalv2 literal (#15629) 2023-01-06 16:02:46 +08:00
9c36278c4a [improvement](pipeline) Support sharing hash table for broadcast join (#15628) 2023-01-06 15:11:28 +08:00
1038093c29 [Pipeline](Exec) disable work steal of hash join build (#15652) 2023-01-06 15:08:10 +08:00
f24659c003 [Refactor](pipeline) refactor the code of channel buffer limit and change the default value (#15650) 2023-01-06 14:52:43 +08:00
7f84db310a [fix](nereids) Convert to datetime when binary expr's left is date and right is int type (#15615)
In the below case, expression ` date > 20200101` should implicit cast date both side to datetime instead of bigint

```sql
        CREATE TABLE `part_by_date`
        (
            `date`                  date   NOT NULL COMMENT '',
            `id`                      int(11) NOT NULL COMMENT ''
        ) ENGINE=OLAP
        UNIQUE KEY(`date`, `id`)
        PARTITION BY RANGE(`date`) 
        (PARTITION p201912 VALUES [('0000-01-01'), ('2020-01-01')),
        PARTITION p202001 VALUES [('2020-01-01'), ('2020-02-01')))
        DISTRIBUTED BY HASH(`id`) BUCKETS 3
        PROPERTIES (
        "replication_allocation" = "tag.location.default: 1"
        );

        INSERT INTO  part_by_date VALUES('0001-02-01', 1),('2020-01-15', 2);

        SELECT
            id
        FROM
           part_by_date
        WHERE date > 20200101;
```
2023-01-06 14:08:05 +08:00
ae77b582f0 [fix](Nereids) add information function and fix bugs in schemaScan (#15608)
1. Add information function
- Database()
- User()
- Current_User()
- Connection_id()

2. Fix bugs in schemaScan
2023-01-06 13:37:27 +08:00
ef72b8d859 [Feature](Nereids): add logical operator || && (#15643) 2023-01-06 12:18:21 +08:00
df2da89b89 [feature](multi-catalog) support postgresql jdbc catalog (#15570)
support postgresql jdbc catalog
2023-01-06 11:00:59 +08:00
b57500d0c3 [Bug](decimalv3) fix wrong result for MOD operation (#15644) 2023-01-06 10:38:53 +08:00
05d72e8919 [fix](join) fix anti join incorrectly outputs null values (#15567) 2023-01-06 09:55:48 +08:00
b41934864e [enhancement](frontendservice) add retry when create connection to frontend service (#15635) 2023-01-06 09:15:08 +08:00
95f2f43c02 [fix](macOS) Failed to run BE UT due to syscall to map cache into shared region failed (#15641)
According to the post https://developer.apple.com/forums/thread/676684, the executable whose size is bigger than 2G may fail to start. The size of the executable `doris_be_test` generated by run-be-ut.sh is 2.1G (> 2G) now and we can't run it on macOS (arm64).

We can separate the debug info from the executable `doris_be_test` to reduce the size. After that, we can run `doris_be_test` successfully.
2023-01-06 01:23:37 +08:00
6d691edcc7 [fix](Nereids): restrict join reorder project. (#15645) 2023-01-06 00:18:05 +08:00
77ffafb766 [vulnerability](CVE-2022-1292) fix CVE-2022-1292 (#15639) 2023-01-05 21:57:16 +08:00
9d1f02c580 [Improvement](topn) runtime prune for topn query (#15558) 2023-01-05 20:10:12 +08:00
d36b93708c [feature](Nereids): add ExtractFilterFromJoin rule to support more (#14896) 2023-01-05 19:09:43 +08:00
5460c873e8 [Feature] (Nereids) support un equals conjuncts in un scalar sub query (#15591)
support un equals conjuncts in un scalar sub query.
[fix] in correlated subquery wrong result
2023-01-05 16:56:14 +08:00
5ee479f45c [Pipeline](load) Support transaction on pipeline engine (#15597) 2023-01-05 15:59:18 +08:00
6523b546ab [chore](vulnerability) fix some high risk vulnerabilities report by bug scanner (#15621)
* [chore](vulnerability) fix some high risk vulnerabilities report by bug scanner
2023-01-05 14:58:23 +08:00
0dfa143140 [enhancement](Nereids) generate colocate join when property is different with require property (#15479)
1. When checking HashProperty which's type is nature, we only need to check whether the required properties contain all shuffle column
2. In ChildrenPropertiesRegulator.java, when colocate/buckte join is not allowed, we will enforce the required property.
2023-01-05 11:41:18 +08:00
4f2a36f032 [project] update year in NOTICE.txt (#15632)
* [project] update year in NOTICE.txt
2023-01-05 10:22:34 +08:00
1018657d9d [Enhancement](SparkLoad): avoid BE OOM in push task, fix #15572 (#15620)
Release memory pool held by the parquet reader when the data has been flushed by rowset writter.
Co-authored-by: spaces-x <weixiang06@meituan.com>
2023-01-05 10:20:32 +08:00
59f34be41f [fix](having-clause) having clause do not works correct with same alias name (#15143) 2023-01-05 10:15:15 +08:00
Pxl
93f5e440eb [Bug](execute) fix get next non stop for eos on streaming preagg (#15611)
* fix get nnext non stop for eos on streaming preagg

* update
2023-01-05 09:36:11 +08:00
5ff5b8fc98 [feature](mark join) Support mark join for hash join node (#15569)
* [feature](mark join) Support mark join for hash join node
2023-01-05 09:32:26 +08:00
61d538c713 [improvement](storage-policy) Add check validity when create storage policy. (#14405) 2023-01-04 22:24:49 +08:00
e67ea1ddb7 [fix](doc): catalog use resource doc error (#15607) 2023-01-04 19:53:25 +08:00
804ea08825 [add_log](checksum) add cost and size log for checksum (#15599) 2023-01-04 19:10:40 +08:00
4075e3aec6 [fix](csv-reader) fix new csv reader's performance issue (#15581) 2023-01-04 18:25:08 +08:00
7ef3940809 [fix](storage-policy) fix some bug (#15585)
1. fix datetime ms transfer to s bug
2. fix alter storage policy notify be missing field(datetime, ttl)
3. support alter storage policy use "h, hour, d, day" as ttl filed
2023-01-04 16:49:51 +08:00
c42c61dcad [fix](bitmapfilter) fix bitmap filter not pushing down (#15532) 2023-01-04 14:33:53 +08:00
a4af1fbf90 [fix](inbitmap) forbid having clause to include in bitmap. (#15494) 2023-01-04 14:33:18 +08:00
wxy
e0c56bcd20 [Feature](export) Support cancel export statement (#15128)
Co-authored-by: wangxiangyu@360shuke.com <wangxiangyu@360shuke.com>
2023-01-04 14:08:25 +08:00
73d4070708 [bugfix](compaction) fix missing key_bounds in vertical compaction (#15578)
When flush last segment of every column, missing set segment key bound
for rowset meta so that rowset tree init error.
2023-01-04 13:39:09 +08:00
36e43c2677 fix 1.2.1 release notes (#15590) 2023-01-04 13:26:54 +08:00
7728794b4a [fix](Nereids) SimplifyArithmeticRule generate wrong expression after process (#15580)
in the case of 'a / b', if a is constant, after apple SimplifyArithmeticRule, expression will be convert to 'b * a' by mistake.
2023-01-04 11:10:15 +08:00
f2f06c1acc [feature](nereids) Support select temp partition (#15579)
Support such grammer:
    select * from t_p temporary partition(tp1);
    select * from t_p temporary partitions(tp1);
    select * from t_p temporary partition tp1;
2023-01-04 11:04:36 +08:00
e5397efb67 [docs](releasenotes)release 1.2.1 (#15583)
* release 1.2.1
2023-01-04 10:12:46 +08:00
eef1f432dd [Bug](datetimev2/decimalv3) Fix wrong predicate infer rule (#15574) 2023-01-04 10:03:43 +08:00
a97f582b93 [fix](nereids) use DAYS as default unit for DATE_ADD and DATE_SUB function (#15559) 2023-01-04 01:55:15 +08:00
18bc354c06 [fix](Nereids) use correct column unique id when read data from non-base index (#15534)
When light schema change is enabled by default, a column in OLAP scan is retrieved by column unique id instead of the column name. Columns with the same name would use different unique IDs among materialized indexes.
This PR ensures that the column in the OLAP scan node could use the correct column unique id.
2023-01-04 01:41:25 +08:00
51a042eab3 [doc](yaml) update the label (#15575) 2023-01-04 00:55:19 +08:00
1f07e54178 [typo](docs)fix logstash readme url (#15573) 2023-01-03 22:57:36 +08:00
8d0c06c897 [fix](nereids) binding priority in agg-sort, having, group_by_key (#15240)
This PR defines order_key and having_key binding priority.

1. order key priority
 ```
                select
                        col1 * -1 as col1    # inner_col1 * -1 as alias_col1
                from
                        t
                order by col1;     # order by order_col1
```
to bind `order_col1`, `alias_col1` has higher priority than `inner_col1`

2. having key priority
```
       select (a-1) as a  # inner_a - 1 as alias_a
       from bind_priority_tbl 
       group by a 
       having a=1;
```
to bind having key, `inner_a` has higher priority than `alias_a`

3. group by key binding priority
```
SELECT date_format(b.k10,
         '%Y%m%d') AS k10
FROM test a
LEFT JOIN 
    (SELECT k10
    FROM baseall) b
    ON a.k10 = b.k10
GROUP BY  k10;
```
group_by_key (k10) binding priority:

- agg.child.output
- agg.output
if binding with agg.child.output failed(the slot not found, or more than one candidate slot found in agg.child.output), nereids try to bind group_by_key with agg.output.
In above example, nereids found 2 candidate slots (a.k10, b.k10) in agg.child.output for group_by_key (k10), binding with agg.child.output failed. Then nereids try to bind group_by_key with agg.output, that is `date_format(b.k10, '%Y%m%d') AS k10`. and finally, group_by_key is bound with `alias k10`
2023-01-03 22:09:28 +08:00
55dc541c90 [Fix](Nereids) aggregate function except COUNT should nullable without group by expr (#15547)
Co-authored-by: mch_ucchi
2023-01-03 21:28:07 +08:00
Pxl
85fe9d2496 [Bug](filter) fix not in(null) return true (#15466)
fix not in(null) return true
2023-01-03 21:14:50 +08:00
4380f1ec54 [Enhancement](load) reduce memory by memory size of global delta writer (#14491) 2023-01-03 20:05:21 +08:00
a365486a25 [fix](Nereids) get datatype for binary arithmetic (#15548)
it is just a temporary fix for binary arithmetic. Next we will refactor the TypeCoercion rule to make the behavior exactly same with Lagecy planner.
2023-01-03 19:09:48 +08:00