Commit Graph

1060 Commits

Author SHA1 Message Date
3ec52dc7da [tpch](nereids) add regression test for tpch_sf500 plan shape #18631
add regression test to check tpch_sf500 plan shape by explain shape plan.
2023-04-16 11:37:33 +08:00
98b8bef05b [bugfix](inverted index) fix inverted index to support NULL value filter (#18302) 2023-04-15 13:20:26 +08:00
f7e129934e [fix](nereids) only order by slot reference could use topn opt (#18622)
select cast(k1 as INT) as id from tbl1 order by id limit 2; 

is not valid for topN optimization, because 'id' is
a cast expr not a table column from scan node.
This pr address this issue.
2023-04-14 20:59:06 +08:00
f2d75cb492 [fix](Nereids) fix signature precision round for decimalv3 (#18639)
add decimalv3 signature to below functions:
ceil
dceil
dfloor
dround
floor
round
round_bankers
truncate
fix ComputePrecisionForRound to get correct signature
2023-04-14 18:18:41 +08:00
362b5a34ae [feat](stats) Support to delete expired stats periodically (#18614)
Support to delete expired stats periodically and manually.

default cleaner running interval is 2 days

Manually clean syntax is
```sql
DROP EXPIRED STATS
```

TODO:
1. process external catalog's stats
2. run drop at the appointed time
3. sleep a short time after drop one batch
2023-04-14 17:32:51 +08:00
4174d5a707 [opt](nereids) optimze aggregation estimation #18607
`select count(*) from T group by A, B`
suppose `ndv(A) > ndv(B)`
the estimated row count of aggregate is between ndv(A) and ndv(A) * ndv(B)

in previous version, we choose upper bound, that is ndv(A) * ndv(B). The drawback of this choice is the estimated row is often bigger that row count of T.

In this version, we choose the lower bound.
2023-04-14 16:13:25 +08:00
4d18ea30f4 [fix](Nereids) get_json_bigint should return bigint type (#18626) 2023-04-14 14:01:44 +08:00
8751f08d5a [bugfix](GEO)fix precision problem (#18642) 2023-04-14 10:39:19 +08:00
6c0af24e9d [Improve](simdjson reader) support UTF-8 unicode (with BOM) (#18585) 2023-04-13 21:58:44 +08:00
aa6b3cc537 [fix](planner)keep all agg functions if there is any virtual slots in group by list (#18630)
Because of the limitation of ProjectPlanner, we have to keep set agg functions materialized if there is any virtual slots in the group by list, such as 'GROUPING_ID' in the group by list etc.
2023-04-13 19:44:46 +08:00
2f64a8b387 [feature](GEO)Support read/write WKB/EWKB to gis types (#18526)
Support mutual conversion from wkb and gis types.also compatible with EWKB format
https://cwiki.apache.org/confluence/display/DORIS/DSIP-033%3A+More+GEO+functions
2023-04-13 16:25:18 +08:00
Pxl
eb46bcb304 [Bug](materialized-view) fix match wrong index on some scan node (#18561)
fix match wrong index on some scan node
2023-04-13 11:50:14 +08:00
df0aaece1d [Function](test) add some test cases for agg functions (#18610) 2023-04-13 10:23:41 +08:00
d57371da13 [feature](struct-type) support basic struct constructor function (#18190)
This commit will support struct and named_struct function.
2023-04-13 09:18:00 +08:00
1f9372558d [improve](regression case) Add more inverted index regression case (#18589)
1. add more inverted index regression case for unique mow
2. add inverted index case with different data types
2023-04-12 20:40:55 +08:00
a9f9366736 [fix](nereids) the data type of compareExpr and listQuery should be the same when creating InSubquery (#18539)
Consider sql

select table_B_alias.b from table_B_alias where table_B_alias.b in ( select a from table_A_alias );

if table_B_alias.b is int and table_A_alias.a is bigint,
we should cast(b as bigint) to make the data type the same as the InSubquery.
2023-04-12 20:02:37 +08:00
db44970685 [feature](stats) Support sync analyze (#18567)
Gammer:

```
ANALYZE [SYNC] TABLE ....
```

Add this feature so that we could test and tune stats framework conveniently.
2023-04-12 17:49:30 +08:00
b93e04ab66 [test](Nereids) add regression test to check join order for tpch queries (#18543)
by explain shape plan command, with stats injection, we add regression test to check tpch queries' plan shape.
2023-04-12 15:43:21 +08:00
43392918cd [Optimization](functions)Optimize function call for const columns. (#18310) 2023-04-12 11:11:01 +08:00
1238f6de97 [bug](array) fix be core in array_with_constant/array_repeat function when the first argument is nullable (#18404)
fix be core in array_with_constant/array_repeat function when the first argument is nullable
2023-04-11 19:46:41 +08:00
Pxl
da2d0bba74 [Chore](case) move some mv case from feut to p0 (#18475)
move some mv case from feut to p0
2023-04-11 11:48:55 +08:00
e562017801 [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2 (#17704) 2023-04-11 11:09:43 +08:00
101737023c [Bug](round) fix wrong scale for round-like function (#18507) 2023-04-11 09:36:59 +08:00
735cd15a3d [fix](nereids) PushdownAliasThroughJoin should handle same column with different alias in project list (#18470) 2023-04-10 11:50:37 +08:00
8d4a7325a9 [regression](join)Add subquery join case to regression test #17865 2023-04-09 18:15:37 +08:00
8a4a92f658 [Enchancement](compatible) show dateV2/datetimeV2 to date/datetime (#18358)
show dateV2/datetimeV2 to date/datetime

modify show create table
modify desc table
use desc table all to get real type from column ColumnType
2023-04-09 10:34:14 +08:00
0517616242 [vectorized](function) support array_repeat function to be compatible with hive syntax (#18028)
---------

Co-authored-by: zhangyu209 <zhangyu209@meituan.com>
2023-04-08 15:50:28 +08:00
0b8bc51b72 [fix](inverted index) Fix key column match query failed (#18436)
* [fix](inverted index) Fix key column match query failed

* [chore](regression case) add regression case

* [fix] fix regression case no order by
2023-04-08 15:45:08 +08:00
d881d71cd1 [Bug](cast) Fix bug for cast function between datetimev2 and string (#18442)
Fix bug for cast function between datetimev2 and string
2023-04-07 22:02:15 +08:00
30f2abe5d3 [FIX](Map)fix calculate map offset in olap convertor (#18295)
Fix be core when load bigger kv data in one row for map.
2023-04-07 17:04:08 +08:00
926c6056c7 [refactor](config) Delete the environment variable enable_vectorized_engine (#18166) 2023-04-07 14:23:16 +08:00
c32adba1cf [Refactor](Pipeline) Refactor pipeline code to improve coverage (#18376)
Refactor pipeline code to improve coverage
2023-04-07 13:09:44 +08:00
Pxl
e77da1519a [Enchancement](materialized-view) adjust desc table all display fields (#18357)
adjust desc table all display fields
2023-04-07 11:14:17 +08:00
Pxl
267b690dad [Bug](materialized-view) fix materialized-view query match not consider with order by elements (#18384)
fix materialized-view query match not consider with order by elements
2023-04-07 11:11:18 +08:00
505f25c580 [fix](planner)use base index if the where clause is a constant value (#18367)
sql : select bitmap_empty() from d_table where true;
should always use base index instead of any mv, because the conjuncts is constant (true) and use none of the column from any mv
2023-04-07 09:15:00 +08:00
4e1cdb9ce7 [fix](agg_sort)fix bug of agg sort group concat with order by(#18447) 2023-04-07 08:42:36 +08:00
759f1da32e [Enhencement](Backends) add HostName filed in backends table and delete backends table in information_schema (#18156)
1.  Add `HostName` field for `show backends` statement and `backends()` tvf.
2. delete the `backends` table in `information_schema` database
2023-04-07 08:30:42 +08:00
550c8aa648 [Bug](DECIMALV3) fix wrong decimal scale returned by function round (#18375) 2023-04-06 14:44:21 +08:00
db766bb073 [fix](planner) decimalv2 castTo decimalv2 should change type directly (#18297) 2023-04-06 13:51:50 +08:00
8b61709ec8 [feature](multi-catalog) support select current_catalog(); (#18163) 2023-04-06 12:06:10 +08:00
4ec6aa1691 [fix](planner) trying register constnat slotRef to table cause NPE (#18356)
could reproduced by:

CREATE TABLE t (
name varchar(128)
) ENGINE=OLAP
UNIQUE KEY(name)
DISTRIBUTED BY HASH(name) BUCKETS 1;

insert into t values('abc');

SELECT cd
FROM
(SELECT cast(now() as string) cd FROM t) t1
JOIN
(select cast(now() as string) td from t GROUP BY now()) t2
ON t1.cd = t2.td;

ERROR 1105 (HY000): errCode = 2, detailMessage = Unexpected exception: null
2023-04-06 11:50:12 +08:00
4ca0c0face [fix](join) fix wrong result of right join (#18365)
When processing data in hash table for right join and full outer join, if the output data rows of one hash bucket excceeds batch size, the logic when continue processing this bucket is wrong, it should differentiate between different join types.
2023-04-06 10:55:58 +08:00
66a0c090b8 [fix](column) Add unimplemented replicate function in ColumnStruct (#18368) 2023-04-06 09:50:27 +08:00
7c36bef6bc [Feature-Wip](MySQL Load)Show load warning for my sql load (#18224)
1. Support the show load warnings for mysql load to get the detail error message.
2. Fix fillByteBufferAsync not mark the load as finished in same data load
3. Fix drain data only in client mode.
2023-04-04 22:44:48 +08:00
87e83081ff [test](compaction) add delete test (#18335) 2023-04-04 12:28:19 +08:00
54dbb4af67 [vectorzied](jdbc) refactor jdbc table read array type (#18187)
jdbc read array type get result from Doris is string, PG is java.sql.array, CK is java.lang.object
it's difficult to maintain and read the code,
so change all database's array result to string, then add a cast function from string to doris array type
2023-04-04 11:57:04 +08:00
8b85c55117 [vectorized](function) Support array_shuffle and shuffle function. (#18116)
---------

Co-authored-by: zhangyu209 <zhangyu209@meituan.com>
2023-04-04 08:53:13 +08:00
eb0fd0017e [Fix](orc-reader) Fix the scale of decimal column is incorrect when query orc tables. (#18324)
The scale of decimal column is incorrect when query orc tables.
2023-04-04 08:50:47 +08:00
88c5e64c4a [fix](nereids) fix bug of SelectMaterializedIndexWithAggregate rule (#18265)
1. create a project node to adjust the output column position when a mv is selected in olap scan node
2. pass SlotReference's column info when call Alias's toSlot() method
3. should compare plan's logical properties when compare two plans after rewrite
2023-04-03 22:32:43 +08:00
fe9d2b00fc [test](jdbc catalog) add clickhouse jdbc catalog base type test (#18007) 2023-04-03 20:18:36 +08:00