Commit Graph

2208 Commits

Author SHA1 Message Date
45b2dbab6a [improve](group commit) Group commit support max filter ratio when rows is less than value in config (#28139) 2023-12-12 16:33:36 +08:00
d25cbdd4dc [feature](Nereids): one side eager aggregation (#28143) 2023-12-12 15:38:31 +08:00
cd8885f63e [fix](Nereids): support Chinese characters set (#28256) 2023-12-12 13:22:14 +08:00
e49ed3d885 [regression test](memtable) add case for aggregation memtable (#28056)
1. create aggregation table
2. insert some data
3. drop the table and create again
4. modify some parameters for some branch
5. insert some data
6. change the parameters back to its default
2023-12-12 11:14:59 +08:00
d4f89389e3 [improve](group commit) Group commit support skip wal (#27957) 2023-12-11 19:38:32 +08:00
9b8de017df [Regression test](inverted index) fix regression case for index_compound_directory_fault_injection (#28232) 2023-12-11 19:17:28 +08:00
3e1e8d2ebe [fix](jdbc catalog) Fixed data conversion problem when all data is null (#28230) 2023-12-11 17:57:57 +08:00
1bbc54d1b2 [regression-test](variant) change p2 case to s3 load (#28193) 2023-12-11 12:31:25 +08:00
ac167f493b [fix](join) fix decimal overflow caused by left outer join (#28221)
For left outer join or full outer join, when build side data is empty, null data is output for build side, but nested column data of nullable column is not properly initialized, which may cause decimal arithmetic overflow
2023-12-11 11:51:05 +08:00
7c163fdf21 [test](decimal) add some cases about overflow (#28198) 2023-12-11 11:22:53 +08:00
1e5ff40e17 [refactor](group commit) remove future block (#27720)
Co-authored-by: huanghaibin <284824253@qq.com>
2023-12-11 08:41:51 +08:00
485d7db516 [fix](partial update) Fix missing rowsets during doing alignment when flushing memtable due to compaction (#28062) 2023-12-10 12:09:48 +08:00
07336980f9 [fix](meta) show partitions with Limit for external HMS tables (27835) (#27835)
This enhancement shall extend existing logic for SHOW PARTITIONS FROM to include: -

Limit/Offset
Where [partition name only] [equal operator and like operator]
Order by [partition name only]
Issue Number: close #27834
2023-12-09 01:44:45 +08:00
Pxl
027b06059a [Feature](materialized-view) support count(1) on materialized view (#28135)
support count(1) on materialized view
fix match failed like select k1, sum(k1) from t group by k1
2023-12-09 01:36:46 +08:00
055b3885c9 [Fix](inverted index) fix compound directory flush buffer error (#28191) 2023-12-09 00:57:35 +08:00
baf85547ae [feature](jdbc) support call function to pass sql directly to jdbc catalog #26492
Support a new stmt in Nereids:
`CALL EXECUTE_STMT("jdbc", "stmt")`

So that we can pass the origin stmt directly to the datasource of a jdbc catalog.

show case:
```
mysql> select * from mysql_catalog.db1.tbl1;
+------+------+
| k1   | k2   |
+------+------+
|  111 | 222  |
+------+------+
1 row in set (0.63 sec)

mysql> call execute("mysql_catalog", "insert into db1.tbl1 values(1,'abc')");
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_catalog.db1.tbl1;
+------+------+
| k1   | k2   |
+------+------+
|  111 | 222  |
|    1 | abc  |
+------+------+
2 rows in set (0.03 sec)

mysql> call execute_stmt("mysql_catalog", "delete from db1.tbl1 where k1=111");
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_catalog.db1.tbl1;
+------+------+
| k1   | k2   |
+------+------+
|    1 | abc  |
+------+------+
1 row in set (0.03 sec)
```
2023-12-08 23:06:05 +08:00
573b594df3 [improvement](Variant Type) Support displaying subcolumns expanded for the variant column (#27764) 2023-12-08 20:34:58 +08:00
51f320a606 [bug](function) fix array_apply function return wrong result (#28133) 2023-12-08 20:14:54 +08:00
bc40025631 [opt](Nereids)Join cluster connectivity (#27833)
* estimation join stats by connectivity
2023-12-08 14:55:10 +08:00
61d556c718 [fix](nereids)runtime filter translator failed on set operator (#28102)
* runtime filter translator failed on set operator
2023-12-08 12:58:42 +08:00
341822ec05 [regression-test](Variant) add compaction case for variant and fix bugs (#28066) 2023-12-08 12:18:46 +08:00
e75d91c91b [regression-test](Variant) Add more cases related to schema changes (#27958)
* [regression-test](Variant) Add more cases related to schema changes

And fix bugs about schema change for variant:
fix bug schema change crash on doing schema change with tablet schema that contains extracted columns
2023-12-08 10:15:12 +08:00
66ed093410 [test](Nereids): fix test push_down_top_n (#26937) 2023-12-07 23:07:32 +08:00
81a0f8c041 [Feature](function) support generating const values from tvf numbers (#28051)
If specified, got a column of constant. otherwise an incremental series like it always be.

mysql> select * from numbers("number" = "5", "const_value" = "-123");
+--------+
| number |
+--------+
|   -123 |
|   -123 |
|   -123 |
|   -123 |
|   -123 |
+--------+
5 rows in set (0.11 sec)
2023-12-07 22:26:43 +08:00
be81eb1a9b [feature](nereids) Support inner join query rewrite by materialized view (#27922)
Work in process. Support inner join query rewrite by materialized view in some scene.
Such as an exmple as following:

> mv = "select  lineitem.L_LINENUMBER, orders.O_CUSTKEY " +
>             "from orders " +
>             "inner join lineitem on lineitem.L_ORDERKEY = orders.O_ORDERKEY "
>     query = "select lineitem.L_LINENUMBER " +
>             "from lineitem " +
>             "inner join orders on lineitem.L_ORDERKEY = orders.O_ORDERKEY "
2023-12-07 20:29:51 +08:00
cc9b4bcddb [Fix](variant) fallback to none partial update for mow table (#28116) 2023-12-07 19:30:24 +08:00
3dcbf16404 [Fix](Outfile) The Struct type data exported from select outfile to the csv file format should contain a column name #28068
If the original data is:
```sql
+-----------------------------------------------------+
| s_info                                              |
+-----------------------------------------------------+
| {"s_id": 2, "s_name": "nereids", "s_address": "20"} |
| {"s_id": 1, "s_name": "doris", "s_address": "18"}   |
+-----------------------------------------------------+
```

In the original logic, the struct type data exported to a csv file format did not contain column names,like
```
{2, "nereids", "20"} 
{1, "doris", "18"}
```

This pr do not need to be merged into branch-2.0
2023-12-07 18:23:36 +08:00
9c63dfd692 [improve](move-memtable) support variant in move-memtable (#28084)
* [improve](move-memtable) support variant in move-memtable

Pass the flush schema to the destination end of the sink.
2023-12-07 16:41:38 +08:00
1a46cf6fb5 [fix](split_by_string) Fix split by string core on column string (#28030) 2023-12-07 16:36:13 +08:00
3f3843fd4f [fix](load) fix loaded rows error (#28061) 2023-12-07 14:43:51 +08:00
8c79b86f5b Revert "[feature](merge-on-write) enable merge-on-write by default (#27188)" (#28096)
This reverts commit 00c8bab84de8154052f9d323800b436cd0ad36e5.
2023-12-07 11:31:36 +08:00
616ba2823f [regression test](broker load) Test case missing files (#27580) 2023-12-07 10:17:07 +08:00
00c8bab84d [feature](merge-on-write) enable merge-on-write by default (#27188) 2023-12-06 21:06:58 +08:00
Pxl
299fcc443e [Bug](agg-state) fix stream load failed on agg-state column (#28050) 2023-12-06 20:41:29 +08:00
605257ccb7 [Enhancement](group commit) Add regression case for wal limit (#27949) 2023-12-06 14:23:50 +08:00
e791e31b7f [test](Nereids): add regression test eliminate/infer rules (#27985) 2023-12-06 14:21:06 +08:00
05adbfdb3d [feature](inverted index) match_phrase_prefix feature added (#27404)
select count() from test_index_match_phrase_prefix where request match_phrase_prefix 'xxx';
2023-12-05 20:15:13 +08:00
e79422addc [refactor](Nereids) compatible with all ability legacy planner (#27947)
refactor:
1. split InsertIntoTableCommand into three sub command
- InsertIntoTableCommand
- InsertOverwriteTableCommand
- BatchInsertIntoTableCommand

feature:
1. support DEFAULT keywords in values list
2. support empty values list
3. support temporary partition
4. support insert into values in txn model

fix:
1. should start transaction before release read lock on target table
2023-12-05 19:10:55 +08:00
Pxl
8a761dff84 [Bug](materialized-view) fix create mv failed on unique table (#27971)
fix create mv failed on unique table
2023-12-05 14:53:09 +08:00
c98b80ae6a [Feature](functions) support ignore and nullable functions (#27848)
support ignore and nullable functions
2023-12-05 14:09:32 +08:00
79f6f85cf1 [FIX](serde)fix datetimev2 serde parse from string with scale (#27965) 2023-12-05 13:58:32 +08:00
17016b9797 [improvement](decimal) use new way for decimal arithmetic precision promotion (#27787)
* [DNM](decimal) use new way for decimal arithmetic precision promotion

* [improvement](decimal) [DNM](decimal) use new way for decimal arithmetic precision promotion
1. [DNM](decimal) use new way for decimal arithmetic precision promotion
2. throw exception if it overflows for decimal arithmetics
3. throw exception if it overflows when casting among number types

* fix compile error of gcc

* improvement

---------

Co-authored-by: morrySnow <morrysnow@126.com>
2023-12-05 12:54:40 +08:00
2f63999066 [fix](Nereids): Preserve "" in single quote strings and '' in double quote strings. (#27959) 2023-12-05 12:30:03 +08:00
358d73a0ae [FIX](complextype) fix empty quote with complex type (#27942) 2023-12-05 12:25:26 +08:00
50ad40a7a8 [test](Nereids): add infer-predicates regression test (#27850) 2023-12-05 10:16:01 +08:00
4934f7ed8d [enhancement](Nereids) add test for some push down filter rule (#27757) 2023-12-04 20:57:57 +08:00
e19af1b2ed [regression](Nereids) add rule test for push down limit + sort test (#26642) 2023-12-04 14:18:55 +08:00
e62d19d90d [improve](partition) support auto list partition with more columns (#27817)
before the partition by column only have one column.
now remove those limit, could have more columns.
2023-12-04 11:33:18 +08:00
f2cfc87aca [fix](nereids) temporary partition is selected only if user manually specified (#27893)
q1: "select * from ut_p temporary partitions(tp1) where val > 0"
in q1, temporary partition tp1 is scaned

q2: "select * from ut_p where val > 0"
in q2, temporary partition tp1 is not scaned.
2023-12-04 09:44:27 +08:00
97d36b4f38 [fix](csv_reader) fix trim_double_quotes behavior change (#27882) 2023-12-03 22:57:55 +08:00