Commit Graph

17549 Commits

Author SHA1 Message Date
66ed093410 [test](Nereids): fix test push_down_top_n (#26937) 2023-12-07 23:07:32 +08:00
cbb238a0ff [improve](env) Add disk usage in not ready msg (#28125) 2023-12-07 22:49:52 +08:00
f9d4690023 [improve](stack_trace) avoid print stack trace in csv and json reader #28129 2023-12-07 22:45:18 +08:00
81a0f8c041 [Feature](function) support generating const values from tvf numbers (#28051)
If specified, got a column of constant. otherwise an incremental series like it always be.

mysql> select * from numbers("number" = "5", "const_value" = "-123");
+--------+
| number |
+--------+
|   -123 |
|   -123 |
|   -123 |
|   -123 |
|   -123 |
+--------+
5 rows in set (0.11 sec)
2023-12-07 22:26:43 +08:00
397a401241 [fix](arrow-flight) Modify FE Arrow version to 14.0.1 #28093
Previously temporarily upgrade Arrow to dev version 15.0.0-SNAPSHOT, because the latest release version Arrow 14.0.1 jdbc:arrow-flight-sql has BUG, jdbc:arrow-flight-sql cannot be used normally, see: apache/arrow#38785

But Arrow 15.0.0-SNAPSHOT was not published to the Maven central repository, and the network could not be connected sometimes, so back to Arrow 14.0.1. jdbc:arrow-flight-sql will be supported after upgrading to Arrow 15.0.0 release version.
2023-12-07 22:25:08 +08:00
a2d66911cd [chore](docs) Fix partition cache design principles #28110 2023-12-07 22:23:46 +08:00
b1c5519aa8 [doc](statistics)Update external catalog statistics doc (#28123) 2023-12-07 21:33:05 +08:00
104a822a2f [Refacotr](RuntimeFilter) refactor rf code to improve performance (#28094) 2023-12-07 20:32:30 +08:00
be81eb1a9b [feature](nereids) Support inner join query rewrite by materialized view (#27922)
Work in process. Support inner join query rewrite by materialized view in some scene.
Such as an exmple as following:

> mv = "select  lineitem.L_LINENUMBER, orders.O_CUSTKEY " +
>             "from orders " +
>             "inner join lineitem on lineitem.L_ORDERKEY = orders.O_ORDERKEY "
>     query = "select lineitem.L_LINENUMBER " +
>             "from lineitem " +
>             "inner join orders on lineitem.L_ORDERKEY = orders.O_ORDERKEY "
2023-12-07 20:29:51 +08:00
f37215a32a [fix](Nereids) insert into target table lock should include finalize (#28085) 2023-12-07 20:15:12 +08:00
65fc2e0438 [fix](Nereids) forbid two TVF in one fragment since the limit of coordinator (#28114) 2023-12-07 19:58:31 +08:00
cc9b4bcddb [Fix](variant) fallback to none partial update for mow table (#28116) 2023-12-07 19:30:24 +08:00
942450a2e5 [Fix](Variant) ColumnObject need to be finalized when doing ColumnObject::update_hash_with_value (#28119)
Otherwise accessing rows at `n` will lead to heap buffer overflow

```
 5# SipHash::update(char const*, unsigned long) at /home/zcp/repo_center/doris_master/doris/be/src/vec/common/sip_hash.h:132
 6# doris::vectorized::ColumnString::update_hash_with_value(unsigned long, SipHash&) const at /home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_string.h:452
 7# doris::vectorized::ColumnObject::update_hash_with_value(unsigned long, SipHash&) const at /home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_object.cpp:1433
 8# doris::vectorized::Block::update_hash(SipHash&) const at /home/zcp/repo_center/doris_master/doris/be/src/vec/core/block.cpp:721
 9# doris::EngineChecksumTask::_compute_checksum() at
```
2023-12-07 18:48:05 +08:00
34642781c2 [fix](meta) fix ConcurrentModificationException when dump image (#28072)
```
Caused by: java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437) ~[?:1.8.0_131]
        at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_131]
        at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_131]
        at org.apache.doris.catalog.CatalogRecycleBin.write(CatalogRecycleBin.java:1047) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.catalog.Env.saveRecycleBin(Env.java:2298) ~[doris-fe.jar:1.2-SNAPSHOT]
```

When calling `/dump` api to dump image, ConcurrentModificationException may be thrown.
Because no lock to protect `CatalogRecycleBin`
2023-12-07 18:26:02 +08:00
3dcbf16404 [Fix](Outfile) The Struct type data exported from select outfile to the csv file format should contain a column name #28068
If the original data is:
```sql
+-----------------------------------------------------+
| s_info                                              |
+-----------------------------------------------------+
| {"s_id": 2, "s_name": "nereids", "s_address": "20"} |
| {"s_id": 1, "s_name": "doris", "s_address": "18"}   |
+-----------------------------------------------------+
```

In the original logic, the struct type data exported to a csv file format did not contain column names,like
```
{2, "nereids", "20"} 
{1, "doris", "18"}
```

This pr do not need to be merged into branch-2.0
2023-12-07 18:23:36 +08:00
394b420180 [Update](inverted index) use session variable for inverted index try query threshold (#28052)
* [Update](inverted index) use session variable for inverted index try query threshold

* remove unused config

* update clucene
2023-12-07 17:54:44 +08:00
172747669e [fix](Nereids)fix regression case:nereids_rules_p0/transposeJoin/transposeSemiJoinAgg #28111 2023-12-07 17:41:08 +08:00
a27c068a9d [improve](move-memtable) make StreamWait time configurable (#28086) 2023-12-07 17:27:43 +08:00
84a651d976 [improve](load) rewrite memtable memory limiter rules (#27759) 2023-12-07 17:26:26 +08:00
bc12a05915 [fix](Nereids) explain graph insert-select NPE (#28007) 2023-12-07 17:25:44 +08:00
9c63dfd692 [improve](move-memtable) support variant in move-memtable (#28084)
* [improve](move-memtable) support variant in move-memtable

Pass the flush schema to the destination end of the sink.
2023-12-07 16:41:38 +08:00
1a46cf6fb5 [fix](split_by_string) Fix split by string core on column string (#28030) 2023-12-07 16:36:13 +08:00
495c01bdfd [fix](session-variable) enable_unicode_name_support need forward to master (#28112)
Session variable `enable_unicode_name_support` is used to enable using unicode for table/column names.
And it should be forward to master when executing `create table` stmt on non-master FE.
Otherwise, even if we set `enable_unicode_name_support` to true on non-master FE,
we can create table with unicode name.
2023-12-07 15:43:11 +08:00
8526b9ffbe [imporvement](table property) support for alter table property disable_auto_compaction (#27961)
in some case, some tablets may cause coredump or OOM when compaction, and it is necessary to manually close the compaction of a specific table by 'disable_auto_compaction' to make be service available

This commit allow modify disable_auto_compaction table property in schema change.

---------

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-12-07 15:08:39 +08:00
cb9a6f63ab [refactor](simd_json_reader) refactor simd json parse to adapt stream parse (#27972) 2023-12-07 14:45:15 +08:00
3f3843fd4f [fix](load) fix loaded rows error (#28061) 2023-12-07 14:43:51 +08:00
1dabd0af43 [Fix](inverted index) fix need read data optimize problem (#28104) 2023-12-07 14:05:29 +08:00
f2477a3fcc [Regression](bitmap index) fix bitmap index test case (#28109) 2023-12-07 13:29:24 +08:00
c5f00fa746 [fix](move-memtable) temporary disable load stream profile (#28029) 2023-12-07 12:05:29 +08:00
8c79b86f5b Revert "[feature](merge-on-write) enable merge-on-write by default (#27188)" (#28096)
This reverts commit 00c8bab84de8154052f9d323800b436cd0ad36e5.
2023-12-07 11:31:36 +08:00
3a7a8bb107 [opt](resource-tag) root and admin user can use any resource tag by default (#28088)
In #25331, I change the behavior of user's default resource tag, that is, if a user does not set resource tag,
it can only use default resource tag.
This PR change this logic. The normal user can only use default resource tag if resource tag is not set,
but root and admin user can use any resource tag if resource tag is not set.
2023-12-07 11:22:30 +08:00
8df94f0d07 [fix](remote-scanner-pool) missing _remote_thread_pool_max_size value (#28057) 2023-12-07 11:18:42 +08:00
0f1da35c25 [community](slack)Update slack channel links (#28100) 2023-12-07 11:02:08 +08:00
6f89fae22c [fix](profile) fix profile TotalTime is 0ns when pipeline (#28048)
* [fix](profile) fix profile TotalTime is 0ns when pipeline

Signed-off-by: nextdreamblue <zxw520blue1@163.com>

* fix

Signed-off-by: nextdreamblue <zxw520blue1@163.com>

---------

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
2023-12-07 10:30:41 +08:00
00bcf0d762 [doc](paimon) add more example for paimon catalog (#28087) 2023-12-07 10:21:02 +08:00
b6722653cf [test](Job)Delete the JOB show syntax (now we use TVF) and add tvf case (#28058) 2023-12-07 10:17:52 +08:00
630de740ea [fix](meta) fix bug for using full name in show_full_columns stmt (#28019) 2023-12-07 10:17:25 +08:00
616ba2823f [regression test](broker load) Test case missing files (#27580) 2023-12-07 10:17:07 +08:00
4cac07be30 [improvement](statistics)Analyze empty table. #28077
Analyze a table even when it's empty. The result should be like this:

mysql> show column stats nation;
+-------------+-------+------+----------+-----------+---------------+------+------+--------+--------------+---------+-------------+---------------------+
| column_name | count | ndv  | num_null | data_size | avg_size_byte | min  | max  | method | type         | trigger | query_times | updated_time        |
+-------------+-------+------+----------+-----------+---------------+------+------+--------+--------------+---------+-------------+---------------------+
| n_comment   | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
| n_nationkey | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
| n_regionkey | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
| n_name      | 0.0   | 0.0  | 0.0      | 0.0       | 0.0           | N/A  | N/A  | FULL   | FUNDAMENTALS | MANUAL  | 0           | 2023-12-06 19:22:09 |
+-------------+-------+------+----------+-----------+---------------+------+------+--------+--------------+---------+----
2023-12-07 10:16:52 +08:00
42b3dd35bb [regression test](broker load) add case for without filepath (#27658) 2023-12-07 10:15:37 +08:00
e3e78a79d7 [chore](StoragePolicy) Don't print stack when invoking the storage manager #28091 2023-12-07 10:06:31 +08:00
54d062ddee [feature](stream load) (step one)Add arrow data type for stream load (#26709)
By using the Arrow data format, we can reduce the streamload of data transferred and improve the data import performance
2023-12-06 23:29:46 +08:00
2ca66ff61c [typo](test) Use correct sql stmt in regression test (#27992) 2023-12-06 23:28:29 +08:00
4a4d137402 [feature](workloadgroup) support nereids internal query and all dml query (#28054)
support nereids internal query to bind a workload group
support insert into select bind workload group
support create table as select bind workload group
change token wait timeout to be query timeout or queue timeout
query queue should not bind to pipeline engine, it could be used every where.
2023-12-06 21:07:55 +08:00
00c8bab84d [feature](merge-on-write) enable merge-on-write by default (#27188) 2023-12-06 21:06:58 +08:00
d0697db7b8 [fix](doris compose) fix up command (#27573) 2023-12-06 21:06:10 +08:00
d4f9e12ec7 [fix](load) fix memtable mem_tracker too large (#25205) 2023-12-06 21:04:21 +08:00
0ff5a1cc25 [fix](doc) spell error and aligned with code (#27609) 2023-12-06 20:58:39 +08:00
bd808ae3b4 [cleanup](load) remove unused LoadChannel::get_tablets_channels (#27981) 2023-12-06 20:50:26 +08:00
0282baaf56 [typo](docker) add logs console (#27570)
doris 2.0.2+ need use start_xx.sh --console
2023-12-06 20:50:09 +08:00