Commit Graph

17549 Commits

Author SHA1 Message Date
53802fe0da [doc] document desc param is incorrect #26063 (#26064) 2023-12-11 10:33:07 +08:00
f236261256 [fix](regression) compaction cases adapt force_olap_table_replica_num option (#28136) 2023-12-11 10:08:21 +08:00
8f2202c89d [minor](log) Add debug info in operators (#28211) 2023-12-11 10:02:24 +08:00
1e5ff40e17 [refactor](group commit) remove future block (#27720)
Co-authored-by: huanghaibin <284824253@qq.com>
2023-12-11 08:41:51 +08:00
320ddf4987 [pipelineX](improvement) Support multiple instances execution on single tablet (#28178) 2023-12-10 20:18:41 +08:00
485d7db516 [fix](partial update) Fix missing rowsets during doing alignment when flushing memtable due to compaction (#28062) 2023-12-10 12:09:48 +08:00
a3cd36ce60 [bug](cooldown) Fix incorrect remote rowset dir after restarting BE (#28140) 2023-12-10 00:44:01 +08:00
5aa90a3bce [pipelineX](local shuffle) Fix bucket hash shuffle (#28202) 2023-12-10 00:35:00 +08:00
61379b141e [fix](insert) fix group commit regression test (#28142) 2023-12-09 16:24:20 +08:00
4e86f9bab5 [improve](move-memtable) include and check offset when append data (#28159) 2023-12-09 16:21:36 +08:00
16e232a8a1 [minor](lower-table-names) use GlobalVariable.lowerCaseTableNames instead of Config.lower_case_table_names (#27911)
GlobalVariable.lowerCaseTableNames instead of Config.lower_case_table_names
2023-12-09 12:04:26 +08:00
363721e066 [Bug](udf) java-udf function open failed cause BE core dump #28063
when the java-udf open function failed, and some JNI have not set,
so in close function can't call jni.
2023-12-09 11:00:30 +08:00
42aa174405 [chore](log) Log to trace before wait rpc timeout #28024 2023-12-09 10:04:43 +08:00
9d9b6462bf [improve](group_commit) optimize group commit select be logic #28190
Group commit choose be always first no decommissioned be in all be.

Choose be with selectBackendIdsByPolicy like common stream load and do not choose decommissioned be may be better.
2023-12-09 05:09:52 +08:00
287bd87a4f [typo](docs)add some faq for flink-connector-doris (#26309)
* add flink-connector-doris faq

* add faq
2023-12-09 02:19:49 +08:00
bd8130154a [fix](doc) spell errors fixes hardware-info-action (#28154) 2023-12-09 01:47:19 +08:00
c6f8b1b2ee [fix](repository) the exist repo_file must contails same name with new repo (#27668)
The user manually adjusted the 'name' field in the __repo_info file under the repo file on S3, but did not modify the folder name. This led to an issue when the user created a repo with the same name as the folder in a certain cluster. The system parsed the 'name' field in the existing __repo_info and used an incorrect name, causing the subsequent repo to be unusable. A judgment has been added here: the 'name' field in the __repo_info must be the same as the new repo's name, otherwise, an error will be reported.
2023-12-09 01:46:54 +08:00
07336980f9 [fix](meta) show partitions with Limit for external HMS tables (27835) (#27835)
This enhancement shall extend existing logic for SHOW PARTITIONS FROM to include: -

Limit/Offset
Where [partition name only] [equal operator and like operator]
Order by [partition name only]
Issue Number: close #27834
2023-12-09 01:44:45 +08:00
99b38ddca7 [improve](env) Ensure next majority is met before drop an alive follower (#28101)
Here is an example:

```
mysql> ALTER SYSTEM DROP FOLLOWER "127.0.0.1:19017";
ERROR 1105 (HY000): errCode = 2, detailMessage = Unable to drop this alive
follower, because the quorum requirements are not met after this command
execution. Current num alive followers 2, num followers 3, majority after
execution 2
```
2023-12-09 01:41:38 +08:00
99be9d6ad3 [fix](memlimiter) refresh memtracker before flush active memtables (#28196)
Currently, _flush_active_memtables() is using stale memtracker data, especially when some other thread has just it.
Refresh memtrackers before flush to avoid this problem.
2023-12-09 01:40:51 +08:00
Pxl
027b06059a [Feature](materialized-view) support count(1) on materialized view (#28135)
support count(1) on materialized view
fix match failed like select k1, sum(k1) from t group by k1
2023-12-09 01:36:46 +08:00
b6e72d57c5 [Improvement](hms catalog) support show_create_database for hms catalog (#28145)
* [Improvement](hms catalog) support show_create_database for hms catalog

* update
2023-12-09 01:34:21 +08:00
055b3885c9 [Fix](inverted index) fix compound directory flush buffer error (#28191) 2023-12-09 00:57:35 +08:00
abc802b5ba [bugfix](core) child block is shared between operator and node, it should be shared ptr (#28106)
_child_block in nest loop join , table value function, repeat node will be shared between ExecNode and related operator, but it should not be a unique ptr in operator, it belongs to exec node.

It will double free the block, if operator's close method is not called correctly.

It should be a shared ptr, then it will not core even if the opeartor's close method is not called.
2023-12-09 00:18:14 +08:00
8eed760704 [fix](planner) separate table's isPartitioned() method (#28163)
This PR #27515 change the logic if Table's `isPartitioned()` method.
But this method has 2 usages:

1. To check whether a table is range or list partitioned, for some DML operation such as Alter, Export.

    For this case, it should return true if the table is range or list partitioned. even if it has only
    one partition and one buckets.

2. To check whether the data is distributed (either by partitions or by buckets), for query planner.

    For this case, it should return true if table has more than one bucket. Even if this table is not
    range or list partitioned, if it has more than one bucket, it should return true.

So we should separate this method into 2, for different usages.
Otherwise, it may cause some unreasonable plan shape
2023-12-08 23:15:45 +08:00
baf85547ae [feature](jdbc) support call function to pass sql directly to jdbc catalog #26492
Support a new stmt in Nereids:
`CALL EXECUTE_STMT("jdbc", "stmt")`

So that we can pass the origin stmt directly to the datasource of a jdbc catalog.

show case:
```
mysql> select * from mysql_catalog.db1.tbl1;
+------+------+
| k1   | k2   |
+------+------+
|  111 | 222  |
+------+------+
1 row in set (0.63 sec)

mysql> call execute("mysql_catalog", "insert into db1.tbl1 values(1,'abc')");
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_catalog.db1.tbl1;
+------+------+
| k1   | k2   |
+------+------+
|  111 | 222  |
|    1 | abc  |
+------+------+
2 rows in set (0.03 sec)

mysql> call execute_stmt("mysql_catalog", "delete from db1.tbl1 where k1=111");
Query OK, 0 rows affected (0.01 sec)

mysql> select * from mysql_catalog.db1.tbl1;
+------+------+
| k1   | k2   |
+------+------+
|    1 | abc  |
+------+------+
1 row in set (0.03 sec)
```
2023-12-08 23:06:05 +08:00
2b914aebb6 [opt](nereids)improve partition prune when Date function is used (#27960)
date func in partition prune
2023-12-08 21:53:39 +08:00
18ef131410 [fix](load) select more active memtables at once in memtable limiter (#28171) 2023-12-08 21:45:35 +08:00
06404114f1 [Fix](point query) fix memleak by increasing scanReplicaIds when using prepared statement (#28184)
OlapScanNode should release memory for `scanReplicaIds`
2023-12-08 21:02:01 +08:00
5e7afa768e [fix](statistics)Avoid potential NPE #28147 2023-12-08 20:42:17 +08:00
573b594df3 [improvement](Variant Type) Support displaying subcolumns expanded for the variant column (#27764) 2023-12-08 20:34:58 +08:00
51f320a606 [bug](function) fix array_apply function return wrong result (#28133) 2023-12-08 20:14:54 +08:00
0931eb536c Revert "[Improvement](auditlog) add column catalog for audit log and audit log table (#26403)" (#28177)
This reverts commit daea751a986823bf5858704663d58f49fd5dfb39.
2023-12-08 18:46:59 +08:00
75b55f8f2f [enhance](session)check invalid value when set parallel instance variables (#28141)
in some case, if set incorrectly, will be cause BE core dump

10:18:19   *** SIGFPE integer divide by zero (@0x564853c204c8) received by PID 2132555 
    int max_scanners =
            config::doris_scanner_thread_pool_thread_num / state->query_parallel_instance_num();
2023-12-08 17:38:48 +08:00
226a0c3b1d [chore](memory) Warning in log when turning on THP (#28122) 2023-12-08 17:38:38 +08:00
bc40025631 [opt](Nereids)Join cluster connectivity (#27833)
* estimation join stats by connectivity
2023-12-08 14:55:10 +08:00
6da36e1077 [feature](merge-cloud) Refactor write path code by abstract base class (#26537)
Refactor write path code by abstract base class. Whether to use `StorageEngine` or `CloudStorageEngine` will be determined during compilation instead of runtime `config::cloud_mode` to avoid unexpected null pointer or undefined behavior issues caused by merging code.

Class that depend on `StorageEngine` but are shared by the cloud mode need to have an abstract base class. Common code should be extracted into the base class, while the code that depends on `StorageEngine` should be implemented in a `StorageEngine` mix-in class of the base class.
2023-12-08 14:50:36 +08:00
16230b5ebd [Enhance](multi-catalog) parse hive view ddl first to avoid NPE. (#28067) 2023-12-08 13:54:50 +08:00
61d556c718 [fix](nereids)runtime filter translator failed on set operator (#28102)
* runtime filter translator failed on set operator
2023-12-08 12:58:42 +08:00
341822ec05 [regression-test](Variant) add compaction case for variant and fix bugs (#28066) 2023-12-08 12:18:46 +08:00
59ec3da899 open workload group in PR pipeline (#27744) 2023-12-08 11:56:03 +08:00
ebed055d2b [chore](clone) rename clone request field (#27591) 2023-12-08 11:53:57 +08:00
d534cdf027 [compile](BE) let arm gcc know some function no return (#28157)
let arm gcc know some function no return
2023-12-08 11:32:08 +08:00
cd108688c1 [Chore](docs)Fix job error docs (#28127) 2023-12-08 10:24:21 +08:00
0947bf4e97 [opt](mysql serde) Avoid core dump when converting invalid block to mysql result (#28069)
BE will core dump if result block is invalid when we doing result serialization.
An existing bug case is described in #28030, so we add check branch to avoid BE core dump due to out of range related problem.
2023-12-08 10:21:09 +08:00
25b90eb782 [Feature](function) support random int from specific range (#28076)
mysql> select rand(-20, -10);
+------------------+
| random(-20, -10) |
+------------------+
|              -13 |
+------------------+
1 row in set (0.10 sec)
2023-12-08 10:15:25 +08:00
e75d91c91b [regression-test](Variant) Add more cases related to schema changes (#27958)
* [regression-test](Variant) Add more cases related to schema changes

And fix bugs about schema change for variant:
fix bug schema change crash on doing schema change with tablet schema that contains extracted columns
2023-12-08 10:15:12 +08:00
1d345877ce [fix](regression-test) load_to_single_tablet assertEquals usage (#28128) 2023-12-08 10:09:44 +08:00
d8d8f15bf3 [improvement](vectorization) Use requires instead of specialization for doris::vectorized::Decimal (#28027)
Use requires instead of specialization for doris::vectorized::Decimal
2023-12-08 09:59:52 +08:00
9461e86b10 [pipelineX](debug) add debug string (#28137)
* [pipelineX](debug) add debug string

* update
2023-12-07 23:21:10 +08:00