Commit Graph

27 Commits

Author SHA1 Message Date
385589c92b [fix](regression)Fix unstable compaction related cases (#46920) (#47003)
bp #46920
2025-01-15 13:42:04 +08:00
c9ab3b9e3a [regression-test](fix) ignore E-2000 in compaction cases (#46522)
### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx
#46436
2025-01-07 14:26:54 +08:00
303f6bd849 [test](regression-test) use unified trigger_and_wait_compaction metho… (#45908) 2024-12-25 13:59:51 +08:00
5f4d1931ff [feature](ES Catalog)Add FE open API for ES Catalog (#40444) (#40526)
bp #40444
2024-09-09 14:25:43 +08:00
688b97e183 [enhancement](Nereids) support reuse sql cache between different comment (#40065)
cherry pick from #40049
2024-08-28 22:15:55 +08:00
e95e777b34 [opt](test)Add expected indices count check (#39024) (#39060)
## Proposed changes

bp #39024
2024-08-09 01:37:44 +08:00
f21d7e3833 [test](inverted index)Add cases for inverted index format v2 (#38132)(#38443) (#38222)
## Proposed changes

backport #38132 #38443
2024-08-02 12:04:26 +08:00
4d980b8235 [feature](http action)Add http action to show nested inverted index file (#38272) (#38672)
backport #38272
2024-08-01 19:30:59 +08:00
6214d6421f [Fix](planner) fix bug of char(255) toSql (#37340) (#37671)
cherry-pick #37340 from master
2024-07-12 10:33:24 +08:00
238e218312 [fix](httpapi) restore compaction/run_status api can show be's overall compaction status and refactor code (#35409) 2024-05-28 09:43:43 +08:00
e02dcecb0a [optimize](regression)Add retry for curl request (#35260)
Co-authored-by: Luennng <luennng@gmail.com>
2024-05-24 16:23:58 +08:00
15f8014e4e [enhancement](Nereids) Enable parse sql from sql cache and fix some bugs (#33867)
* [enhancement](Nereids) Enable parse sql from sql cache (#33262)

Before this pr, the query must pass through parser, analyzer, rewriter, optimizer and translator, then we can check whether this query can use sql cache, if the query is too long, or the number of join tables too big, the plan time usually >= 500ms.

This pr reduce this time by skip the fashion plan path, because we can reuse the previous physical plan and query result if no any changed. In some cases we should not parse sql from sql cache, e.g. table structure changed, data changed, user policies changed, privileges changed, contains non-deterministic functions, and user variables changed.

In my test case: query a view which has lots of join and union, and the tables has empty partition, the query latency is about 3ms. if not parse sql from sql cache, the plan time is about 550ms

## Features
1. use Config.sql_cache_manage_num to control how many sql cache be reused in on fe
2. if explain plan appear some plans contains `LogicalSqlCache` or `PhysicalSqlCache`, it means the query can use sql cache, like this:
```sql
mysql> set enable_sql_cache=true;
Query OK, 0 rows affected (0.00 sec)

mysql> explain physical plan select * from test.t;
+----------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                  |
+----------------------------------------------------------------------------------+
| cost = 3.135                                                                     |
| PhysicalResultSink[53] ( outputExprs=[c1#0, c2#1] )                              |
| +--PhysicalDistribute[50]@0 ( stats=3, distributionSpec=DistributionSpecGather ) |
|    +--PhysicalOlapScan[t]@0 ( stats=3 )                                          |
+----------------------------------------------------------------------------------+
4 rows in set (0.02 sec)

mysql> select * from test.t;
+------+------+
| c1   | c2   |
+------+------+
|    1 |    2 |
|   -2 |   -2 |
| NULL |   30 |
+------+------+
3 rows in set (0.05 sec)

mysql> explain physical plan select * from test.t;
+-------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                           |
+-------------------------------------------------------------------------------------------+
| cost = 0.0                                                                                |
| PhysicalSqlCache[2] ( queryId=78511f515cda466b-95385d892d6c68d0, backend=127.0.0.1:9050 ) |
| +--PhysicalResultSink[52] ( outputExprs=[c1#0, c2#1] )                                    |
|    +--PhysicalDistribute[49]@0 ( stats=3, distributionSpec=DistributionSpecGather )       |
|       +--PhysicalOlapScan[t]@0 ( stats=3 )                                                |
+-------------------------------------------------------------------------------------------+
5 rows in set (0.01 sec)
```

(cherry picked from commit 03bd2a337d4a56ea9c91673b3bd4ae518ed10f20)

* fix

* [fix](Nereids) fix some sql cache consistence bug between multiple frontends (#33722)

fix some sql cache consistence bug between multiple frontends which introduced by [enhancement](Nereids) Enable parse sql from sql cache #33262, fix by use row policy as the part of sql cache key.
support dynamic update the num of fe manage sql cache key

(cherry picked from commit 90abd76f71e73702e49794d375ace4f27f834a30)

* [fix](Nereids) fix bug of dry run query with sql cache (#33799)

1. dry run query should not use sql cache
2. fix test sql cache in cloud mode
3. enable cache OneRowRelation and EmptyRelation in frontend to skip parse sql

(cherry picked from commit dc80ecf7f33da7b8c04832dee88abd09f7db9ffe)

* remove cloud mode

* remove @NotNull
2024-04-19 15:22:14 +08:00
44ba9e102c [feature](statistics)support statistics for iceberg/paimon/hudi table (#29868) 2024-01-18 12:03:07 +08:00
f9ddf8c7ef [improvement](be report) add be report http (#28424) 2023-12-19 10:39:19 +08:00
f236261256 [fix](regression) compaction cases adapt force_olap_table_replica_num option (#28136) 2023-12-11 10:08:21 +08:00
e695cb18b1 [improvement](regression test) add fe debug point (#28047) 2023-12-06 18:18:05 +08:00
8b33b0c4a4 [Fix](row store) cache invalidate key should not include sequence column (#26771) 2023-11-11 01:30:32 -06:00
8bb6606662 [fix](nestedType)fix nested data type to create table (#26506) 2023-11-09 14:38:41 +08:00
1e02de1846 [feature](nestedType)add nested data type to create table (#26129) 2023-10-31 13:35:29 +08:00
81dd00f6e4 [Feature](Compaction) Support do full compaction by table id (#22010) 2023-08-21 11:54:51 +08:00
adc44d9f46 [regression-test] add list partition case and multi partition keys case (#22042)
* [regression-test] add list partition case and multi partition keys case

* fix delete failed
2023-07-28 10:12:35 +08:00
c409fa0f58 [Feature](Compaction)Support full compaction (#21177) 2023-07-16 13:21:15 +08:00
4a682a0a46 [fix][regression-test] set timeout of curl in regression test to avoid hanged when be crashed. (#20222)
Currently in regression-test, when a be crash, because curl does not set a timeout, suite-thread will get stuck.
To solve this, encapsulate the call to be into a function, set the timeout uniformly, and avoid getting stuck
2023-06-01 11:00:09 +08:00
a713c225a5 [regressiontest](statistics) Collate and supplement statistics regression test (#19901)
This pr is mainly supplement statistics regression test. include the following:

analyze stats p0 tests:

1. Universal analysis

analyze stats p1 tests:

1. Universal analysis
2. Sampled analysis
3. Incremental analysis
4. Automatic analysis
5. Periodic analysis

manage stats p0 tests:

1. Alter table stats
2. Show table stats
3. Alter column stats
4. Show column stats and histogram
5. Drop column stats
6. Drop expired stats

TODO:

1. Supplement related documents
2. Optimize for unstable cases encountered during testing
3. Add other cases

For pr related to statistics, should ensure that all of these cases pass!
2023-05-24 20:17:28 +08:00
1938899aa3 [regression-test] add grouping sets test case (#18194) 2023-03-31 11:00:38 +08:00
d37cf0a41b [regression-test](p0) add p0 test cases (#11624)
Add p0 test cases, including:
aggregate
join
union
order by
group by
keyword
arithmetic operators
logical operators
case function
coalesce
between
in
like
limit
where
regexp
window function
runtime filter
schema change
2022-08-15 23:12:07 +08:00
e6c3557b1b [improvement](regression-test) Support suite plugin to add third-part… (#9294)
Support register suite plugin to add third-party function.

See 
1. register in: ${DORIS_HOME}/regression-test/plugins/plugin_example.groovy
2. usage: ${DORIS_HOME}/regression-test/suites/demo/test_plugin.groovy
3. doc: ${DORIS_HOME}/docs/zh-CN/developer-guide/regression-testing.md
2022-05-05 20:45:45 +08:00