Commit Graph

54 Commits

Author SHA1 Message Date
6531e4c540 [improve](regression test)Add test for time series compact empty rowset (#29509) 2024-04-24 17:13:49 +08:00
6c30695eda [enhancement](regressiontest) fix unstable regression tests (#33939)
Co-authored-by: yiguolei <yiguolei@gmail.com>
2024-04-22 11:28:22 +08:00
5e73d7a281 [fix](compaction) fix incorrect grouping of vertical compaction columns in tables only with key columns (#32896) (#33470) 2024-04-10 16:04:33 +08:00
797b8fa456 [FIX](agg) fix vertical_compaction_reader for agg table with array/map type (#33130) 2024-04-03 18:09:45 +08:00
ea2fbfaffa [feature](Nereids) support agg state type in create table (#32171)
this PR introduce a behavior change, syntax of create table with agg_state type is changed.
2024-03-15 18:04:49 +08:00
Pxl
1d39e16eda [Bug](compaction) pass arena to function->add_batch_range (#30709) 2024-02-04 14:28:38 +08:00
1f754c55d5 [chore](show replica) show replica print path (#30402) 2024-02-01 19:00:50 +08:00
f236261256 [fix](regression) compaction cases adapt force_olap_table_replica_num option (#28136) 2023-12-11 10:08:21 +08:00
97105e9a16 [regression](compaction) Add case to test single replica compaction (#27199) 2023-11-30 21:27:13 +08:00
553e4a8903 [feature-wip](merge-on-write) MOW table support different primary keys and sort keys (#24788) 2023-11-24 16:37:30 +08:00
d5908b0c8b [regression](compaction) Add case to test base_compaction with dup_key_max_file_size_limit (#27088) 2023-11-24 10:27:51 +08:00
5c9785045f [Fix](Full compaction) Fix local variable using in test case (#24440) 2023-09-17 00:08:32 +08:00
68b13ab50f [Fix](Full compaction) Fix full compaction by table id case (#24265) 2023-09-14 18:03:28 +08:00
c31cb5fd11 [enhance] use correct default value for show config action (#19284) 2023-09-01 11:28:26 +08:00
84006dd8c7 [Fix](Full compaction) Fix full compaction regressison test (#23487) 2023-08-29 18:27:19 +08:00
e1367d509f [Fix](Full compaction) Fix full compaction by table id regressison test #23496 2023-08-25 18:07:06 +08:00
81dd00f6e4 [Feature](Compaction) Support do full compaction by table id (#22010) 2023-08-21 11:54:51 +08:00
b91bb9f503 [fix](alter table property) fix alter property if rpc failed (#22845)
* fix alter property

* add regression case

* do not repeat
2023-08-17 18:02:34 +08:00
df26fb2de4 [fix][alter table property] fix alter table property failed (#22791) 2023-08-10 17:12:42 +08:00
f036cdfde6 [feature](compaction) support delete in cumulative compaction (#19609) 2023-08-07 15:22:21 +08:00
c01b6e0ba7 [fix](regression test) redefine variable in the test_table_level_compaction_policy (#22604)
redefine variable in the test_table_level_compaction_policy
2023-08-05 12:55:06 +08:00
c63e3e6959 [fix](regression] fix test_table_level_compaction_policy
[fix](regression] fix test_table_level_compaction_policy
2023-08-03 15:24:17 +08:00
19d1f49fbe [improvement](compaction) compaction policy and options in the properties of a table (#22461) 2023-08-01 22:02:23 +08:00
c409fa0f58 [Feature](Compaction)Support full compaction (#21177) 2023-07-16 13:21:15 +08:00
4bee226698 [fix](regression-test) fix compile test_vertical_compaction_agg_keys failed (#20792)
fix compile test_vertical_compaction_agg_keys failed.
2023-06-14 23:25:17 +08:00
ce3050d75c [fix](regression) fix vertical compaction test (#20601) 2023-06-13 10:31:22 +08:00
01770ba68a [fix](regression-test) variable's scope returned by curl (#20347) 2023-06-01 23:38:39 +08:00
05b7c65509 [fix](regression-test) fix multi-thread problem of regression-test #20322 2023-06-01 18:57:17 +08:00
4a682a0a46 [fix][regression-test] set timeout of curl in regression test to avoid hanged when be crashed. (#20222)
Currently in regression-test, when a be crash, because curl does not set a timeout, suite-thread will get stuck.
To solve this, encapsulate the call to be into a function, set the timeout uniformly, and avoid getting stuck
2023-06-01 11:00:09 +08:00
a7f3bfec89 [refactor](cluster)(step-2) remove cluster related to Backend (#19842) 2023-05-21 09:00:35 +08:00
6748ae4a57 [Feature] Collect the information statistics of the query hit (#18805)
1. Show the query hit statistics for `baseall`

   ```sql
    MySQL [test_query_db]> show query stats from baseall;
    +-------+------------+-------------+
    | Field | QueryCount | FilterCount |
    +-------+------------+-------------+
    | k0    | 0          | 0           |
    | k1    | 0          | 0           |
    | k2    | 0          | 0           |
    | k3    | 0          | 0           |
    | k4    | 0          | 0           |
    | k5    | 0          | 0           |
    | k6    | 0          | 0           |
    | k10   | 0          | 0           |
    | k11   | 0          | 0           |
    | k7    | 0          | 0           |
    | k8    | 0          | 0           |
    | k9    | 0          | 0           |
    | k12   | 0          | 0           |
    | k13   | 0          | 0           |
    +-------+------------+-------------+
    14 rows in set (0.002 sec)

    MySQL [test_query_db]> select k0, k1,k2, sum(k3) from baseall  where k9 > 1 group by k0,k1,k2;
    +------+------+--------+-------------+
    | k0   | k1   | k2     | sum(`k3`)   |
    +------+------+--------+-------------+
    |    0 |    6 |  32767 |        3021 |
    |    1 |   12 |  32767 | -2147483647 |
    |    0 |    3 |   1989 |        1002 |
    |    0 |    7 | -32767 |        1002 |
    |    1 |    8 |    255 |  2147483647 |
    |    1 |    9 |   1991 | -2147483647 |
    |    1 |   11 |   1989 |       25699 |
    |    1 |   13 | -32767 |  2147483647 |
    |    1 |   14 |    255 |         103 |
    |    0 |    1 |   1989 |        1001 |
    |    0 |    2 |   1986 |        1001 |
    |    1 |   15 |   1992 |        3021 |
    +------+------+--------+-------------+
    12 rows in set (0.050 sec)

    MySQL [test_query_db]> show query stats from baseall;
    +-------+------------+-------------+
    | Field | QueryCount | FilterCount |
    +-------+------------+-------------+
    | k0    | 1          | 0           |
    | k1    | 1          | 0           |
    | k2    | 1          | 0           |
    | k3    | 1          | 0           |
    | k4    | 0          | 0           |
    | k5    | 0          | 0           |
    | k6    | 0          | 0           |
    | k10   | 0          | 0           |
    | k11   | 0          | 0           |
    | k7    | 0          | 0           |
    | k8    | 0          | 0           |
    | k9    | 1          | 1           |
    | k12   | 0          | 0           |
    | k13   | 0          | 0           |
    +-------+------------+-------------+
    14 rows in set (0.001 sec)
   ```

2. Show the query hit statistics summary for all the mv in a table

   ```sql
   MySQL [test_query_db]> show query stats from baseall all;
    +-----------+------------+
    | IndexName | QueryCount |
    +-----------+------------+
    | baseall   | 1          |
    +-----------+------------+
    1 row in set (0.005 sec)
   ```

3. Show the query hit statistics detail info for all the mv in a table

   ```sql
    MySQL [test_query_db]> show query stats from baseall all verbose;
    +-----------+-------+------------+-------------+
    | IndexName | Field | QueryCount | FilterCount |
    +-----------+-------+------------+-------------+
    | baseall   | k0    | 1          | 0           |
    |           | k1    | 1          | 0           |
    |           | k2    | 1          | 0           |
    |           | k3    | 1          | 0           |
    |           | k4    | 0          | 0           |
    |           | k5    | 0          | 0           |
    |           | k6    | 0          | 0           |
    |           | k10   | 0          | 0           |
    |           | k11   | 0          | 0           |
    |           | k7    | 0          | 0           |
    |           | k8    | 0          | 0           |
    |           | k9    | 1          | 1           |
    |           | k12   | 0          | 0           |
    |           | k13   | 0          | 0           |
    +-----------+-------+------------+-------------+
    14 rows in set (0.017 sec)
   ```

4. Show the query hit for a database

   ```sql
    MySQL [test_query_db]> show query stats for test_query_db;
    +----------------------------+------------+
    | TableName                  | QueryCount |
    +----------------------------+------------+
    | compaction_tbl             | 0          |
    | bigtable                   | 0          |
    | empty                      | 0          |
    | tempbaseall                | 0          |
    | test                       | 0          |
    | test_data_type             | 0          |
    | test_string_function_field | 0          |
    | baseall                    | 1          |
    | nullable                   | 0          |
    +----------------------------+------------+
    9 rows in set (0.005 sec)
   ```

5. Show query hit statistics for all the databases

   ```sql
    MySQL [(none)]> show query stats;
    +-----------------+------------+
    | Database        | QueryCount |
    +-----------------+------------+
    | test_query_db   | 1          |
    +-----------------+------------+
    1 rows in set (0.005 sec)
   ```
2023-05-15 10:56:34 +08:00
b129c9901b [improvement](FQDN)Change the implementation of fqdn (#19123)
Main changes:
1. If fqdn is enabled in the configuration file, when fe starts, localAddr will obtain fqdn instead of IP, priority_ Networks will fail
2. The IP and host names of Backend and Front are combined into one field, host. When fqdn is enabled, it represents the host name, and when not enabled, it represents the IP address
3. The communication between clusters directly uses fqdn, and various Connection pool add authentication mechanisms to prevent the IP address of the domain name from changing and the connection between nodes from making errors
4. No longer requires polling to verify if the IP has changed, delete fqdnManager
5. Change the method of verifying the legitimacy of nodes between FEs from obtaining client IP to displaying the identity of the transmitting node itself in the HTTP request header or the message body of the throttle
6. When processing the heartbeat, if BE finds that the host stored by itself is inconsistent with the host stored by the master, after verifying the legitimacy of the host, it will change its own host instead of directly reporting an error
7. Simplify the generation logic of fe name

Scope of influence:
1. Establishing communication connections between clusters
2. Determine whether it is the same node through attributes such as IP
3. Print Log
4. Information display
5. Address Splicing
6. k8s deployment
7. Upgrade compatibility

Test plan:
1. Change the IP address of the node, while keeping the fqdn unchanged, change the IP addresses of fe and be, and verify whether the cluster can read and write data normally
2. Use the master code to generate metadata, and use the previous metadata on the current pr to verify whether it is compatible with the old version (upgrading is no longer supported if fqdn has been enabled before)
3. Deploy fe and be clusters using k8s to verify whether the cluster can read and write data normally
4. According to https://doris.apache.org/zh-CN/docs/dev/admin-manual/cluster-management/fqdn?_highlight=fqdn#%E6%97%A7%E9%9B%86%E7%BE%A4%E5%90%AF%E7%94%A8fqdn Upgrading old clusters
5. Use streamload to specify the fqdn of fe and be to import data separately
6. Use different users to start transactions and write data using insert statements
2023-05-11 00:44:48 +08:00
7bc242d665 [regression-test](prepared statement) Fix connection error when test framework used lower jdbc version (#18665) 2023-04-16 18:13:45 +08:00
e562017801 [feature](table-metadata) support altering the property "light_schema_change" for the tables which created before 1.2 (#17704) 2023-04-11 11:09:43 +08:00
a2858bbbfc [fix](regression-test) fix wrong http port in regression-test case compaction-with-delete #18484 2023-04-08 15:17:52 +08:00
851793aa66 [fix](test) fix p0 regression case test_compaction_with_delete failed (#18479) 2023-04-07 22:11:40 +08:00
759f1da32e [Enhencement](Backends) add HostName filed in backends table and delete backends table in information_schema (#18156)
1.  Add `HostName` field for `show backends` statement and `backends()` tvf.
2. delete the `backends` table in `information_schema` database
2023-04-07 08:30:42 +08:00
87e83081ff [test](compaction) add delete test (#18335) 2023-04-04 12:28:19 +08:00
eb7da1c0ee [fix](datatype) fix some bugs about data type array datetimev2 and decimalv3 (#16132) 2023-01-29 14:26:08 +08:00
116e17428b [Enhancement](point query optimize) improve performace of point query on primary keys (#15491)
1. support row format using codec of jsonb
2. short path optimize for point query
3. support prepared statement for point query
4. support mysql binary format
2023-01-20 13:33:01 +08:00
90be1a22a9 [bugfix](vertical compaction) fix dcheck failed in MOW tablet (#15638)
fix a dcheck error for vertical compaction on Merge-On-Write table。
When merge rowsets with empty segment, VerticalHeapMergeIterator::init
return ok directly and _record_rowids not set, dcheck failed when
_unique_key_next_block call current_block_row_locations。
2023-01-08 10:39:52 +08:00
4dbe30d37b [regression](vectorized) delete vectorized config in regression tests (#15126) 2022-12-16 17:08:29 +08:00
94a6ffb906 [feature](compaction) support vertical_compaction & ordered_data_compaction (#14524) 2022-12-01 22:15:41 +08:00
b85c78ee00 [fix](regression) add 'if not exists' to 'create table' to support parallel test (#13576) (#13578)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2022-10-25 16:37:07 +08:00
32b1456b28 [feature-wip](array) remove array config and check array nested depth (#13428)
1. remove FE config `enable_array_type`
2. limit the nested depth of array in FE side.
3. Fix bug that when loading array from parquet, the decimal type is treated as bigint
4. Fix loading array from csv(vec-engine), handle null and "null"
5. Change the csv array loading behavior, if the array string format is invalid in csv, it will be converted to null. 
6. Remove `check_array_format()`, because it's logic is wrong and meaningless
7. Add stream load csv test cases and more parquet broker load tests
2022-10-20 15:52:31 +08:00
588dc5f12a [feature](cold_on_s3) Show remote data usage via SHOW BACKENDS and SHOW TABLETS statements (#11450) 2022-08-25 15:36:15 +08:00
1f9eec5462 [Regression](datev2) Add test cases for datev2/datetimev2 (#11831) 2022-08-19 10:57:55 +08:00
f2292a3b1d [Enhancement](array-type) enable_array_type flag update (#11785)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-08-16 14:41:57 +08:00
7be2ef79ed array column support read by rowids (#10886)
Co-authored-by: cambyzju <zhuxiaoli01@baidu.com>
2022-07-15 21:19:02 +08:00
9c990b073f [regression] modify compaction cases, not depend on beHttpAddress (#10553) 2022-07-04 22:36:12 +08:00