7d423b3a6a
Pick [Fix](group commit) Fix group commit block queue mem estimate faule #35314 ## Proposed changes Issue Number: close #xxx <!--Describe your changes.--> **Problem:** When `group commit=async_mode` and NULL data is imported into a `variant` type column, it causes incorrect memory statistics for group commit backpressure, leading to a stuck issue. **Cause:** In group commit mode, blocks are first added to a queue in batches using `add block`, and then blocks are retrieved from the queue using `get block`. To track memory usage during backpressure, we add the block size to the memory statistics during `add block` and subtract the block size from the memory statistics during `get block`. However, for `variant` types, during the `add block` write to WAL, serialization occurs, which can merge types (e.g., merging `int` and `bigint` into `bigint`), thereby changing the block size. This results in a discrepancy between the block size during `get block` and `add block`, causing memory statistics to overflow. **Solution:** Record the block size at the time of `add block` and use this recorded size during `get block` instead of the actual block size. This ensures consistency in the memory addition and subtraction. ## Further comments If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc... ## Proposed changes Issue Number: close #xxx <!--Describe your changes.-->
44 lines
1.5 KiB
Groovy
44 lines
1.5 KiB
Groovy
// Licensed to the Apache Software Foundation (ASF) under one
|
|
// or more contributor license agreements. See the NOTICE file
|
|
// distributed with this work for additional information
|
|
// regarding copyright ownership. The ASF licenses this file
|
|
// to you under the Apache License, Version 2.0 (the
|
|
// "License"); you may not use this file except in compliance
|
|
// with the License. You may obtain a copy of the License at
|
|
//
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
//
|
|
// Unless required by applicable law or agreed to in writing,
|
|
// software distributed under the License is distributed on an
|
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
// KIND, either express or implied. See the License for the
|
|
// specific language governing permissions and limitations
|
|
// under the License.
|
|
|
|
suite("test_group_commit_variant") {
|
|
|
|
sql "set group_commit=async_mode"
|
|
|
|
def testTable = "test_group_commit_variant"
|
|
|
|
sql """
|
|
CREATE TABLE IF NOT EXISTS ${testTable} (
|
|
k bigint,
|
|
var variant
|
|
)
|
|
UNIQUE KEY(`k`)
|
|
DISTRIBUTED BY HASH (`k`) BUCKETS 5
|
|
properties("replication_num" = "1",
|
|
"disable_auto_compaction" = "false");
|
|
"""
|
|
|
|
try {
|
|
sql "insert into ${testTable} (k) values (1),(2);"
|
|
sql "insert into ${testTable} (k) values (3),(4);"
|
|
} catch (Exception e) {
|
|
// should not throw exception
|
|
logger.info(e.getMessage())
|
|
assertTrue(False)
|
|
}
|
|
}
|