database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
20,619 Commits 1 Branch 0 Tags
443e87e20327eaa5577cc10f08a63ec1694de358
Commit Graph

76 Commits

Author SHA1 Message Date
meiyi
1c9a798b68 [fix](case) fix insert_group_commit_into (#42397) 2024-10-25 22:15:38 +08:00
hui lai
2984a1b6c8 [improve](stream load) set NumberLoadedRows to zero when stream load transaction failed (#41946) (#42265) 
pick (#41946)

Set NumberLoadedRows to zero when stream load failed.

before:
```
stream load result: {
    "TxnId": 8589,
    "Label": "c8e7c4fe-56b2-4e3b-b4cc-4f2a94cdd003",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 3,
    "NumberLoadedRows": 1,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 77,
    "LoadTimeMs": 78,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 72,
    "ReceiveDataTimeMs": 7,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "XXX"
}
```

now:
```
stream load result: {
    "TxnId": 8589,
    "Label": "c8e7c4fe-56b2-4e3b-b4cc-4f2a94cdd003",
    "Comment": "",
    "TwoPhaseCommit": "false",
    "Status": "Fail",
    "Message": "[DATA_QUALITY_ERROR]too many filtered rows",
    "NumberTotalRows": 3,
    "NumberLoadedRows": 0,
    "NumberFilteredRows": 2,
    "NumberUnselectedRows": 0,
    "LoadBytes": 77,
    "LoadTimeMs": 78,
    "BeginTxnTimeMs": 0,
    "StreamLoadPutTimeMs": 4,
    "ReadDataTimeMs": 0,
    "WriteDataTimeMs": 72,
    "ReceiveDataTimeMs": 7,
    "CommitAndPublishTimeMs": 0,
    "ErrorURL": "XXX"
}
```
 2024-10-25 01:01:46 +08:00
meiyi
4c9ebbb3b9 [fix](cloud) cloud group commit should skip repaly wal if label is already used and the txn state is committed or visible (#41262) (#41461) 
pick https://github.com/apache/doris/pull/41262
 2024-10-10 22:27:04 +08:00
meiyi
070bd6cf99 [fix](jdbc) fix Unknown command(27) (#41621) (#41637) 
pick https://github.com/apache/doris/pull/41621
 2024-10-10 17:46:45 +08:00
Xin Liao
3249931fa6 [Fix](test) fix flake case in both test_insert_strict_fail_url and test_insert_partition_fail_url #40998 (#41024) 
cherry pick from #40998
 2024-09-20 09:52:09 +08:00
morrySnow
51c8b62d1c [opt](Nereids) fix several insert into related issues (#40467) (#40755) 
pick from master #40467

- http_stream TVF should always generate one fragment plan
- http_stream TVF plan should not check root as scan node
- distinguish group_commit TVF with normal insert statement
- index and generate slot should based on type cast base slot
- agg_state could cast from nullable to non-nullable
- colocated and bucket scan range compute should only on scan node
 2024-09-13 10:19:56 +08:00
abmdocrt
0b9817ea4b [cherry-pick](branch-2.1) Pick "[Enhancement](txn) Block new insert into if schema change happens during transaction (#39483)" (#40115) 
## Proposed changes

Pick #39483 

<!--Describe your changes.-->
 2024-08-30 10:01:10 +08:00
meiyi
eea3676791 [fix](group commit) fix group commit insert rpc may stuck (#39391) (#39458) 
pick https://github.com/apache/doris/pull/39391
 2024-08-16 13:19:00 +08:00
meiyi
fff26fe2fc [fix](group commit) fix group commit core if be inject FragmentMgr.exec_plan_fragment.failed (#39339) (#39396) 
pick https://github.com/apache/doris/pull/39339
 2024-08-15 17:54:11 +08:00
meiyi
4668ebd1f6 [fix](regression) fix test_group_commit_timeout failed due to different error message (#38995) (#39068) 
pick https://github.com/apache/doris/pull/38995
 2024-08-08 14:37:10 +08:00
meiyi
749c9f7b56 [fix](group commit) fix repaly wal check label status (#38883) (#38997) 
pick https://github.com/apache/doris/pull/38883
 2024-08-07 22:06:59 +08:00
meiyi
aaee1d9bbd [fix](regression) fix prepare_insert when execute prepare stmt in observer fe (#38545) (#38850) 
pick https://github.com/apache/doris/pull/38545
 2024-08-05 13:45:13 +08:00
morrySnow
f24d55fc94 [fix](syntax) multi statements must delim with semicolon (#38670) (#38753) 
pick from master #38670
 2024-08-02 14:49:51 +08:00
meiyi
e8690b62ee [fix](group commit) Pick add debug log show why group commit not work; delete wal when replay success (#38611) (#38659) 
Pick https://github.com/apache/doris/pull/38611
 2024-08-01 16:59:54 +08:00
924060929
a328e01d97 [fix](nereids) fix insert stmt throw MultiCastDataSink cannot be cast to DataStreamSink (#38526) (#38547) 
fix `insert ... with ... select ...`, which not use some cte, and throw an exception:
```
errCode = 2, detailMessage = class org.apache.doris.planner.MultiCastDataSink cannot be cast to class org.apache.doris.planner.DataStreamSink (org.apache.doris.planner.MultiCastDataSink and org.apache.doris.planner.DataStreamSink are in unnamed module of loader 'app')
```
 2024-07-31 00:47:30 +08:00
meiyi
91f0301b43 [fix](group commit) Pick some group commit pr (#38320) 
Pick https://github.com/apache/doris/pull/38292,
https://github.com/apache/doris/pull/34021,
https://github.com/apache/doris/pull/38228, some modify of
https://github.com/apache/doris/pull/37260, some modify of
https://github.com/apache/doris/pull/37595
 2024-07-25 17:32:44 +08:00
meiyi
e23c1339a8 [fix](group commit) Fix prepare stmt setNull return too many filtered rows error (#38262) (#38276) 
Pick https://github.com/apache/doris/pull/38262
 2024-07-24 19:02:59 +08:00
Kaijie Chen
47096f2083 [test](regression) add cases for data quality error url (#34987) (#37777) 
cherry-pick #34987
 2024-07-16 11:12:52 +08:00
abmdocrt
2d5eb55310 [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add some partial update case (#36385)" (#37382) 
#36385
 2024-07-09 10:52:04 +08:00
meiyi
1e3ab0ff8c [fix](group commit) Pick make group commit cancel in time (#36249) (#37404) 
pick https://github.com/apache/doris/pull/36249/
 2024-07-09 09:25:11 +08:00
abmdocrt
7d423b3a6a [chery-pick](branch-2.1) Pick "[Fix](group commit) Fix group commit block queue mem estimate fault" (#37379) 
Pick [Fix](group commit) Fix group commit block queue mem estimate faule
#35314

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

**Problem:** When `group commit=async_mode` and NULL data is imported
into a `variant` type column, it causes incorrect memory statistics for
group commit backpressure, leading to a stuck issue. **Cause:** In group
commit mode, blocks are first added to a queue in batches using `add
block`, and then blocks are retrieved from the queue using `get block`.
To track memory usage during backpressure, we add the block size to the
memory statistics during `add block` and subtract the block size from
the memory statistics during `get block`. However, for `variant` types,
during the `add block` write to WAL, serialization occurs, which can
merge types (e.g., merging `int` and `bigint` into `bigint`), thereby
changing the block size. This results in a discrepancy between the block
size during `get block` and `add block`, causing memory statistics to
overflow.
**Solution:** Record the block size at the time of `add block` and use
this recorded size during `get block` instead of the actual block size.
This ensures consistency in the memory addition and subtraction.

## Further comments

If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
 2024-07-07 18:27:49 +08:00
lihangyu
14c991f09b [Feature](Prepared Statement) fix and enable enable_server_side_prepared_statement by default #36581 (#36818) 
picked from #36581
 2024-07-01 14:35:17 +08:00
xy720
c5bb0e3a21 [bug](prepared statement) fix prepared statement throw exception when inserting null value (#36484) 
## Proposed changes

bp #36426

<!--Describe your changes.-->
 2024-06-20 11:31:59 +08:00
Kaijie Chen
2392477f76 [test](shuffle) test insert row count when rows filtered by ExchangeNode (#34657) 2024-05-11 11:47:49 +08:00
meiyi
7c52b9f36e [fix](txn insert) Txn insert stuck if execute on observer fe (#34547) 2024-05-10 22:05:53 +08:00
meiyi
5abc84af71 [fix](txn insert) Fix txn insert commit failed when schema change (#33706) 2024-04-19 15:01:57 +08:00
meiyi
9ada38327b [feature](txn insert) txn insert support insert into select (#31666) 2024-04-12 10:11:22 +08:00
meiyi
741d4ff97e [fix](group commit) Fix syntax error when insert into table which column names contain keyword (#33322) 2024-04-10 16:22:09 +08:00
meiyi
d640c54b80 [fix](regression) prepare_insert failed when connect to observer fe (#32223) 2024-03-21 14:07:24 +08:00
abmdocrt
62db7094ea Revert "Problem: When the old optimizer processes an INSERT INTO statement that contains two quotation marks, it results in only one quotation mark being written into the database. (#31890)" (#31986) 
This reverts commit 8c309652e04698f311b6c9158105352e8416c69a.
 2024-03-09 19:45:46 +08:00
feiniaofeiafei
5909237ab1 [Fix](nereids) Add semantic check that the hash bucket column must be a key column when creating table for aggregate and unique models (#31951) 2024-03-09 19:45:46 +08:00
abmdocrt
5b52812af2 Problem: When the old optimizer processes an INSERT INTO statement that contains two quotation marks, it results in only one quotation mark being written into the database. (#31890) 
Reason: During syntax parsing, the old optimizer interprets two quotation marks as a single quotation mark.
Solution: Remove the logic that consolidates two quotation marks into one.
 2024-03-09 19:43:21 +08:00
meiyi
ef22bd3318 [fix](txn insert) Txn insert can not write to table with mv (#31167) 2024-02-22 13:01:49 +08:00
meiyi
ca89ac0c23 [fix](regression) insert_group_commit_into_unique is unstable when cluster has multi BE (#31188) 2024-02-21 19:18:45 +08:00
huanghaibin
1e3968fe7e [fix](group_commit) Need to wait wal to be deleted when creating MaterializedView (#30956) 2024-02-21 13:53:19 +08:00
morrySnow
e93a16ac6e [fix](Nereids) support complex literal cast in fe (#29599) 2024-01-12 11:59:52 +08:00
abmdocrt
1ea51e9f20 [Feature](group commit) Support table property "group commit data bytes" (#29484) 2024-01-07 19:46:42 +08:00
meiyi
9be0f04506 (improv)[group commit] refactor some group commit code (#29180) 2023-12-29 00:26:10 +08:00
meiyi
0af6bd6390 [fix](group-commit) check if wal need recovery is abnormal (#28769) 2023-12-22 11:06:11 +08:00
yujun
34fd376f33 [fix](publish version) fix publish fail but return ok (#28425) 2023-12-21 11:10:08 +08:00
meiyi
ee24667b9f [fix](group commit) Fix some group commit problems (#28319) 2023-12-14 14:38:56 +08:00
meiyi
eda2818b95 [fix](group commit) Fix insert_group_commit_into_unique_sync_mode (#28283) 2023-12-13 01:12:57 +08:00
meiyi
45b2dbab6a [improve](group commit) Group commit support max filter ratio when rows is less than value in config (#28139) 2023-12-12 16:33:36 +08:00
meiyi
d4f89389e3 [improve](group commit) Group commit support skip wal (#27957) 2023-12-11 19:38:32 +08:00
huanghaibin
61379b141e [fix](insert) fix group commit regression test (#28142) 2023-12-09 16:24:20 +08:00
meiyi
1f9aa8ab16 [fix](group commit) Fix some group commit problems (#27769) 2023-11-29 23:43:21 +08:00
meiyi
7087250b4a [fix](insert) txn insert and group commit should write \N string corr… (#27637) 2023-11-28 17:32:50 +08:00
abmdocrt
5d31bc99b8 [Fix](Group_commit) Fix group commit regression test failure (#27475) 2023-11-23 23:03:38 +08:00
abmdocrt
96e3636905 [Fix](Group_commit) Fix group commit regression test failure (#27380) 2023-11-22 18:55:20 +08:00
abmdocrt
1cd1c58eee [Feature](group commit) move group_commit_interval_ms from be.conf to table property (#27116) 2023-11-21 20:50:02 +08:00