Commit Graph

24 Commits

Author SHA1 Message Date
27eed5399d [Fix](auto-inc) Fix partial update auto inc publish case failure #31987 2024-03-12 14:07:00 +08:00
609761567c [Fix](partial-update) Fix wrong column number passing to BE when partial and enable nereids (#31461)
* Problem:
Inconsistent behavior occurs when executing partial column update `UPDATE` statements and `INSERT` statements on merge-on-write tables with the Nereids optimizer enabled. The number of columns passed to BE differs; `UPDATE` operations incorrectly pass all columns, while `INSERT` operations correctly pass only the updated columns.

Reason:
The Nereids optimizer does not handle partial column update `UPDATE` statements properly. The processing logic for `UPDATE` statements rewrites them as equivalent `INSERT` statements, which are then processed according to the logic of `INSERT` statements. For example, assuming a MoW table structure with columns k1, k2, v1, v2, the correct rewrite should be:
* `UPDATE` table t1 set v1 = v1 + 1 where k1 = 1 and k2 = 2
 * =>
 * `INSERT` into table (v1) select v1 + 1 from table t1 where k1 = 1 and k2 = 2

However, the actual rewriting process does not consider the logic for partial column updates, leading to all columns being included in the `INSERT` statement, i.e., the result is:
* `INSERT` into table (k1, k2, v1, v2) select k1, k2, v1 + 1, v2 from table t1 where k1 = 1 and k2 = 2

This results in `UPDATE` operations incorrectly passing all columns to BE.

Solution:
Having analyzed the cause, the solution is straightforward: when rewriting partial column update `UPDATE` statements to `INSERT` statements, only retain the updated columns and all key columns (as partial column updates must include all key columns). Additionally, this PR includes error injection cases to verify the number of columns passed to BE is correct.

* 2

* 3

* 4

* 5
2024-03-09 19:45:42 +08:00
7c30cb20fd [Fix](partial update) Fix partial update load false when schema includes auto increment column (#31725)
Problem:
When partially updating columns without specifying the auto-increment column, and the imported data contains new keys, an error stating the auto-increment column could not be found occurs.

Reason:
The logic for partial column updates does not account for new keys in auto-increment columns. Since auto-increment columns can be generated by the system, it's possible to omit this column data during import. However, partial column updates treat this as a regular column, expecting it to be nullable or have a default value for automatic filling, overlooking the fact that auto-increment columns can also be auto-filled. This oversight leads to the error.

Solution:
Incorporate a check for auto-increment columns into the partial column update logic, and include the logic for generating auto-increment column values in the process of completing partial updates.
2024-03-06 13:06:27 +08:00
82bb3ed50f [Fix](group commit) Fix pre allocated err handling for group commit async load and add regression test #30718 2024-02-02 13:31:47 +08:00
1ac5b45180 [fix](invert index) fixed the issue of insufficient index idx generation during partial column updates. (#30678) 2024-02-01 19:01:08 +08:00
5e4674ab66 [fix](partial update) mishandling of exceptions in the publish phase may result in data loss (#30366) 2024-01-27 09:09:02 +08:00
c9a949130b [Case](wal) Add wal group commit sink case with low disk space fault injection (#29731) 2024-01-12 11:59:52 +08:00
87023d3b7a [Fix](inverted index) fix memory leak in inverted index when encountering fault (#29676) 2024-01-12 11:44:21 +08:00
53f1521308 [feature](inverted index)Support failover when index compaction failed (#29553) 2024-01-12 11:33:15 +08:00
0d0b9d64dd [improve](move-memtable) add move memtable too many segments fault injection (#29342) 2024-01-03 21:27:54 +08:00
1edf5b31b6 [Regression-test](wal) Add fault injection case for wal mem back pressure (#29298) 2024-01-03 00:06:52 +08:00
17268346d3 [Revert](partial update) Revert "Fix missing rowsets during doing alignment when flushing memtable due to compaction (#28062)" (#28674)
This reverts commit 485d7db5161efb011b43a10237df1ca83d5e5c3d.
2023-12-20 10:13:43 +08:00
e362bf674f [regression-test](memtable) test memtable flush is high priority for vtable writerV1 (#28502) 2023-12-19 12:33:12 +08:00
61de49c727 [case](regression) Test duplicated load id (#28251)
Co-authored-by: qinhao <qinhao@newland.com.cn>
2023-12-16 22:41:51 +08:00
8ab63a9434 [regression-test][memtable] test memtable flush is high priority for vtable writerV2 (#28503) 2023-12-16 22:29:06 +08:00
9b8de017df [Regression test](inverted index) fix regression case for index_compound_directory_fault_injection (#28232) 2023-12-11 19:17:28 +08:00
485d7db516 [fix](partial update) Fix missing rowsets during doing alignment when flushing memtable due to compaction (#28062) 2023-12-10 12:09:48 +08:00
055b3885c9 [Fix](inverted index) fix compound directory flush buffer error (#28191) 2023-12-09 00:57:35 +08:00
b93dd1d5f7 [enhancement](load) improve error msg for load when cancelled by mem gc (#26809)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-11-28 17:36:11 +08:00
da71fde066 [fix](build index) Fix inverted index hardlink leak and missing problem (#26903) 2023-11-24 10:30:21 +08:00
b63f00987f [regression](invered index) add test null index case and fix fault injection index case (#27265) 2023-11-21 14:04:57 +08:00
273cbfc36c [fix](inverted index) reset fs_writer to nullptr before throw exception (#27202) 2023-11-20 17:40:56 +08:00
0929ad568f [fix](regression) move fault-injection data to the right place (#26618)
Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
2023-11-09 12:22:13 +08:00
f883d1acfa [enhancement](regression) fault injection for segcompaction test (#25709)
1. generalized debug point facilities from docker suites for
   fault-injection/stubbing cases
2. add segcompaction fault-injection cases for demonstration
3. add -238 TOO_MANY_SEGMENTS fault-injection case for good
2023-10-30 17:36:17 +08:00