Commit Graph

60 Commits

Author SHA1 Message Date
2d5eb55310 [cherry-pick](branch-2.1) Pick "[Enhancement](partial update) Add some partial update case (#36385)" (#37382)
#36385
2024-07-09 10:52:04 +08:00
c66975b244 [branch-2.1] Pick "[Fix](regression) Fix p0 case test_unique_table_auto_inc_concurrent #37048" (#37050)
## Proposed changes

pick https://github.com/apache/doris/pull/37048
2024-07-02 11:15:50 +08:00
4210a6a8d6 [branch-2.1] PIck "[Fix](autoinc) Hanlde the processing of auto_increment column on exchange node rather than on TabletWriter when using TABLET_SINK_SHUFFLE_PARTITIONED #36836" (#37029)
## Proposed changes

pick https://github.com/apache/doris/pull/36836
2024-07-01 09:56:30 +08:00
12dddfc26c [branch-2.1] Pick "[Fix](autoinc) try fix concurrent load problem with auto inc column #36421" (#37027)
## Proposed changes

pick https://github.com/apache/doris/pull/36421
2024-06-30 13:10:03 +08:00
42425808a1 [Cherry-Pick](branch-2.1) Pick "Fix multiple replica partial update auto inc data inconsistency problem #34788" (#35056)
* [Fix](auto inc) Fix multiple replica partial update auto inc data inconsistency problem (#34788)

* **Problem:** For tables with auto-increment columns, updating partial columns can cause data inconsistency among replicas.

**Cause:** Previously, the implementation for updating partial columns in tables with auto-increment columns was done independently on each BE (Backend), leading to potential inconsistencies in the auto-increment column values generated by each BE.

**Solution:** Before distributing blocks, determine if the update involves partial columns of a table with an auto-increment column. If so, add the auto-increment column to the last column of the block. After distributing to each BE, each BE will check if the data key for the partial column update exists. If it exists, the previous auto-increment column value is used; if not, the auto-increment column value from the last column of the block is used. This ensures that the auto-increment column values are consistent across different BEs.

* 2

* [Fix](regression-test) Fix auto inc partial update unstable regression test (#34940)
2024-05-20 15:43:46 +08:00
75470ede1a [fix](test) Fix some testcases #34203 2024-04-30 08:35:03 +08:00
4079a7b6ab [fix](txn insert) Fix txn insert into values for sequence column or column name is keyword (#33336) 2024-04-10 16:21:31 +08:00
0537952573 [Fix](auto-inc) Fix auto inc unique table case (#32393) 2024-03-22 08:52:16 +08:00
6acdd9cd48 The issue introduced by the recently added auto-increment column works fine on a single node but may result in discontinuous auto-increment IDs when running on a cluster. This PR has been modified to check for the uniqueness of the auto-increment column values instead of checking for equality to a fixed value. (#32115) 2024-03-12 21:51:36 +08:00
7c30cb20fd [Fix](partial update) Fix partial update load false when schema includes auto increment column (#31725)
Problem:
When partially updating columns without specifying the auto-increment column, and the imported data contains new keys, an error stating the auto-increment column could not be found occurs.

Reason:
The logic for partial column updates does not account for new keys in auto-increment columns. Since auto-increment columns can be generated by the system, it's possible to omit this column data during import. However, partial column updates treat this as a regular column, expecting it to be nullable or have a default value for automatic filling, overlooking the fact that auto-increment columns can also be auto-filled. This oversight leads to the error.

Solution:
Incorporate a check for auto-increment columns into the partial column update logic, and include the logic for generating auto-increment column values in the process of completing partial updates.
2024-03-06 13:06:27 +08:00
2cb46eed94 [Feature](auto-inc) Add start value for auto increment column (#30512) 2024-02-16 10:12:23 +08:00
161134913d [fix](regression-test) fix non stable case test_unique_table_like (#29643)
test_unique_table.groovy and test_unique_table_like.groovy both use database test_unique_db.

If they run at the same time, we may got the following errors:

java.sql.SQLException: errCode = 2, detailMessage = There are still some transactions in the COMMITTED state waiting to be completed. The database [default_cluster:test_unique_db] cannot be dropped. If you want to forcibly drop(cannot be recovered), please use "DROP database FORCE".

Both Master & Branch2.0 have this problem.
2024-01-12 11:44:21 +08:00
6503aaf7db [feature](planner) allow HLL and QUANTILE_STATE types on duplicate and unique table (#28546) 2023-12-19 09:54:24 +08:00
088bb80a9c [fix](test) fix case of test_unique_table_new_sequence (#28442)
pr #28105 and #28031 merging conflict caused this case to fail.
2023-12-15 15:10:39 +08:00
8ca7bd8f98 [enhancement](bitmap)support bitmap type for non-key column in duplicate table (#28392) 2023-12-14 23:59:12 +08:00
4300fdce43 [feature](merge-on-write) enable merge-on-write by default again (#28105)
fix #27188, #28096
2023-12-14 19:35:52 +08:00
0868526f6a [fix](sequence column) fix update fail on nereids planner (#28031)
1. if we set enable_fallback_to_original_planner to false, the UPDATE sql in regression case test_unique_table_sequence will fail due to:“Table test_uniq_sequence has sequence column, need to specify the sequence column”,The bug is introduced by [fix](sequence column) insert into should require sequence column in all scenario #27780
2. fix insert fail in transaction mode, which is introduced by [refactor](Nereids) let insert into compatible with legacy planner #27947
3. add cases with session variable enable_fallback_to_original_planner false
2023-12-13 19:41:41 +08:00
dee89d2c4a [refactor](Nereids) let create table compatible with legacy planner (#28078) 2023-12-13 16:35:40 +08:00
8c79b86f5b Revert "[feature](merge-on-write) enable merge-on-write by default (#27188)" (#28096)
This reverts commit 00c8bab84de8154052f9d323800b436cd0ad36e5.
2023-12-07 11:31:36 +08:00
00c8bab84d [feature](merge-on-write) enable merge-on-write by default (#27188) 2023-12-06 21:06:58 +08:00
e79422addc [refactor](Nereids) compatible with all ability legacy planner (#27947)
refactor:
1. split InsertIntoTableCommand into three sub command
- InsertIntoTableCommand
- InsertOverwriteTableCommand
- BatchInsertIntoTableCommand

feature:
1. support DEFAULT keywords in values list
2. support empty values list
3. support temporary partition
4. support insert into values in txn model

fix:
1. should start transaction before release read lock on target table
2023-12-05 19:10:55 +08:00
2b2c2dd772 [fix](sequence column) insert into should require sequence column in all scenario (#27780) 2023-11-30 23:27:58 +08:00
fef627c0ba [Fix](Txn) Fix transaction write to sequence column error (#26748) 2023-11-14 10:30:10 +08:00
9b494f4b36 [Fix](autoinc) skip to fill the auto increment column when the input column is not nullable (#23905) 2023-09-07 11:13:22 +08:00
c7b9eb5f9c [enhancement](bitmap)support bitmap type for non-key column in unique table (#23228) 2023-08-23 14:21:22 +08:00
a8d63ef93b [fix](case) Update test_dup_tab_auto_inc_10000.groovy, add sync after streamload #23082 2023-08-18 14:20:31 +08:00
0aa00026bb [fix](autoinc) ignore column property isAutoInc() for create table as select ... statement(#22827) 2023-08-10 23:25:54 +08:00
7890e464ee [fix](case) 1. disable unstable case window_function 2. add sync after stream load (#22677)
* Update test_dup_tab_auto_inc_with_null.groovy,add sync after streamload

* Update test_unique_table_auto_inc.groovy, add sync after streamload* [fix](case) disable unstable case window_function
2023-08-09 11:03:31 +08:00
ef218d79da [fix](case) add sync after stream load (#22232)
add sync after stream load
2023-07-28 17:05:20 +08:00
2b4bfe5be7 [fix](autoinc) fix _fill_auto_inc_cols when the input column is ColumnConst (#22175) 2023-07-25 14:41:36 +08:00
74313c7d54 [feature-wip](autoinc)(step-3) add auto increment support for unique table (#22036) 2023-07-21 13:24:41 +08:00
367ad9164a [feature-wip](auto-inc)(step-2) support auto-increment column for duplicate table (#19917) 2023-07-20 18:03:39 +08:00
0469c02202 [Test](regression) Temporarily disable quickTest for SHOW CREATE TABLE to adapt to enable_feature_binlog=true (#21247) 2023-07-05 10:12:02 +08:00
cc53391c9a Revert "[feature](merge-on-write) enable merge on write by default (#… (#21041) 2023-06-21 18:36:46 +08:00
187bf14d81 [feature-wip](auto-inc)(step-1) add syntax support for duplicate table (#20284)
Co-authored-by: yifeng <cnissnzg@126.com>
2023-06-07 22:01:28 +08:00
1b94b6368f [fix](load) in strict mode, return error for insert if datatype convert fails (#20378)
* [fix](load) in strict mode, return error for load and insert if datatype convert fails

Revert "[fix](MySQL) the way Doris handles boolean type is consistent with MySQL (#19416)"

This reverts commit 68eb420cabe5b26b09d6d4a2724ae12699bdee87.

Since it changed other behaviours, e.g. in strict mode insert into t_int values ("a"),
it will result 0 is inserted into table, but it should return error instead.

* fix be ut

* fix regression tests
2023-06-06 12:04:03 +08:00
05d497d21e [fix](sequence) value predicates shouldn't be push down when has sequence column (#20408)
* (fix)[sequence] value predicates shouldn't be push down when has sequence column

* add case
2023-06-05 19:18:34 +08:00
ecdc5124be [feature-wip](duplicate-no-keys) schame change support for duplicate no keys (#19326) 2023-06-02 09:22:41 +08:00
55ccddb62c [Conf](decimalv3) enable decimalv3 by default 2023-05-29 15:38:31 +08:00
ee34b6de2d [Refact] (serde) refact mysql serde with data type (#19543)
refact mysql output (de)serialize with data type serde , avoid accoriding switch case Primitive type writed in mysqlWriter
2023-05-26 14:11:17 +08:00
5167dc1251 [feature](merge-on-write) enable merge on write by default (#19017) 2023-05-11 11:10:48 +08:00
39b5682d59 [Pipeline](shared_scan_opt) Support shared scan opt in pipeline exec engine 2023-03-13 10:33:57 +08:00
fe6361f4b5 [regression-test](p0) fix some unstable p0 cases (#17518)
drop database before create
remove some large, unused debug log
2023-03-10 10:21:39 +08:00
849b5b7b8f [fix](sequence) fix that the result is wrong when load multiple duplicate keys (#17575) 2023-03-09 20:59:23 +08:00
edead494cb [Enhancement](storage) add a new hidden column __DORIS_VERSION_COL__ for unique key table (#16509) 2023-02-23 15:47:17 +08:00
6a5277b391 [fix](sequence-column) MergeIterator does not use the correct seq column for comparison (#16494) 2023-02-10 17:51:15 +08:00
69f34cd1c3 [fix](load) sequence column do not compare correctly in memtable (#16211) 2023-02-02 11:00:23 +08:00
5ec4e5586f [refactor]remove seek block in segmentIterator (#15413)
* remove seek block

* add reg test

Co-authored-by: Wang Bo <wangbo36@meituan.com>
2022-12-30 14:14:16 +08:00
4dbe30d37b [regression](vectorized) delete vectorized config in regression tests (#15126) 2022-12-16 17:08:29 +08:00
fb140d0180 [Enhancement](sequence-column) optimize the use of sequence column (#13872)
When you create the Uniq table, you can specify the mapping of sequence column to other columns.
You no longer need to specify mapping column when importing.
2022-11-17 22:39:09 +08:00