Commit Graph

233 Commits

Author SHA1 Message Date
4031808f00 [enhancement](rowset-meta) Remove rowset meta from olap meta directly when rowsets deleted (#41716) (#43183)
pick: #41716
2024-11-05 00:40:12 +08:00
7d64c8cbc6 [branch-2.1] Picks "[opt](autoinc) Remove some restrictions on schema change on table that has auto-increment column #40280" (#41096)
picks https://github.com/apache/doris/pull/40280
2024-09-23 09:30:15 +08:00
e0b22b5104 [enhancement](schema-change) Log out end version before truncating new tablet data (#39924) (#40239)
## Proposed changes

As title.
2024-09-09 10:46:41 +08:00
0b9817ea4b [cherry-pick](branch-2.1) Pick "[Enhancement](txn) Block new insert into if schema change happens during transaction (#39483)" (#40115)
## Proposed changes

Pick #39483 

<!--Describe your changes.-->
2024-08-30 10:01:10 +08:00
cb312cabb2 [Fix](tablet-meta) limit the data size of tablet meta (#39455) (#39974)
pick master #39455
2024-08-27 20:10:17 +08:00
f8f5be7ce7 [fix](schema-change) Fix wrong intput column for cast validity check (#38894) (#39107)
## Proposed changes

1. Use column idx of ref block instead of new block to indicate the ref
column.
2. Rename some variables to clarify their meanings.
3. Clarify some log msg.
4. Add a minimal case to verify the change.
2024-08-08 19:36:06 +08:00
217eac790b [pick](Variant) pick some refactor and fix #34925 #36317 #36201 #36793 (#37526) 2024-07-11 21:25:34 +08:00
23cf494b48 [fix](schema-change) Fix schema-change from non-null to null (#36389)
https://github.com/apache/doris/pull/32913
2024-06-26 20:20:50 +08:00
5541fd11e9 [branch-2.1](partial update)add logs for partial update (#35416)
add logs for partial update

the master PR is #35802

If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
2024-06-04 22:47:48 +08:00
746c6207fc [fix](index) bitmap and bloomfilter index should not do light index change (#35225) 2024-05-29 10:09:31 +08:00
11f7d11a96 [branch-2.1](regression-test) fix test_chema_change_fail (#33753) #33788 2024-04-17 23:42:13 +08:00
f8acb1ee2e [fix](merge-on-write) schema change may cause mow duplicate key (#33536) (#33660) 2024-04-17 23:42:12 +08:00
55be6c6309 [fix](schema change) follow fe set sc fail replicas as bad (#33569) 2024-04-17 23:42:12 +08:00
fdc19b4892 [Fix](Variant) Initialize original_tablet_schema in _expand_variant_to_subcolumns to address potential nullptr issue (#32184) (#32678) 2024-03-22 18:02:58 +08:00
27973b6999 [fix](schema-change) fix the bug of handling empty blocks in schema change (#32396)
* [fix](schema-change) fix the bug of handling empty blocks in schema change

* add case
2024-03-19 22:12:26 +08:00
Pxl
5687ca977d [Bug](java-udf) fix core dump when javaudf input 0 row block (#30720)
fix core dump when javaudf input 0 row block
2024-02-03 20:25:25 +08:00
a94f2564e3 [fix] (schema change) ignore the error of repeatedly writing rowset during schema change (#29733) 2024-01-12 11:48:39 +08:00
797238cbb7 [fix](merge-on-write) fix schema change may result in delete bitmap incorrect (#29386) 2024-01-02 23:45:04 +08:00
673a0137ce [fix] (schema change) fix missing data bug in schema change (#28938) (#29064) 2023-12-30 20:45:58 +08:00
Pxl
118775f913 [Bug](schame-change) fix wrong result after reorder mor table (#29045)
* fix wrong result after reorder mor table

* update
2023-12-28 14:57:31 +08:00
f374beaa4e [fix](log) regularise some BE error type and fix a load task check #28729 2023-12-25 10:45:19 +08:00
73f7b61019 [refactor](scanner) use weak ptr to lock task execution context to avoid core in scanner dctor (#28493)
using weak ptr as a lock between fragment execute thread and scanner thread, to solve the core problem in scanner's dctor to access scannode's profile.
2023-12-18 14:09:32 +08:00
Pxl
027b06059a [Feature](materialized-view) support count(1) on materialized view (#28135)
support count(1) on materialized view
fix match failed like select k1, sum(k1) from t group by k1
2023-12-09 01:36:46 +08:00
e75d91c91b [regression-test](Variant) Add more cases related to schema changes (#27958)
* [regression-test](Variant) Add more cases related to schema changes

And fix bugs about schema change for variant:
fix bug schema change crash on doing schema change with tablet schema that contains extracted columns
2023-12-08 10:15:12 +08:00
Pxl
96f2ef3d99 [Improvement](schema-change) Reserve some memory for use by other parts except hold block of schem… (#27800)
Reserve some memory for use by other parts except hold block of schema change job
2023-11-30 17:01:51 +08:00
Pxl
f573918aa4 [Chore](materialized-view) output reference column info when create mv can't find ref column (#27182)
output reference column info when create mv can't find ref column
2023-11-30 16:48:06 +08:00
7398c3daf1 [Feature-Variant](Variant Type) support variant type query and index (#27676) 2023-11-29 10:37:28 +08:00
d767804815 [feature](merge-cloud) Decouple rowset id generator and local rowsets gc implementation (#25921) 2023-11-10 10:07:02 +08:00
a5ef90dacc [enhancement](recover) support skipping missing version in select by session variable (#25654) 2023-11-02 20:01:51 +08:00
f31c1d858a [fix](merge-on-write) fix duplicate key in schema change (#25705)
It should be ensured that the obtained versions are continuous when calculate delete bitmap calculations in publish.
The remaining NOTREADY tablet in the schema change failure should be dropped.
When a rowset was deleted, the delete bitmap cannot be deleted until there are no read requests to use the rowset.
2023-10-25 05:59:48 -05:00
Pxl
2e2d5bcba2 [Improvements](status) catch some error status (#25677)
catch some error status
2023-10-23 10:19:08 +08:00
9c9fc84f39 [feature](merge-cloud) Abstract BaseTablet for CloudTablet (#24929) 2023-10-18 20:29:04 +08:00
cda8fb6b8b [fix](load) return Status when error in RowsetWriter::build (#25381) 2023-10-17 09:40:23 +08:00
642e5cdb69 [Fix](Status) Make Status [[nodiscard]] and handle returned Status correctly (#23395) 2023-09-29 22:38:52 +08:00
c04078f3b8 [improvement](compaction) output tablet_id when be core dumped. (#24952) 2023-09-27 16:50:18 +08:00
8191cd1dad [Bug](ScanNode) Fix potential incorrect query result caused by concurrent NewOlapScanNode initialization and Compaction (#24638)
* Optimize fetch delete predicates

* Fix incorrect query result when compaction eliminate delete predicates between `NewOlapScanNode::_init_scanners` and `NewOlapScanner::init`

* Fix be ut
2023-09-25 22:24:35 +08:00
Pxl
3049533e63 [Bug](materialized-view) fix core dump on create materialized view when diffrent mv column have same reference base column (#23425)
* Remove redundant predicates on scan node

update

fix core dump on create materialized view when diffrent mv column have same reference base column

Revert "update"

This reverts commit d9ef8dca123b281dc8f1c936ae5130267dff2964.

Revert "Remove redundant predicates on scan node"

This reverts commit f24931758163f59bfc47ee10509634ca97358676.

* update

* fix

* update

* update
2023-08-28 14:40:51 +08:00
c251a574e8 [Fix](MoW) Fix dup key when do schema change add new key (#22154) 2023-07-25 14:18:01 +08:00
103c473b96 [Bug](pipeline) fix pipeline shared scan + topn optimization (#21940) 2023-07-25 12:48:27 +08:00
db69af1165 [fix](meger-on-write) fix query result wrong when schema change (#22044) 2023-07-21 15:29:04 +08:00
Pxl
ca71048f7f [Chore](status) avoid empty error msg on status (#21454)
avoid empty error msg on status
2023-07-11 13:48:16 +08:00
Pxl
1fc1e76fc7 [Bug](alter table) return error status to avoid core dump on schema change meet invalid input (#21273)
return error status to avoid core dump on schema change meet invalid input
2023-06-28 14:20:16 +08:00
31a4f96f01 [refactor](exprcontext) move close to expr context's dector method (#20747)
The close method does nothing. But I am not sure we could remove it. So that I add it to dector method and remove many many calls.
2023-06-14 18:01:07 +08:00
24fb05ec83 [Bug](row-store) Fix row store with materialize index (#20356)
If a query hits a materialized view that has row storage enabled, but the row storage column is not present in the materialized view, it will result in a query crash. Therefore, it is necessary to include the row storage column when creating the materialized view, and serialize the row storage column during the execution of SchemaChange.
2023-06-08 10:55:22 +08:00
9f8de89659 [refactor](exec) replace the single pointer with an array of 'conjuncts' in ExecNode (#19758)
Refactoring the filtering conditions in the current ExecNode from an expression tree to an array can simplify the process of adding runtime filters. It eliminates the need for complex merge operations and removes the requirement for the frontend to combine expressions into a single entity.

By representing the filtering conditions as an array, each condition can be treated individually, making it easier to add runtime filters without the need for complex merging logic. The array can store the individual conditions, and the runtime filter logic can iterate through the array to apply the filters as needed.

This refactoring simplifies the codebase, improves readability, and reduces the complexity associated with handling filtering conditions and adding runtime filters. It separates the conditions into discrete entities, enabling more straightforward manipulation and management within the execution node.
2023-05-29 11:47:31 +08:00
42239d635a [fix](tablet_manager_lock) fix create tablet timeout #20067 (#20069) 2023-05-28 23:05:13 +08:00
ae352997b4 [Enhancement](alter inverted index) Improve alter inverted index performance with light weight add or drop inverted index (#19063) 2023-05-28 11:23:07 +08:00
Pxl
15a7420661 [Chore](ub) fix some undefined behaviors (#19986)
/home/zcp/repo_center/doris_master/doris/be/src/olap/rowset/segment_v2/column_reader.cpp:895:21: runtime error: load of value 423208544, which is not a valid value for type 'doris::ReaderType'

/home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_decimal.cpp:260:33: runtime error: load of misaligned address 0x7fa3348b301c for type 'int64_t' (aka 'long'), which requires 8 byte alignment

/home/zcp/repo_center/doris_master/doris/be/src/olap/block_column_predicate.cpp:82:24: runtime error: variable length array bound evaluates to non-positive value 0

/home/zcp/repo_center/doris_master/doris/be/src/vec/columns/column_string.h:225:26: runtime error: null pointer passed as argument 2, which is declared to never be null
2023-05-26 14:08:40 +08:00
Pxl
d64be9565d [Bug](function) fix function in get wrong result when input const column (#19791)
fix function in get wrong result when input const column
2023-05-22 10:58:29 +08:00
Pxl
dfad7b6b38 [Feature](generic-aggregation) some prowork of generic aggregation (#19343)
some prowork of generic aggregation
2023-05-09 21:42:21 +08:00