Commit Graph

395 Commits

Author SHA1 Message Date
beab6a81c1 [fix] (compaction) fix time series compaction policy (#39170) (#39228)
## Proposed changes

pick from master #39170
2024-08-13 10:33:04 +08:00
e2f45225d6 [branch-2.1] Picks "[opt](merge-on-write) eliminate reading the old values of non-key columns for delete stmt in publish phase #38703" (#39074)
picks https://github.com/apache/doris/pull/38703
2024-08-09 10:42:52 +08:00
7e95d7cbec [bugfix](backup)(cooldown) cancel backup properly when be backup failed (#38724) (#38993)
Co-authored-by: zhangyuan <ayuanzhang@tencent.com>
2024-08-07 15:58:11 +08:00
8ce30963cd [fix] (compaction) fix time series compaction policy (#38220) (#38917)
## Proposed changes

pick from #38220
2024-08-06 14:26:42 +08:00
64b69ed1ba [branch-2.1] Picks "[opt](merge-on-write) Skip the alignment process of some rowsets in partial update #38487" (#38682)
## Proposed changes

picks https://github.com/apache/doris/pull/38487
2024-08-02 20:05:31 +08:00
327069fdbc [branch-2.1](log) add tablet clear cache log (#38713) 2024-08-02 08:40:02 +08:00
4d980b8235 [feature](http action)Add http action to show nested inverted index file (#38272) (#38672)
backport #38272
2024-08-01 19:30:59 +08:00
359e50fc58 [fix](load) change tablet schema pointer to shared_ptr in memtable (#37927) (#37939)
backport #37927
2024-07-16 22:32:03 +08:00
7887f51e9b [fix](partial update) fix a mem leak issue (#37706) (#37730)
cherry-pick #37706
2024-07-13 09:20:01 +08:00
217eac790b [pick](Variant) pick some refactor and fix #34925 #36317 #36201 #36793 (#37526) 2024-07-11 21:25:34 +08:00
97945af947 [fix](merge-on-write) when full clone failed, duplicate key might occur (#37001) (#37229)
cherry-pick #37001
2024-07-03 19:48:10 +08:00
7443e8fcf2 [cherry-pick](branch-2.1) fix single compaction test p2 #34568 #36881 (#37075) 2024-07-02 15:22:04 +08:00
798d9d6fc6 [pick21][opt](mow) reduce memory usage for mow table compaction (#36865) (#36968)
cherry-pick https://github.com/apache/doris/pull/36865 to branch-2.1
2024-07-01 15:33:18 +08:00
22cb7b8fcb [improvement](compaction) be do not compact invisible version to avoid query error -230 #28082 (#36222)
cherry pick from #28082
2024-06-27 13:45:21 +08:00
6ec9a731e8 [branch-2.1](cherry-pick) partial update should not read old fileds from rows with delete sign (#36210) (#36755)
cherry-pick #36210
2024-06-24 21:13:24 +08:00
5541fd11e9 [branch-2.1](partial update)add logs for partial update (#35416)
add logs for partial update

the master PR is #35802

If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
2024-06-04 22:47:48 +08:00
72489a04c3 [cherry-pick](branch-2.1) remove some CHECKs in Tablet::revise_tablet_meta (#31268) (#34702)
## Proposed changes

Issue Number: close #xxx

cherry-pick #31268 

## Further comments

If this is a relatively large or complex change, kick off the discussion
at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why
you chose the solution you did and what alternatives you considered,
etc...
2024-06-02 00:15:31 +08:00
7b271f916d [branch-2.1](partial-update) duplicate key occurred when BE restart (#35678)
We should save new delete bitmap in rocksdb when conflicts are handled in publish phase, which is introduced introduced by #30366
2024-05-31 09:38:06 +08:00
6e17dc1e87 (cherry-pick)[branch-2.1] add calc tablet file crc and fix single compaction test #33076 #34915 (#35215)
* [fix](compaction test) show single replica compaction status and fix test (#33076)
* [improve](http action) add http interface to calculate the crc of all files in tablet (#34915)
2024-05-26 17:15:09 +08:00
1e4a83e17b fix compile 2024-05-22 01:04:34 +08:00
b4a798240a [fix](inverted_index) donot use int32_t for index id to avoid overflow (#35062) 2024-05-21 12:58:38 +08:00
c22f42121b [fix](compaction test) show single replica compaction status and fix test (#33076) (#34285) (#34438) 2024-05-06 21:00:34 +08:00
b15fc2a906 [Cherry-pick](branch-2.1) Pick #34043 and #34112 (#34318)
* [Enhancement](full compaction) Add run status support for full compaction (#34043)

* The usage is `curl http://{ip}:{host}/api/compaction/run_status?tablet_id={tablet_id}`
e.g. `curl http://127.0.0.1:8040/api/compaction/run_status?tablet_id=10084`

If full compaction is running, the output will be
```
{
"status" : "Success",
"run_status" : true,
"msg" : "compaction task for this tablet is running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```
else the ouput will be
```
{
"status" : "Success",
"run_status" : false,
"msg" : "compaction task for this tablet is not running",
"tablet_id" : 10084,
"compact_type" : "full"
}
```

* 2

* 2

* [Fix](partial update) Fix rowset not found error when doing partial update (#34112)

Cause: In the logic of partial column updates, the existing data columns are read first, and then the data is supplemented and written back. During the reading process, initialization involves initially fetching rowset IDs, and the actual rowset object is fetched only when needed later. However, between fetching the rowset IDs and the rowset object, compaction may occur, turning the old rowset into a stale rowset. If too much time passes, the stale rowset might be directly deleted. Thus, when the rowset object is needed for an update, it cannot be found. Although the update operation with partial column logic should be able to read all keys and should not encounter new keys, if the rowset disappears, the Backend (BE) will consider these keys as missing. Consequently, it will check whether other columns have default values or are nullable. If this check fails, the aforementioned error is thrown.

Solution: To avoid such issues during partial column updates, the initialization step should involve fetching both the rowset IDs and the shared pointer to the rowset object simultaneously. This ensures that the rowset can always be found during data retrieval.
2024-04-30 07:26:23 +08:00
5277a55791 (pick 34003) release fd for shutdown tablets (#34224) 2024-04-29 10:51:19 +08:00
7e91e69eb9 [fix](compaction) fix single compaction (#33907)
* [fix](compaction)Fix single compaction to get all local versions #33849

add test and comment

* remove single replica compaction prepare input rowsets

reviesd
2024-04-19 23:30:25 +08:00
46a258dc85 [improvement](binlog)Support inverted index format v2 in CCR (#33415) 2024-04-17 23:42:12 +08:00
3d66723214 [branch-2.1](auto-partition) pick auto partition and some more prs (#33523) 2024-04-11 17:12:17 +08:00
2add3bc13a [fix](partial update) compaction may cause update failue (#31551) (#32361) 2024-03-18 10:58:51 +08:00
5f125bbaaa [improvement](binlog)Support inverted index in CCR (#31743) (#32101) 2024-03-12 15:34:08 +08:00
ffa904c487 [enhance](Cooldown) Skip cooldown if the tablet is dropped (#32079) 2024-03-12 14:20:39 +08:00
779ca464a5 [Fix](Status) Handle returned overall Status correctly (#31692)
Handle returned overall Status correctly
2024-03-09 19:44:39 +08:00
65d45daf8a [Bug](coredump) fix regresstion test coredump in multi thread access map (#31664) 2024-03-03 19:30:55 +08:00
e86cc7e8e8 [chore](log) reduce a lot inject debug point log #31474 2024-02-28 13:07:47 +08:00
82fd3af54b [chore](log) change merge-on-write correctness check log to VLOG_NOTICE (#31414) (#31467) 2024-02-27 23:36:24 +08:00
5e4674ab66 [fix](partial update) mishandling of exceptions in the publish phase may result in data loss (#30366) 2024-01-27 09:09:02 +08:00
25c71e386b [minor](Cooldown) Log the old rowset id when uploading rowset to S3 #30339 2024-01-27 09:07:02 +08:00
ccde65b942 [fix](Cooldown) Enhance calculate logic of _has_data_to_cooldown (#30244) (#30299) 2024-01-25 13:25:34 +08:00
5213f941dd [improvement](cooldown) print the cooldown version when follow cooldown version (#30239) 2024-01-24 10:02:03 +08:00
d525f576e1 [improve] Use lru cache to count the number of column in tablet schema to control memory (#29668) 2024-01-12 13:58:19 +08:00
0d16ec7345 [improvement](cooldown) do not cooldown tablet without cold data (#29690) 2024-01-12 11:57:16 +08:00
7c7dbf15bc [feature](merge-cloud) Decouple Tablet/TabletManager/TxnManager from global StorageEngine instance (#29736) 2024-01-12 11:57:16 +08:00
eb2b22bff1 [improve](cooldown) skip empty tablet (#29620) 2024-01-07 18:57:06 +08:00
xy
2a6150b057 [optimize](cooldown)If cooldowned_rowsets is empty, there is no need to follow the logic of traversing the files (#27324)
Co-authored-by: xingying01 <xingying01@corp.netease.com>
2024-01-05 18:59:09 +08:00
xy
d86277c2c2 [optimize](cooldown)Improved filtering conditions for cooldown tablet (#27338)
Co-authored-by: xingying01 <xingying01@corp.netease.com>
2024-01-05 15:03:54 +08:00
797238cbb7 [fix](merge-on-write) fix schema change may result in delete bitmap incorrect (#29386) 2024-01-02 23:45:04 +08:00
59fdd5e42b [fix](compaction) time series compaction policy -> compact consecutive empty rowsets (#27299)
Sometimes we need to merge a large number of empty versions to reduce meta-information.
2023-12-29 22:20:41 +08:00
xy
fd90c3a6a6 [optimize](cooldown)Reduce the number of calls to the pick_cooldown_rowset (#27091)
Co-authored-by: xingying01 <xingying01@corp.netease.com>
2023-12-28 13:03:33 +08:00
1aa9ac4fe4 Prevent making snapshot on remote rowset in single replica compaction (#28716) 2023-12-27 23:43:43 +08:00
7081139bdc [fix](block) fix be core while mutable block merge may cause different row size between columns in origin block (#27943) 2023-12-25 20:35:22 +08:00
de6c7a792e [fix](chore) update dcheck to avoid core during stress test (#28895) 2023-12-23 18:49:57 +08:00