database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[feature](cold-data) move cold data to object storage without losing any feature(BE) (#10280)

Browse Source 
This PR supports rowset level data upload on the BE side, so that there can be both cold data and hot data in a tablet,
and there is no necessary to prohibit loading new data to cooled tablets.

Each rowset is bound to a `FileSystem`, so that the storage layer can read and write rowsets without
perceiving the underlying filesystem.

The abstracted `RemoteFileSystem` can try local caching strategies with different granularity,
instead of caching segment files as before.

To avoid conflicts with the code in be/src/io, we temporarily put the file system related code in the be/src/io/fs directory.
In the future, `FileReader`s and `FileWriter`s should be unified.
This commit is contained in:
plat1ko
2022-07-08 12:18:39 +08:00
committed by GitHub
parent e159e748df
commit 331fa50501
144 changed files with 4152 additions and 1413 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
be/src/util/file_cache.cpp
View File

@ -47,9 +47,12 @@ bool FileCache<FileType>::lookup(const std::string& file_name,
template <class FileType>
void FileCache<FileType>::insert(const std::string& file_name, FileType* file,
OpenedFileHandle<FileType>* file_handle) {
OpenedFileHandle<FileType>* file_handle,
void (*deleter)(const CacheKey&, void*)) {
DCHECK(_cache != nullptr);
auto deleter = [](const CacheKey& key, void* value) { delete (FileType*)value; };
if (!deleter) {
deleter = [](const CacheKey& key, void* value) { delete (FileType*)value; };
}
CacheKey key(file_name);
auto lru_handle = _cache->insert(key, file, 1, deleter);
*file_handle = OpenedFileHandle<FileType>(_cache.get(), lru_handle);
@ -57,5 +60,6 @@ void FileCache<FileType>::insert(const std::string& file_name, FileType* file,
// Explicit specialization for callers outside this compilation unit.
template class FileCache<RandomAccessFile>;
template class FileCache<int>;
} // namespace doris