[refactor](file-system)(step-1) refactor file sysmte on BE and remove storage_backend (#17586)

See #17764 for details
I have tested:
- Unit test for local/s3/hdfs/broker file system: be/test/io/fs/file_system_test.cpp
- Outfile to local/s3/hdfs/broker.
- Load from local/s3/hdfs/broker.
- Query file on local/s3/hdfs/broker file system, with table value function and catalog.
- Backup/Restore with local/s3/hdfs/broker file system

Not test:
- cold & host data separation case.
This commit is contained in:
Mingyu Chen
2023-03-21 21:08:38 +08:00
committed by GitHub
parent 82716ec99d
commit cb79e42e5c
171 changed files with 3523 additions and 5288 deletions

View File

@ -120,11 +120,14 @@ void FileCacheManager::_add_file_cache_for_gc_by_disk(std::vector<GCContextPerDi
void FileCacheManager::_gc_unused_file_caches(std::list<FileCachePtr>& result) {
std::vector<TabletSharedPtr> tablets =
StorageEngine::instance()->tablet_manager()->get_all_tablet();
bool exists = true;
for (const auto& tablet : tablets) {
std::vector<Path> seg_file_paths;
if (io::global_local_filesystem()->list(tablet->tablet_path(), &seg_file_paths).ok()) {
for (Path seg_file : seg_file_paths) {
std::string seg_filename = seg_file.native();
std::vector<FileInfo> seg_files;
if (io::global_local_filesystem()
->list(tablet->tablet_path(), true, &seg_files, &exists)
.ok()) {
for (auto& seg_file : seg_files) {
std::string seg_filename = seg_file.file_name;
// check if it is a dir name
if (!BetaRowset::is_segment_cache_dir(seg_filename)) {
continue;