database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[refactor](file-system)(step-2) remove env, file_utils and filesystem_utils (#18009)

Browse Source 
Follow #17586.
This PR mainly changes:

Remove env/
Remove FileUtils/FilesystemUtils
Some methods are moved to LocalFileSystem
Remove olap/file_cache
Add s3 client cache for s3 file system
In my test, the time of open s3 file can be reduced significantly
Fix cold/hot separation bug for s3 fs.
This is the last PR of #17764.
After this, all IO operation should be in io/fs.

Except for tests in #17586, I also tested some case related to fs io:

clone
concurrency query on local/s3/hdfs
load error log create and clean
disk metrics
This commit is contained in:
Mingyu Chen
2023-03-29 09:00:52 +08:00
committed by GitHub
parent c3fe113894
commit 05db6e9b55
165 changed files with 1905 additions and 4835 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
be/src/runtime/user_function_cache.cpp
View File

@ -23,11 +23,10 @@
#include "common/config.h"
#include "common/status.h"
#include "env/env.h"
#include "gutil/strings/split.h"
#include "http/http_client.h"
#include "io/fs/local_file_system.h"
#include "util/dynamic_util.h"
#include "util/file_utils.h"
#include "util/jni-util.h"
#include "util/md5.h"
#include "util/spinlock.h"
@ -165,23 +164,24 @@ Status UserFunctionCache::_load_entry_from_lib(const std::string& dir, const std
Status UserFunctionCache::_load_cached_lib() {
// create library directory if not exist
RETURN_IF_ERROR(FileUtils::create_dir(_lib_dir));
RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(_lib_dir));
for (int i = 0; i < kLibShardNum; ++i) {
std::string sub_dir = _lib_dir + "/" + std::to_string(i);
RETURN_IF_ERROR(FileUtils::create_dir(sub_dir));
RETURN_IF_ERROR(io::global_local_filesystem()->create_directory(sub_dir));
auto scan_cb = [this, &sub_dir](const char* file) {
if (is_dot_or_dotdot(file)) {
auto scan_cb = [this, &sub_dir](const io::FileInfo& file) {
if (!file.is_file) {
return true;
}
auto st = _load_entry_from_lib(sub_dir, file);
auto st = _load_entry_from_lib(sub_dir, file.file_name);
if (!st.ok()) {
LOG(WARNING) << "load a library failed, dir=" << sub_dir << ", file=" << file;
LOG(WARNING) << "load a library failed, dir=" << sub_dir
<< ", file=" << file.file_name;
}
return true;
};
RETURN_IF_ERROR(Env::Default()->iterate_dir(sub_dir, scan_cb));
RETURN_IF_ERROR(io::global_local_filesystem()->iterate_directory(sub_dir, scan_cb));
}
return Status::OK();
}