[feature](cold-data) move cold data to object storage without losing any feature(BE) (#10280)
This PR supports rowset level data upload on the BE side, so that there can be both cold data and hot data in a tablet, and there is no necessary to prohibit loading new data to cooled tablets. Each rowset is bound to a `FileSystem`, so that the storage layer can read and write rowsets without perceiving the underlying filesystem. The abstracted `RemoteFileSystem` can try local caching strategies with different granularity, instead of caching segment files as before. To avoid conflicts with the code in be/src/io, we temporarily put the file system related code in the be/src/io/fs directory. In the future, `FileReader`s and `FileWriter`s should be unified.
This commit is contained in:
@ -24,6 +24,7 @@
|
||||
#include "olap/page_cache.h"
|
||||
#include "olap/segment_loader.h"
|
||||
#include "olap/storage_engine.h"
|
||||
#include "olap/storage_policy_mgr.h"
|
||||
#include "runtime/broker_mgr.h"
|
||||
#include "runtime/bufferpool/buffer_pool.h"
|
||||
#include "runtime/bufferpool/reservation_tracker.h"
|
||||
@ -139,6 +140,7 @@ Status ExecEnv::_init(const std::vector<StorePath>& store_paths) {
|
||||
_stream_load_executor = new StreamLoadExecutor(this);
|
||||
_routine_load_task_executor = new RoutineLoadTaskExecutor(this);
|
||||
_small_file_mgr = new SmallFileMgr(this, config::small_file_dir);
|
||||
_storage_policy_mgr = new StoragePolicyMgr();
|
||||
|
||||
_backend_client_cache->init_metrics("backend");
|
||||
_frontend_client_cache->init_metrics("frontend");
|
||||
|
||||
Reference in New Issue
Block a user