database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[feature](cold-data) move cold data to object storage without losing any feature(BE) (#10280)

Browse Source 
This PR supports rowset level data upload on the BE side, so that there can be both cold data and hot data in a tablet,
and there is no necessary to prohibit loading new data to cooled tablets.

Each rowset is bound to a `FileSystem`, so that the storage layer can read and write rowsets without
perceiving the underlying filesystem.

The abstracted `RemoteFileSystem` can try local caching strategies with different granularity,
instead of caching segment files as before.

To avoid conflicts with the code in be/src/io, we temporarily put the file system related code in the be/src/io/fs directory.
In the future, `FileReader`s and `FileWriter`s should be unified.
This commit is contained in:
plat1ko
2022-07-08 12:18:39 +08:00
committed by GitHub
parent e159e748df
commit 331fa50501
144 changed files with 4152 additions and 1413 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
be/src/agent/utils.cpp
View File

@ -131,6 +131,46 @@ Status MasterServerClient::report(const TReportRequest& request, TMasterResult*
return Status::OK();
}
Status MasterServerClient::refresh_storage_policy(TGetStoragePolicyResult* result) {
Status client_status;
FrontendServiceConnection client(_client_cache, _master_info.network_address,
config::thrift_rpc_timeout_ms, &client_status);
if (!client_status.ok()) {
LOG(WARNING) << "fail to get master client from cache. "
<< "host=" << _master_info.network_address.hostname
<< ", port=" << _master_info.network_address.port
<< ", code=" << client_status.code();
return Status::InternalError("Fail to get master client from cache");
}
try {
try {
client->refreshStoragePolicy(*result);
} catch (TTransportException& e) {
LOG(WARNING) << "master client, retry refresh_storage_policy: " << e.what();
client_status = client.reopen(config::thrift_rpc_timeout_ms);
if (!client_status.ok()) {
LOG(WARNING) << "fail to get master client from cache. "
<< "host=" << _master_info.network_address.hostname
<< ", port=" << _master_info.network_address.port
<< ", code=" << client_status.code();
return Status::InternalError("Master client refresh storage policy failed");
}
client->refreshStoragePolicy(*result);
}
} catch (TException& e) {
client.reopen(config::thrift_rpc_timeout_ms);
LOG(WARNING) << "fail to report to master. "
<< "host=" << _master_info.network_address.hostname
<< ", port=" << _master_info.network_address.port
<< ", code=" << client_status.code();
return Status::InternalError("Fail to refresh storage policy from master");
}
return Status::OK();
}
bool AgentUtils::exec_cmd(const string& command, string* errmsg, bool redirect_stderr) {
// The exit status of the command.
uint32_t rc = 0;