database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[opt](FileReader) InMemoryReader is only used in s3 (#23486)

Browse Source 
If file size < 8MB, the file will be read into memory, and this idea is from https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/site/markdown/tools/hadoop-aws/prefetching.md#s3inmemoryinputstream. However, in some cases, we only read one or two columns in a file, and the actually required bytes is only 1%, resulting in a multiple fold increase in the amount of data read. Therefore, `InMemoryReader` can only used in object storage, and reduce the threshold.
This commit is contained in:
Ashin Gau
2023-08-30 20:43:39 +08:00
committed by GitHub
parent b7404896fa
commit 449c595f9d
4 changed files with 13 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
be/src/io/fs/buffered_reader.cpp
View File

@ -778,8 +778,12 @@ Status DelegateReader::create_file_reader(RuntimeProfile* profile,
io::FileReaderSPtr reader;
RETURN_IF_ERROR(FileFactory::create_file_reader(system_properties, file_description,
reader_options, file_system, &reader, profile));
if (reader->size() < IN_MEMORY_FILE_SIZE) {
*file_reader = std::make_shared<InMemoryFileReader>(reader);
if (reader->size() < config::in_memory_file_size) {
if (typeid_cast<io::S3FileReader*>(reader.get())) {
*file_reader = std::make_shared<InMemoryFileReader>(reader);
} else {
*file_reader = std::move(reader);
}
} else if (access_mode == AccessMode::SEQUENTIAL) {
bool is_thread_safe = false;
if (typeid_cast<io::S3FileReader*>(reader.get())) {