database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
16,993 Commits 1 Branch 0 Tags
2f9bd3e3bb0e4b2dbddd9faf7818d86fb24f97f2
Commit Graph

6 Commits

Author SHA1 Message Date
AlexYue
2f9bd3e3bb (enhance)(S3) Change s3 metric from bvar adder to latency recorder (#28861) 2024-02-19 17:22:03 +08:00
plat1ko
b0cac0014d [enhance](FS) Improve FS error code (#29432) 2024-01-06 21:17:22 +08:00
Chuanle Chen
71807ceb5f [Enhancement](tvf) Table value function support reading local file (#17404) 
I tested the local tvf with tpch queries. First, generate `lineitem` datasets with 6001215 rows, and load it into `lineitem` table by:
```
insert into lineitem select c11, c1, c4, c2, c3, c5, c6, c7, c8, c9, c10, c12, c13, c14, c15, c16 
from local(
        "file_path" = "tools/tpch-tools/bin/tpch-data/lineitem.tbl.1", 
        "backend_id" = "10003", 
        "format" = "csv", 
        "column_separator" = "|"
);
```
Then, run `q1` and `q16` tpch queries, the query result is correct.

It can also analyze the BE's log directly like:

```
mysql> select * from local(
        "file_path" = "log/be.out",
        "backend_id" = "10006",
        "format" = "csv")
       where c1 like "%start_time%" limit 10;
+--------------------------------------------------------+
| c1                                                     |
+--------------------------------------------------------+
| start time: 2023年 08月 07日 星期一 23:20:32 CST       |
| start time: 2023年 08月 07日 星期一 23:32:10 CST       |
| start time: 2023年 08月 08日 星期二 00:20:50 CST       |
| start time: 2023年 08月 08日 星期二 00:29:15 CST       |
+--------------------------------------------------------+
```
 2023-08-10 20:07:42 +08:00
Adonis Ling
e412dd12e8 [chore](build) Use include-what-you-use to optimize includes (PART II) (#18761) 
Currently, there are some useless includes in the codebase. We can use a tool named include-what-you-use to optimize these includes. By using a strict include-what-you-use policy, we can get lots of benefits from it.
 2023-04-19 23:11:48 +08:00
Mingyu Chen
7e61a85331 [refactor](libhdfs) introduce hadoop libhdfs (#18204) 
1. Introduce hadoop libhdfs 
2. For Linux-X86 platform, use the hadoop libhdfs
3. For other platform, use libhdfs3, because currently we don't have  hadoop libhdfs binary for other platform

Co-authored-by: adonis0147 <adonis0147@gmail.com>
 2023-03-31 18:41:39 +08:00
Mingyu Chen
05db6e9b55 [refactor](file-system)(step-2) remove env, file_utils and filesystem_utils (#18009) 
Follow #17586.
This PR mainly changes:

Remove env/
Remove FileUtils/FilesystemUtils
Some methods are moved to LocalFileSystem
Remove olap/file_cache
Add s3 client cache for s3 file system
In my test, the time of open s3 file can be reduced significantly
Fix cold/hot separation bug for s3 fs.
This is the last PR of #17764.
After this, all IO operation should be in io/fs.

Except for tests in #17586, I also tested some case related to fs io:

clone
concurrency query on local/s3/hdfs
load error log create and clean
disk metrics
 2023-03-29 09:00:52 +08:00