database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
13,073 Commits 1 Branch 0 Tags
0318da1df3661663cd0affdf6c073e686eb99dfe
Commit Graph

5 Commits

Author SHA1 Message Date
Chuanle Chen
71807ceb5f [Enhancement](tvf) Table value function support reading local file (#17404) 
I tested the local tvf with tpch queries. First, generate `lineitem` datasets with 6001215 rows, and load it into `lineitem` table by:
```
insert into lineitem select c11, c1, c4, c2, c3, c5, c6, c7, c8, c9, c10, c12, c13, c14, c15, c16 
from local(
        "file_path" = "tools/tpch-tools/bin/tpch-data/lineitem.tbl.1", 
        "backend_id" = "10003", 
        "format" = "csv", 
        "column_separator" = "|"
);
```
Then, run `q1` and `q16` tpch queries, the query result is correct.

It can also analyze the BE's log directly like:

```
mysql> select * from local(
        "file_path" = "log/be.out",
        "backend_id" = "10006",
        "format" = "csv")
       where c1 like "%start_time%" limit 10;
+--------------------------------------------------------+
| c1                                                     |
+--------------------------------------------------------+
| start time: 2023年 08月 07日 星期一 23:20:32 CST       |
| start time: 2023年 08月 07日 星期一 23:32:10 CST       |
| start time: 2023年 08月 08日 星期二 00:20:50 CST       |
| start time: 2023年 08月 08日 星期二 00:29:15 CST       |
+--------------------------------------------------------+
```
 2023-08-10 20:07:42 +08:00
Adonis Ling
16a394da0e [chore](build) Use include-what-you-use to optimize includes (PART III) (#18958) 
Currently, there are some useless includes in the codebase. We can use a tool named include-what-you-use to optimize these includes. By using a strict include-what-you-use policy, we can get lots of benefits from it.
 2023-04-24 14:51:51 +08:00
Mingyu Chen
ea47a6ae59 [fix](hdfs) not setting hadoop username when kerberos enabled (#18485) 
1. If we set hadoop user property along with kerberos info, the authentication will fail.
2. fix some minor issue of local fs, follow up #18397
3. Add KW_HOSTNAME to keywords region, follow up #17329
4. Fix tvf not working with pipeline engine, follow up #18376
 2023-04-10 09:32:27 +08:00
Mingyu Chen
1050df7076 [fix](fs) fix local file system copy bug (#18243) 
`copy_dirs` has a bug that will cause infinity iteration
 2023-03-30 21:36:07 +08:00
Mingyu Chen
05db6e9b55 [refactor](file-system)(step-2) remove env, file_utils and filesystem_utils (#18009) 
Follow #17586.
This PR mainly changes:

Remove env/
Remove FileUtils/FilesystemUtils
Some methods are moved to LocalFileSystem
Remove olap/file_cache
Add s3 client cache for s3 file system
In my test, the time of open s3 file can be reduced significantly
Fix cold/hot separation bug for s3 fs.
This is the last PR of #17764.
After this, all IO operation should be in io/fs.

Except for tests in #17586, I also tested some case related to fs io:

clone
concurrency query on local/s3/hdfs
load error log create and clean
disk metrics
 2023-03-29 09:00:52 +08:00