database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity
4,732 Commits 1 Branch 0 Tags
0683181fefdaf68e48e080c2cf0ae17623be729b
Commit Graph

8 Commits

Author SHA1 Message Date
caiconghui
d51545a952 [fix](ut)(memory-leak) Fix be asan ut failed and hdfs file reader memory leak (#8905) 2022-04-08 00:07:00 +08:00
zhannngchen
71d050d0bc [improvement][test] (log)Add more error message on connect to hdfs failure, and corresponding ut (#8755) 
I met a failure of reading hdfs files in broker load, the error message is unclear and
I spent a lot of time to locate the problem.

```
W0330 11:08:01.093812 2755268 broker_scan_node.cpp:364] Scanner[0] process failed. status=connect failed.
W0330 11:08:01.097682 2018787 fragment_mgr.cpp:234] Got error while opening fragment 712ae2b848324cb6-94a83d646173c1e9: Internal error: connect failed.
W0330 11:08:01.097702 2018787 tablet_sink.cpp:148] connect failed.
```

We should add more information when connect to hdfs failed.
 2022-03-31 13:56:25 +08:00
Zhengguo Yang
4bc5ba8819 mark the load job fail when more than a half of replica write failed of a tablet, (#7126) 
the code before is counting all replica has more than a half write failed.
 2021-11-17 10:18:04 +08:00
pengxiangyu
632f8fcc75 [libhdfs] Add errno for hdfs writer. when no dir, hdfs writer open failed, the dir need to be created. (#7050) 
1. Add errno message for hdfs writer failed.
2. When call openWrite for hdfs, the dir will be created when it doesn't exist,
 2021-11-11 15:21:21 +08:00
Mingyu Chen
59017cebe6 [ARM64] Fix some problem when compiling on ARM64 platform (#6836) 
1. Refactor the create method of hdfs reader & writer.

    libhdfs3 does not support arm64. So we should not support hdfs reader & writer on arm64.

2. And micro for LowerUpperImpl
 2021-10-16 21:56:49 +08:00
pengxiangyu
01bef4b40d [Load] Add "LOAD WITH HDFS" model, and make hdfs_reader support hdfs ha (#6161) 
Support load data from HDFS by using `LOAD WITH HDFS` syntax and read data directly via libhdfs3
 2021-07-10 10:11:52 +08:00
Zhengguo Yang
01a45e8691 add read buffer when use s3 reader (#5791) 2021-05-17 11:46:38 +08:00
pengxiangyu
29a3fa1084 [Feature] Support read data with format of parquet from hdfs, using libhdfs3 (#5686) 
Add new lib, Backend can read data from hdfs without broker,
this patch include libhdfs3.a which can read file on hdfs.
This patch will make reading the data from hdfs with parquet possible.
By this, we will support more format of file on hdfs in the future,
and we will support other metadata in the future.
 2021-04-24 17:41:48 +08:00