00dda79735
[fix](broker-load) Correction of kerberos authentication time determination rule ( #11793 )
...
Every time a new broker load comes in, Doris will update the start time of Kerberos authentication,
but this logic is wrong.
Because the authentication duration of Kerberos is calculated from the moment when the ticket is obtained.
This PR change the logic:
1. If it is kerberos, check fs expiration by create time.
2.Otherwise, check fs expiration by access time
2022-09-18 17:46:13 +08:00
d7e032bc38
Modify the startup script and print the log without using the --daemon parameter. ( #12218 )
2022-08-31 14:36:14 +08:00
4fa53b4cdb
[chore](workflow) Add shellcheck to check shell scripts ( #11744 )
2022-08-18 16:07:28 +08:00
583b44dfa8
[enhancement](broker) Improve the availability of broker load ( #10699 )
2022-08-09 17:00:48 +08:00
388db05ef9
[bugfix](log4j) Upgrade log4j to 2.18.0 ( #11368 )
2022-07-31 22:21:33 +08:00
67f341f44e
[TLP](step-1) Remove incubator prefix ( #10230 )
...
Remove some `incubator-` prefix in source code.
The document is not modified, will be done in next PR.
2022-06-19 19:34:52 +08:00
4ccaa0dfc5
[Bug] (load) Broker load kerberos auth fail ( #9494 )
2022-05-12 15:43:29 +08:00
419ec3b96c
[Fix Bug] Fix ehco command not found ( #9021 )
2022-04-15 13:43:47 +08:00
6af1c52e13
[Feature] add support for tencent chdfs ( #8963 )
...
Co-authored-by: chengwu <chengwu@tencent.com >
2022-04-12 16:02:42 +08:00
13f1f94f86
[chore] upgrade log4j version to 2.17.2 ( #8774 )
...
upgrade log4j version to 2.17.2
2022-04-02 21:29:25 +08:00
50a59f3f86
[license] Organize third-party dependent licenses for bianry releases ( #8350 )
2022-03-07 23:18:58 +08:00
c0e59e59aa
[fix][refactor] fix bugs and refactor some code by lint ( #7871 )
...
1. Fix some `passedByValue` issues.
2. Fix some `dereferenceBeforeCheck` issues.
3. Fix some `uninitMemberVar` issues.
4. Fix some iterator `eraseDereference` issues.
5. Fix compile issue introduced from #7923 #7905 #7848
2022-02-01 14:31:14 +08:00
4bdeef3b64
[chore][fix][doc](fe-plugin)(mysqldump) fix build auditlog plugin error ( #7804 )
...
1. fix problems when build fe_plugins
2. format
3. add docs about dump data using mysql dump
2022-01-26 09:11:23 +08:00
946fa2960d
[improvement](broker) add some properties that can be set in the broker conf file ( #7499 )
2022-01-18 10:24:54 +08:00
ad35067a2a
[chore][docs] add deploy spark/flink connectors to maven release repo docs ( #7616 )
2022-01-06 23:23:33 +08:00
738d2d2e07
[refactor] update parent pom version and optimize build scripts ( #7548 )
2022-01-05 10:45:11 +08:00
a60d86c1e1
[improvement](broker) add disable cache config for broker ( #7506 )
2021-12-31 16:48:55 +08:00
85c30fc720
[deps] Upgrade Log4j to 2.7.1 to solve the CVE-2021-44832 security vulnerability ( #7536 )
...
Upgrade Log4j to 2.7.1 to solve the CVE-2021-44832 security vulnerability
Co-authored-by: Zhengguo Yang <yangzhgg@gmail.com >
2021-12-30 10:21:37 +08:00
bc4ceeca44
[improvement] optimize java cmd find ( #7428 )
...
* optimize java cmd find, if java_home not set use java in PATH
2021-12-30 10:16:56 +08:00
2872dbfeb8
[refactor] Standardize the writing of pom files, prepare for deployment to maven ( #7477 )
2021-12-30 10:16:37 +08:00
755e0693b9
[feature](broker) support ks3 for kmr in ksyun ( #7484 )
2021-12-26 11:10:47 +08:00
7a1bb5b335
log4j upgrade to 2.17.0 ( #7440 )
...
Solved the third security vulnerability CVE-2021-45105 that was discovered
2021-12-21 09:28:02 +08:00
e64da03866
[deps](log4j) Upgrade log4j 2 to 2.16.0 ( #7394 )
...
Upgrade log4j 2 to 2.16.0, the official strongly recommends upgrading to this version
2021-12-14 15:57:16 +08:00
568f6611df
[deps](log4j) upgrade log4j ( #7364 )
...
to 2.15.0
2021-12-10 23:19:11 +08:00
dcad6ff5e5
[License] Add License header for missing files ( #7130 )
...
1. Add License header for missing files
2. Modify the spark pom.xml to correct the location of `thrift`
2021-11-16 18:37:54 +08:00
adb6bfdf74
[Bug] Fix bug that truncate table may change the storage medium property ( #6905 )
2021-10-25 10:07:27 +08:00
24d38614a0
[Dependency] Upgrade thirdparty libs ( #6766 )
...
Upgrade the following dependecies:
libevent -> 2.1.12
OpenSSL 1.0.2k -> 1.1.1l
thrift 0.9.3 -> 0.13.0
protobuf 3.5.1 -> 3.14.0
gflags 2.2.0 -> 2.2.2
glog 0.3.3 -> 0.4.0
googletest 1.8.0 -> 1.10.0
snappy 1.1.7 -> 1.1.8
gperftools 2.7 -> 2.9.1
lz4 1.7.5 -> 1.9.3
curl 7.54.1 -> 7.79.0
re2 2017-05-01 -> 2021-02-02
zstd 1.3.7 -> 1.5.0
brotli 1.0.7 -> 1.0.9
flatbuffers 1.10.0 -> 2.0.0
apache-arrow 0.15.1 -> 5.0.0
CRoaring 0.2.60 -> 0.3.4
orc 1.5.8 -> 1.6.6
libdivide 4.0.0 -> 5.0
brpc 0.97 -> 1.0.0-rc02
librdkafka 1.7.0 -> 1.8.0
after this pr compile doris should use build-env:1.4.0
2021-10-15 13:03:04 +08:00
f6bcabe0d1
[Bug] Fixed bug that caused export and backup to fail when principal keytab file was created failed ( #6404 )
...
Co-authored-by: Geoffrey <gaofeng01@rd.netease.com >
2021-08-11 16:39:01 +08:00
2f3cd0573a
[Broker] Fix ugi confusion bug ( #6325 )
...
Use UserGroupInformation.loginUserFromKeytabAndReturnUGI instead of UserGroupInformation.loginUserFromKeytab in multiple principal scenario.
2021-08-05 14:33:18 +08:00
12730e7a3b
[Broker] Fix bug that broker can not handle FSDataInputStream which does not implement ByteBufferReadable ( #6308 )
...
Currently, Doris supports loading OSS/S3A files by using params like fs.s3a.access.key, but there is a bug when using it to load such type files. The root cause is broker can not handle FSDataInputStream which does not implement ByteBufferReadable.
See Issue #6307
S3A input stream to support ByteBufferReadable
https://issues.apache.org/jira/browse/HADOOP-14603
2021-08-05 14:16:36 +08:00
904a2ac86a
[Bug] keytab file maybe not thread-safe ( #5578 )
...
* make keytab file thread-safe
* remove delte file code
Co-authored-by: wangxixu <wangxixu@xiaomi.com >
2021-04-08 09:12:45 +08:00
fdd13ea79a
[Broker] Fix broker load fail using SIMPLE auth after KERBEROS auth fail ( #5412 )
...
Co-authored-by: liwei5 <liwei5@vipkid.com.cn >
2021-02-25 09:53:50 +08:00
67b842ce04
[License] Organize and modify the license of the code ( #4371 )
...
1. Disable the MySQL client and LZO library by default when building the Doris.
MySQL client library is used for MySQL external table feature.
This feature will be replaced by the new ODBC external table soon.
LZO library is used to compress/decompress data of some old data format of Doris,
which is no longer used anymore.
2. Add missing license to some files.
3. For all non-Apache-License code, all are explained in NOTICE file and the corresponding license is declared.
4. Remove the js source code from webroot, it will be downloaded as thirdparty
2020-08-24 21:51:55 +08:00
3fcbb646f6
[Bug][Broker] Fix bug that the error exception is not caught when closing the broker writer ( #4034 )
...
Broker should return the error status.
Fix #4033
2020-07-10 20:42:39 +08:00
8e895958d6
[Bug] Checkpoint thread is not running ( #3913 )
...
This bug is introduced by PR #3784
In #3784 , I remove the `Catalog.getInstance()`, and use `Catalog.getCurrentCatalog()` instead.
But actually, there are some place that should use the serving catalog explicitly.
Mainly changed:
1. Add a new method `getServingCatalog()` to explicitly return the real catalog instance.
2. Fix a compile bug of broker introduced by #3881
2020-06-20 09:32:14 +08:00
1d9fa5071d
[BUG][Broker] Fix broker read buffer size from input stream ( #3881 )
...
This commit fixs a bug that broker cannot read the full length of buffer size, when the buffer size is set larger than 128k.
This bug will cause the data size returned by pread request to be less than 128K all the time.
2020-06-19 09:33:09 +08:00
ee260d5721
[Bug][FsBroker] NPE throw when username is empty ( #3731 )
...
When using Broker with an empty username, a NPE is thrown, which is
not expected.
2020-06-01 21:03:21 +08:00
79bac50361
Fix the bug that 'username' in broker load is invalid ( #3237 )
2020-04-06 22:15:37 +08:00
4a5164ab9d
Fix 'Filesystem closed' in broker load ( #3216 )
2020-03-28 09:14:45 +08:00
c032d634f4
[FsBroker] Fix bug that broker cannot read file with %3A in name ( #3028 )
...
The hdfs support file with name like: "2018-01-01 00%3A00%3A00",
we should support it.
Also change the default broker log level to INFO.
2020-03-04 11:03:01 +08:00
078e35a62e
Support Amazon S3 data source in Broker Load ( #3004 )
2020-03-01 12:53:50 +08:00
35b09ecd66
[JDK] Support OpenJDK ( #2804 )
...
Support compile and running Frontend process and Broker process with OpenJDK.
OpenJDK 13 is tested.
2020-02-20 23:47:02 +08:00
03c8bc91ca
Change MVN to MVN_CMD same as #2837 for builing hdfs broker ( #2874 )
2020-02-10 18:40:12 +08:00
2581d6d574
Support hdfs broker reading config from hdfs-site.xml ( #2148 )
...
In the production environment, hdfs broker should read config from hdfs-site.xml instead of manual input due to the complexity of hdfs config
add dfs.namenode.kerberos.principal.pattern config for kerberos auth
fix some getting config code for brief
2019-11-07 22:23:31 +08:00
78d7a8f315
Add Apache license header in config files ( #2081 ) ( #2110 )
2019-10-31 17:21:32 +08:00
e67b398916
Fix bug that backup may create an empty file on remote storage. ( #1869 )
...
Sometime the broker writer failed to close, but we do not handle this failure.
This may create an empty file on remote storage but be treated as normal.
Also enhance some usabilities:
1. getting latest 2000 transactions instead of getting the earliest.
2. Show backend which download and upload tasks are being executed.
2019-09-28 00:11:43 +08:00
b4ba77a594
Fix bug that encounter "No more data to read" when accessing broker ( #1621 )
...
The errorMsg in TBrokerOperationStatus is set to null because of
invalid string joint operation.
2019-08-12 14:06:47 +08:00
97718a35a2
Do not get file size in Broker openReader() method ( #1560 )
...
The file is already got when listing files.
Get file size in openReader() again is unnecessary and inefficient.
2019-07-29 23:05:01 +08:00
7eab12a40e
Support reading Parquet file when loading data ( #1173 )
2019-07-01 18:39:27 +08:00
3cc78fd952
Add trim in hdfs broker name node string ( #1028 )
2019-04-25 13:48:20 +08:00
