Commit Graph

14 Commits

Author SHA1 Message Date
d800434859 [Chore](Fe)Upgrade dependencies (#38509) (#38747)
## Proposed changes

upgrade spring-boot to 2.7.18
upgrade zookeeper to 3.9.2
upgrade jetty to 9.4.55.v20240627
upgrade ivy to 2.5.2
upgrade icu4j to 75.1
upgrade ini4j to 0.5.4

(cherry picked from commit 3f633c2018e86c6c842647262853d88ad63672bf)

pick #38509
## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
2024-08-02 12:34:17 +08:00
d91376cd52 [bugfix](paimon)adding dependencies for clang #37512 (#37737)
cherry pick from #37512

Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>
2024-07-13 09:59:35 +08:00
9dc1196f1c [update](hudi) update hudi-spark bundle to 3.4.3 (#35013) (#35718)
backport: #35013
2024-05-31 20:51:28 +08:00
2e20e38523 [improvement](jdbc catalog) remove useless jdbc catalog code (#34986) (#35418) 2024-05-27 14:25:26 +08:00
3495ed58e0 [Enhancement](jdbc catalog) Change Jdbc connection pool to hikari (#34045) (#34310) 2024-04-29 20:22:48 +08:00
fbd2c9db2d [upgrade](hive-shade)(paimon) upgrade hive shade to 2.0.0 and paimon to 0.7 (#34085)
* Adapt paimon 0.6.0 (#33943)

Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.

* [chore](dependencies)Upgrade paimon to 0.7.0 (#33987)

---------

Co-authored-by: Calvin Kirs <kirs@apache.org>
2024-04-25 12:01:44 +08:00
e9b67bc82d [bugfix](paimon)merge meta-inf/services for paimon FileIOLoader (#33166)
We introduced paimon's oss and s3 packages, but did not register them in meta-info/service. As a result, when be used the s3  or oss interface, an error was reported and the class could not be found(`Could not find a file io implementation for scheme 's3' in the classpath.`).

FYI:
https://stackoverflow.com/questions/47310215/merging-meta-inf-services-files-with-maven-assembly-plugin
https://stackoverflow.com/questions/1607220/how-can-i-merge-resource-files-in-a-maven-assembly
2024-04-07 22:13:00 +08:00
9c6180d9ba [revert](jni) revert part of #32455 #32904 2024-03-27 20:45:44 +08:00
c0d7a5660e [fix](paimon) support paimon with hive2 (#32455)
In order to support paimon with hive2, we need to modify the origin HiveMetastoreClient.java
to let it compatible with both hive2 and hive3.
And this modified HiveMetastoreClient should be at the front of the CLASSPATH, so that
it can overwrite the HiveMetastoreClient in hadoop jar.

This PR mainly changes:

1. Copy HiveMetastoreClient.java in FE to BE's preload jar.

2. Split the origin `preload-extensions-jar-with-dependencies.jar` into 2 jars
    1. `preload-extensions-project.jar`, which contains the modified HiveMetastoreClient.
    2. `preload-extensions-jar-with-dependencies.jar`, which contains other dependency jars.

3. Modify the `start_be.sh`, to let `preload-extensions-project.jar` be loaded first.

4. Change the way the assemble the jni scanner jar
    Only need to assemble the project jar, without other dependencies.
    Because actually we only use classed under `org.apache.doris` package.
    So remove other unused dependency jars can also reduce the output size of BE.

5. fix bug that the prefix of paimon properties should be `paimon.`, not `paimon`

6. Support paimon with hive2
    User can set `hive.version` in paimon catalog properties to specify the hive version.
2024-03-26 15:31:07 +08:00
450422556a [fix](paimon) auto deplay paimon oss/s3 jar file (#31568)
No need to deploy paimon oss/s3 jar files manually.
Include them in preload-extensions-jar-with-dependencies.jar
2024-02-29 16:44:40 +08:00
9b7c6af581 [Fix](JDK17) Fixed that BE could not be started using JDK17 (#30286)
Issue Number: #30484

This is because hadoop-client-api relies on hadoop-common.
In the case of JDK17, it will still include hadoop-common.
2024-02-01 23:14:14 +08:00
12af86176a [fix](class-loader) fix class loader conflict on BE side (#29942)
1. make `hadoop-common` in be java extension as `provided`.
2. must load be java extension jars before hadoop jars
2024-01-14 15:53:33 +08:00
f66f161017 [fix](multi-catalog)fix hive table with cosn location issue (#23409)
Sometimes, the partitions of a hive table may on different storage, eg, some is on HDFS, others on object storage(cos, etc).
This PR mainly changes:

1. Fix the bug of accessing files via cosn.
2. Add a new field `fs_name` in TFileRangeDesc
    This is because, when accessing a file, the BE will get a hdfs client from hdfs client cache, and different file in one query
request may have different fs name, eg, some of are `hdfs://`, some of are `cosn://`, so we need to specify fs name
for each file, otherwise, it may return error:

`reason: IllegalArgumentException: Wrong FS: cosn://doris-build-1308700295/xxxx, expected: hdfs://[172.xxxx:4007](http://172.xxxxx:4007/)`
2023-08-26 00:16:00 +08:00
5ba505ebf4 [fix](multi-catalog)fix avro and jdbc scanner dependency (#23015)
add preload-extensions module, put all conflict dependencies to pom.xml in preload-extensions
2023-08-20 19:28:17 +08:00