## Proposed changes
upgrade commons-configuration2 to 2.11.0
upgrade logging-interceptor to 4.12.0
upgrade commons-compress to 1.27.1
upgrade jetty-bom to 9.4.56.v20240826
upgrade azure-sdk to 1.2.27
Iceberg depends on configuration2, and configuration2 relies on a newer
version of commons-lang3. However, there were significant breaking
changes in commons-lang3, which made it
incompatible.https://issues.apache.org/jira/browse/LANG-1705 As a
result, I rewrote the clone method.
(cherry picked from commit 945edf8dbffaa25c987bcefad59b6cde52772d4f)
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
bp #40225 , #40888 ,#41386
## Proposed changes
Among them, #40225 is the new api of mc,
#40888 is used to fix the bug when reading null between the new and old
apis,
#41386 is used for compatibility between the new and old versions
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
(cherry picked from commit a692af226a8292d1b0a409c914d616157b64c82a)
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
…
## Proposed changes
javax.el is the API for Java Expression Language, which provides a
simple and flexible way for Java Web applications to access and
manipulate data. EL expressions are commonly used in JSP pages, but we
are not involved in their use here, so I removed it.
- upgrade jackson to 2.16.0
(cherry picked from commit 33a0ea5565c2e0d7e5909ca3b512fb0b539f05f4)
#38843
Issue Number: close #xxx
<!--Describe your changes.-->
…
improve the flexibility of the project by decoupling direct dependencies
on the hadoop-cos and hadoop-huaweicloud libraries. These changes allow
users to control whether COS and OBS dependencies are included in the
final build, enabling a more customizable setup.
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
pick (#39341)
In previous versions, we used a method based on JDBC 4.2 to read data,
so it was equivalent to abandoning support for ojdbc6. However, we
recently found that a large number of users still use Oracle version
11g, which will have some unexpected compatibility issues when using
ojdbc8 to connect. Therefore, I use version verification to make it
compatible with both ojdbc6 and ojdbc8, so that good compatibility can
be obtained through ojdbc6, and better reading efficiency can be
obtained through ojdbc8.
## Proposed changes
upgrade spring-boot to 2.7.18
upgrade zookeeper to 3.9.2
upgrade jetty to 9.4.55.v20240627
upgrade ivy to 2.5.2
upgrade icu4j to 75.1
upgrade ini4j to 0.5.4
(cherry picked from commit 3f633c2018e86c6c842647262853d88ad63672bf)
pick #38509
## Proposed changes
Issue Number: close #xxx
<!--Describe your changes.-->
## Proposed changes
### log
```
at org.apache.doris.DorisFE.main(DorisFE.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-07-09 11:56:53,826 WARN (mysql-nio-pool-0|316) [ConnectProcessor.handleQueryException():524] Process one query failed because unknown reason:
java.lang.NoSuchMethodError: 'void javax.ws.rs.core.NewCookie.<init>(java.lang.String, java.lang.String, java.lang.String, java.lang.String, int, java.lang.String, int, java.util.Date, boolean, boolean)'
at org.glassfish.jersey.message.internal.CookiesParser$MutableNewCookie.getImmutableNewCookie(CookiesParser.java:130) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.CookiesParser.parseNewCookie(CookiesParser.java:176) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.HttpHeaderReader.readNewCookie(HttpHeaderReader.java:335) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:87) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:34) ~[jersey-common-2.35.jar:?]
at javax.ws.rs.core.NewCookie.valueOf(NewCookie.java:126) ~[jsr311-api-1.1.1.jar:?]
at com.sun.jersey.api.client.ClientResponse.getCookies(ClientResponse.java:783) ~[jersey-bundle-1.19.3.jar:1.19.3]
at org.apache.ranger.admin.client.RangerAdminRESTClient.setCookieReceivedFromRoleDownloadSession(RangerAdminRESTClient.java:1364) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1220) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.doris.catalog.authorizer.ranger.hive.RangerHivePlugin.<init>(RangerHivePlugin.java:25) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessController.<init>(RangerHiveAccessController.java:59) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessControllerFactory.createAccessController(RangerHiveAccessControllerFactory.java:28) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.privilege.AccessControllerManager.createAccessController(AccessControllerManager.java:102) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.initAccessController(ExternalCatalog.java:326) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogFactory.createCatalog(CatalogFactory.java:163) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogFactory.createFromStmt(CatalogFactory.java:91) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogMgr.createCatalog(CatalogMgr.java:250) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:381) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:2992) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1019) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:619) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:537) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:385) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:241) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:260) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:288) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:342) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
### changes
JAX-RS (Java API for RESTful Web Services) is the implementation of the
JSR 311 specification, providing the Java API for creating RESTful web
services. JAX-RS 1.0 was the first version based on the JSR 311
specification. Subsequent versions of JAX-RS, such as JAX-RS 2.0, have
extended and improved upon the initial JSR 311 specification while
maintaining compatibility. JSR 311 is the specification, and JAX-RS is
the actual implementation of that specification. They share the same
package name, but when both coexist in a project, it can lead to loading
exceptions.
Therefore, I excluded the jsr311-api jar and retained only jax-rs.
(cherry picked from commit 400c1fd5e02ede895a036df34a7fdf54323e9865)
## Proposed changes
Issue Number: close#37575
<!--Describe your changes.-->
Previously, FE logs were written to files. The main FE logs include
fe.log, fe.warn.log, fe.audit.log, fe.out, and fe.gc.log.
In a K8s deployment environment, logs usually need to be output to
standard output, and then other components process the log stream.
This PR made the following changes:
1. Modified the log4j configuration template
- When started with `--daemon`, logs are still written to various files,
and the format remains unchanged.
- When started with `--console`, all logs are output to standard output
and marked with different prefixes:
- `StdoutLogger`: logs for standard output
- `StderrLogger`: logs for standard error output
- `RuntimeLogger`: logs for fe.log or fe.warn.log
- `AuditLogger:` logs for fe.audit.log
- No prefix: logs for fe.gc.log
Examples are as follows:
```
RuntimeLogger 2024-06-03 14:54:51,229 INFO (binlog-gcer|62)
[BinlogManager.gc():359] begin gc binlog
```
2. Added a new FE config: `enable_file_logger`
Defaults to true. Indicates that logs will be recorded to files
regardless of the startup method. For example, if it is started with
`--console`, the log will be output to both the file and the standard
output. If it is `false`, the log will not be recorded in the file
regardless of the startup method.
3. Optimized the log format of standard output
The byte streams of stdout and stderr are captured. The logs previously
outputted using `System.out` will be captured in fe.log for unified
management.
followup #35241
In #35241, we update the doris-shade version to 2.1.0, which already contains dlf dependencies.
pick part of #34749, to remove dlf dependencies in fe/pom.xml
* Adapt paimon 0.6.0 (#33943)
Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.
* [chore](dependencies)Upgrade paimon to 0.7.0 (#33987)
---------
Co-authored-by: Calvin Kirs <kirs@apache.org>
The tomcat-embed-el dependency is primarily used for standardizing EL functionality, which we don't require in our application. Therefore, we can safely remove it.
This PR makes the following changes to the connection pool of JDBC Catalog
1. Set the maximum connection survival time, the default is 30 minutes
- Moreover, one-half of the maximum survival time is the recyclable time,
- One-tenth is the check interval for recycling connections
2. Keepalive only takes effect on the connection pool on BE, and will be activated based on one-fifth of the maximum survival time.
3. The maximum number of existing connections is changed from 100 to 10
4. Add the connection cache recycling thread on BE, and add a parameter to control the recycling time, the default is 28800 (8 hours)
5. Add CatalogID to the key of the connection pool cache to achieve better isolation, requires refresh catalog to take effect
6. Upgrade druid connection pool to version 1.2.20
7. Added JdbcResource's setting of default parameters when upgrading the FE version to avoid errors due to unset parameters.
Issue Number: close#30484
problem:
gson will use Java's reflection mechanism to generate a default Adapter, but JDK17 is prohibited from visiting such an access.
solution:
gson has provided solutions since 2.9.1, which can bypass this problem: Add support for reflection access filter by Marcono1234 · Pull Request #1905 · google/gson
We need to upgrade the gson version and use this solution
The current logic for SQL dialect conversion is all in the `fe-core` module, which may lead to the following issues:
- Changes to the dialect conversion logic may occur frequently, requiring users to upgrade the Doris version frequently within the fe-core module, leading to a longer change cycle.
- The cost of customized development is high, requiring users to replace the fe-core JAR package.
Turning it into a plugin can address the above issues properly.
Previously temporarily upgrade Arrow to dev version 15.0.0-SNAPSHOT, because the latest release version Arrow 14.0.1 jdbc:arrow-flight-sql has BUG, jdbc:arrow-flight-sql cannot be used normally, see: apache/arrow#38785
But Arrow 15.0.0-SNAPSHOT was not published to the Maven central repository, and the network could not be connected sometimes, so back to Arrow 14.0.1. jdbc:arrow-flight-sql will be supported after upgrading to Arrow 15.0.0 release version.
This reverts commit 5b641ebd40fff71e632ee9be4ede58b744b602b9.
Currently, Deltalake Catalog is not a usable feature. We will continue to implement it in the datalake plug-in system in the future, so we will delete it from the FE code for now.
