## Proposed changes
### log
```
at org.apache.doris.DorisFE.main(DorisFE.java:95) ~[doris-fe.jar:1.2-SNAPSHOT]
2024-07-09 11:56:53,826 WARN (mysql-nio-pool-0|316) [ConnectProcessor.handleQueryException():524] Process one query failed because unknown reason:
java.lang.NoSuchMethodError: 'void javax.ws.rs.core.NewCookie.<init>(java.lang.String, java.lang.String, java.lang.String, java.lang.String, int, java.lang.String, int, java.util.Date, boolean, boolean)'
at org.glassfish.jersey.message.internal.CookiesParser$MutableNewCookie.getImmutableNewCookie(CookiesParser.java:130) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.CookiesParser.parseNewCookie(CookiesParser.java:176) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.HttpHeaderReader.readNewCookie(HttpHeaderReader.java:335) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:87) ~[jersey-common-2.35.jar:?]
at org.glassfish.jersey.message.internal.NewCookieProvider.fromString(NewCookieProvider.java:34) ~[jersey-common-2.35.jar:?]
at javax.ws.rs.core.NewCookie.valueOf(NewCookie.java:126) ~[jsr311-api-1.1.1.jar:?]
at com.sun.jersey.api.client.ClientResponse.getCookies(ClientResponse.java:783) ~[jersey-bundle-1.19.3.jar:1.19.3]
at org.apache.ranger.admin.client.RangerAdminRESTClient.setCookieReceivedFromRoleDownloadSession(RangerAdminRESTClient.java:1364) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdatedWithCred(RangerAdminRESTClient.java:1220) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.admin.client.RangerAdminRESTClient.getRolesIfUpdated(RangerAdminRESTClient.java:167) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRolesFromAdmin(RangerRolesProvider.java:183) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.RangerRolesProvider.loadUserGroupRoles(RangerRolesProvider.java:123) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.PolicyRefresher.loadRoles(PolicyRefresher.java:495) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:144) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:245) ~[ranger-plugins-common-2.4.0.jar:2.4.0]
at org.apache.doris.catalog.authorizer.ranger.hive.RangerHivePlugin.<init>(RangerHivePlugin.java:25) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessController.<init>(RangerHiveAccessController.java:59) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.catalog.authorizer.ranger.hive.RangerHiveAccessControllerFactory.createAccessController(RangerHiveAccessControllerFactory.java:28) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.privilege.AccessControllerManager.createAccessController(AccessControllerManager.java:102) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.initAccessController(ExternalCatalog.java:326) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogFactory.createCatalog(CatalogFactory.java:163) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogFactory.createFromStmt(CatalogFactory.java:91) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogMgr.createCatalog(CatalogMgr.java:250) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.DdlExecutor.execute(DdlExecutor.java:381) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.handleDdlStmt(StmtExecutor.java:2992) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:1019) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:619) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:537) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:385) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:241) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:260) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:288) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:342) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
```
### changes
JAX-RS (Java API for RESTful Web Services) is the implementation of the
JSR 311 specification, providing the Java API for creating RESTful web
services. JAX-RS 1.0 was the first version based on the JSR 311
specification. Subsequent versions of JAX-RS, such as JAX-RS 2.0, have
extended and improved upon the initial JSR 311 specification while
maintaining compatibility. JSR 311 is the specification, and JAX-RS is
the actual implementation of that specification. They share the same
package name, but when both coexist in a project, it can lead to loading
exceptions.
Therefore, I excluded the jsr311-api jar and retained only jax-rs.
(cherry picked from commit 400c1fd5e02ede895a036df34a7fdf54323e9865)
## Proposed changes
Issue Number: close#37575
<!--Describe your changes.-->
Previously, FE logs were written to files. The main FE logs include
fe.log, fe.warn.log, fe.audit.log, fe.out, and fe.gc.log.
In a K8s deployment environment, logs usually need to be output to
standard output, and then other components process the log stream.
This PR made the following changes:
1. Modified the log4j configuration template
- When started with `--daemon`, logs are still written to various files,
and the format remains unchanged.
- When started with `--console`, all logs are output to standard output
and marked with different prefixes:
- `StdoutLogger`: logs for standard output
- `StderrLogger`: logs for standard error output
- `RuntimeLogger`: logs for fe.log or fe.warn.log
- `AuditLogger:` logs for fe.audit.log
- No prefix: logs for fe.gc.log
Examples are as follows:
```
RuntimeLogger 2024-06-03 14:54:51,229 INFO (binlog-gcer|62)
[BinlogManager.gc():359] begin gc binlog
```
2. Added a new FE config: `enable_file_logger`
Defaults to true. Indicates that logs will be recorded to files
regardless of the startup method. For example, if it is started with
`--console`, the log will be output to both the file and the standard
output. If it is `false`, the log will not be recorded in the file
regardless of the startup method.
3. Optimized the log format of standard output
The byte streams of stdout and stderr are captured. The logs previously
outputted using `System.out` will be captured in fe.log for unified
management.
followup #35241
In #35241, we update the doris-shade version to 2.1.0, which already contains dlf dependencies.
pick part of #34749, to remove dlf dependencies in fe/pom.xml
* Adapt paimon 0.6.0 (#33943)
Version 2.0.0 of the shade package eliminates potential jar conflicts, resolves dependency component issues, and significantly reduces package size.
Utilize the directly-dependent guava library instead of relying on transitively included libraries.
* [chore](dependencies)Upgrade paimon to 0.7.0 (#33987)
---------
Co-authored-by: Calvin Kirs <kirs@apache.org>
The tomcat-embed-el dependency is primarily used for standardizing EL functionality, which we don't require in our application. Therefore, we can safely remove it.
This PR makes the following changes to the connection pool of JDBC Catalog
1. Set the maximum connection survival time, the default is 30 minutes
- Moreover, one-half of the maximum survival time is the recyclable time,
- One-tenth is the check interval for recycling connections
2. Keepalive only takes effect on the connection pool on BE, and will be activated based on one-fifth of the maximum survival time.
3. The maximum number of existing connections is changed from 100 to 10
4. Add the connection cache recycling thread on BE, and add a parameter to control the recycling time, the default is 28800 (8 hours)
5. Add CatalogID to the key of the connection pool cache to achieve better isolation, requires refresh catalog to take effect
6. Upgrade druid connection pool to version 1.2.20
7. Added JdbcResource's setting of default parameters when upgrading the FE version to avoid errors due to unset parameters.
Issue Number: close#30484
problem:
gson will use Java's reflection mechanism to generate a default Adapter, but JDK17 is prohibited from visiting such an access.
solution:
gson has provided solutions since 2.9.1, which can bypass this problem: Add support for reflection access filter by Marcono1234 · Pull Request #1905 · google/gson
We need to upgrade the gson version and use this solution
The current logic for SQL dialect conversion is all in the `fe-core` module, which may lead to the following issues:
- Changes to the dialect conversion logic may occur frequently, requiring users to upgrade the Doris version frequently within the fe-core module, leading to a longer change cycle.
- The cost of customized development is high, requiring users to replace the fe-core JAR package.
Turning it into a plugin can address the above issues properly.
Previously temporarily upgrade Arrow to dev version 15.0.0-SNAPSHOT, because the latest release version Arrow 14.0.1 jdbc:arrow-flight-sql has BUG, jdbc:arrow-flight-sql cannot be used normally, see: apache/arrow#38785
But Arrow 15.0.0-SNAPSHOT was not published to the Maven central repository, and the network could not be connected sometimes, so back to Arrow 14.0.1. jdbc:arrow-flight-sql will be supported after upgrading to Arrow 15.0.0 release version.
This reverts commit 5b641ebd40fff71e632ee9be4ede58b744b602b9.
Currently, Deltalake Catalog is not a usable feature. We will continue to implement it in the datalake plug-in system in the future, so we will delete it from the FE code for now.
Support transforming trino dialect SQL to logical plan (#21854)
## Proposed changes
Issue Number: #21854
Use io.trino.sql.tree.AstVisitor as vistor, visit coorresponding trino node and transform it to doris logical plan.
## Further comments
Here are some examples for function transforming as following:
**ascii('a')** function is in doris and **codepoint('a')** funtion in trino, they have the same feature and have the same method signature, so we can use [TrinoFnCallTransformer](3b37b76886/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/trino/TrinoFnCallTransformer.java) to handle them.
another example for ComplexTransformer as following:
**date_diff('second', TIMESTAMP '2020-12-25 22:00:00', TIMESTAMP '2020-12-25 21:00:00')"** fuction in trino
and **seconds_diff(2020-12-25 22:00:00, 2020-12-25 21:00:00)")** fuction in doris. They have different method signature, we cant not handle it by TrinoFnCallTransformer simply and we should handle it by individual complex transformer [DateDiffFnCallTransformer](3b37b76886/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/trino/DateDiffFnCallTransformer.java).
PR https://github.com/apache/doris/pull/24606 has updated hbase version to 2.5.5, but it conflict with hudi, causing error like:
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: Failed to get hudi partitions
at org.apache.doris.qe.StmtExecutor.analyze(StmtExecutor.java:1021) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.qe.StmtExecutor.executeByLegacy(StmtExecutor.java:696) ~[doris-fe.jar:1.2-SNAPSHOT]
...
Caused by: java.lang.NullPointerException
at org.apache.hadoop.fs.FilterFileSystem.getConf(FilterFileSystem.java:524) ~[hadoop-common-3.3.6.jar:?]
at org.apache.hadoop.hbase.io.hfile.ReaderContext.<init>(ReaderContext.java:53) ~[hbase-server-2.5.5.jar:2.5.5]
at org.apache.hadoop.hbase.io.hfile.ReaderContextBuilder.build(ReaderContextBuilder.java:106) ~[hbase-server-2.5.5.jar:2.5.5]
```
* be scanner
- Upgrade avro to 1.11.2
fe
- Upgrade quartz to 2.5.0-rc1
- Upgrade maxcompute to 0.45-2-publish
- Binding avro-ipc to 1.11.2
* Binding hbase version to 2.5.5
binding nimbusds version to 9.35
