[fix](multi-catalog) fix image loading failture when create catalog with resource (#15692)
Bug fix
fix image loading failture when create catalog with resource
When creating jdbc catalog with resource, the metadata image will failed to be loaded.
Because when loading jdbc catalog image, it will try to get resource from ResourceMgr,
but ResourceMgr has not been loaded, so NPE will be thrown.
This PR fix this bug, and refactor some logic about catalog and resource.
When loading jdbc catalog image, it will not get resource from ResourceMgr.
And now user can create catalog with resource and properties, like:
create catalog jdbc_catalog with resource jdbc_resource
properites("user" = "user1");
The properties in "properties" clause will overwrite the properties in "jdbc_resource".
force adding tinyInt1isBit=false to jdbc url
The default value of tinyInt1isBit is true, and it will cause tinyint in mysql to be bit type.
force adding tinyInt1isBit=false to jdbc url so that the tinyint in mysql will be tinyint in Doris.
Avoid calculate checksum of jdbc driver jar multiple times
Refactor
Refactor the notification logic when updating properties in resource.
When updating properties in resource, it will notify the corresponding catalog to update its own properties.
This PR change this logic. After updating properties in resource, it will only uninitialize the catalog's internal
objects such "jdbc client" or "hms client". And this objects will be re-initialized lazily.
And all properties will be got from Resource at runtime, so that it will always get the latest properties
Regression test cases
Because we add tinyInt1isBit=false to jdbc url, some of cases need to be changed.
This commit is contained in:
@ -99,12 +99,16 @@ CREATE RESOURCE hms_resource PROPERTIES (
|
||||
'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007',
|
||||
'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'
|
||||
);
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource;
|
||||
|
||||
// The properties in 'PROPERTIES' will overwrite the properties in "Resource"
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource PROPERTIES(
|
||||
'key' = 'value'
|
||||
);
|
||||
```
|
||||
|
||||
**Create catalog through properties**
|
||||
|
||||
Version `1.2.0` creates a catalog through properties. This method will be deprecated in subsequent versions.
|
||||
Version `1.2.0` creates a catalog through properties.
|
||||
```sql
|
||||
CREATE CATALOG hive PROPERTIES (
|
||||
'type'='hms',
|
||||
|
||||
@ -39,14 +39,14 @@ Syntax:
|
||||
```sql
|
||||
CREATE CATALOG [IF NOT EXISTS] catalog_name
|
||||
[WITH RESOURCE resource_name]
|
||||
| [PROPERTIES ("key"="value", ...)];
|
||||
[PROPERTIES ("key"="value", ...)];
|
||||
```
|
||||
|
||||
`RESOURCE` can be created from [CREATE RESOURCE](../../../sql-reference/Data-Definition-Statements/Create/CREATE-RESOURCE.md), current supports:
|
||||
|
||||
* hms:Hive MetaStore
|
||||
* es:Elasticsearch
|
||||
* jdbc:数据库访问的标准接口(JDBC), 当前只支持`jdbc:mysql`
|
||||
* jdbc: Standard interface for database access (JDBC), currently supports MySQL and PostgreSQL
|
||||
|
||||
### Create catalog
|
||||
|
||||
@ -58,12 +58,14 @@ CREATE RESOURCE catalog_resource PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
...
|
||||
);
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource;
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource PROPERTIES (
|
||||
'key' = 'value'
|
||||
);
|
||||
```
|
||||
|
||||
**Create catalog through properties**
|
||||
|
||||
Version `1.2.0` creates a catalog through properties. This method will be deprecated in subsequent versions.
|
||||
Version `1.2.0` creates a catalog through properties.
|
||||
```sql
|
||||
CREATE CATALOG catalog_name PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
|
||||
@ -99,12 +99,16 @@ CREATE RESOURCE hms_resource PROPERTIES (
|
||||
'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007',
|
||||
'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'
|
||||
);
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource;
|
||||
|
||||
// 在 PROERPTIES 中指定的配置,将会覆盖 Resource 中的配置。
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource PROPERTIES(
|
||||
'key' = 'value'
|
||||
);
|
||||
```
|
||||
|
||||
**通过 properties 创建 catalog**
|
||||
|
||||
`1.2.0` 版本通过 properties 创建 catalog,该方法将在后续版本弃用。
|
||||
`1.2.0` 版本通过 properties 创建 catalog
|
||||
```sql
|
||||
CREATE CATALOG hive PROPERTIES (
|
||||
'type'='hms',
|
||||
|
||||
@ -43,14 +43,14 @@ CREATE CATALOG
|
||||
```sql
|
||||
CREATE CATALOG [IF NOT EXISTS] catalog_name
|
||||
[WITH RESOURCE resource_name]
|
||||
| [PROPERTIES ("key"="value", ...)];
|
||||
[PROPERTIES ("key"="value", ...)];
|
||||
```
|
||||
|
||||
`RESOURCE` 可以通过 [CREATE RESOURCE](../../../sql-reference/Data-Definition-Statements/Create/CREATE-RESOURCE.md) 创建,目前支持三种 Resource,分别连接三种外部数据源:
|
||||
|
||||
* hms:Hive MetaStore
|
||||
* es:Elasticsearch
|
||||
* jdbc:数据库访问的标准接口(JDBC), 当前只支持`jdbc:mysql`
|
||||
* jdbc:数据库访问的标准接口(JDBC), 当前支持 MySQL 和 PostgreSQL
|
||||
|
||||
### 创建 catalog
|
||||
|
||||
@ -62,12 +62,16 @@ CREATE RESOURCE catalog_resource PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
...
|
||||
);
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource;
|
||||
|
||||
// 在 PROERPTIES 中指定的配置,将会覆盖 Resource 中的配置。
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource PROPERTIES(
|
||||
'key' = 'value'
|
||||
)
|
||||
```
|
||||
|
||||
**通过 properties 创建 catalog**
|
||||
|
||||
`1.2.0` 版本通过 properties 创建 catalog,该方法将在后续版本弃用。
|
||||
`1.2.0` 版本通过 properties 创建 catalog。
|
||||
```sql
|
||||
CREATE CATALOG catalog_name PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
|
||||
Reference in New Issue
Block a user