[fix](multi-catalog) fix image loading failture when create catalog with resource (#15692)
Bug fix
fix image loading failture when create catalog with resource
When creating jdbc catalog with resource, the metadata image will failed to be loaded.
Because when loading jdbc catalog image, it will try to get resource from ResourceMgr,
but ResourceMgr has not been loaded, so NPE will be thrown.
This PR fix this bug, and refactor some logic about catalog and resource.
When loading jdbc catalog image, it will not get resource from ResourceMgr.
And now user can create catalog with resource and properties, like:
create catalog jdbc_catalog with resource jdbc_resource
properites("user" = "user1");
The properties in "properties" clause will overwrite the properties in "jdbc_resource".
force adding tinyInt1isBit=false to jdbc url
The default value of tinyInt1isBit is true, and it will cause tinyint in mysql to be bit type.
force adding tinyInt1isBit=false to jdbc url so that the tinyint in mysql will be tinyint in Doris.
Avoid calculate checksum of jdbc driver jar multiple times
Refactor
Refactor the notification logic when updating properties in resource.
When updating properties in resource, it will notify the corresponding catalog to update its own properties.
This PR change this logic. After updating properties in resource, it will only uninitialize the catalog's internal
objects such "jdbc client" or "hms client". And this objects will be re-initialized lazily.
And all properties will be got from Resource at runtime, so that it will always get the latest properties
Regression test cases
Because we add tinyInt1isBit=false to jdbc url, some of cases need to be changed.
This commit is contained in:
Mingyu Chen
@ -99,12 +99,16 @@ CREATE RESOURCE hms_resource PROPERTIES (
|
||||
'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007',
|
||||
'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'
|
||||
);
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource;
|
||||
|
||||
// The properties in 'PROPERTIES' will overwrite the properties in "Resource"
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource PROPERTIES(
|
||||
'key' = 'value'
|
||||
);
|
||||
```
|
||||
|
||||
**Create catalog through properties**
|
||||
|
||||
Version `1.2.0` creates a catalog through properties. This method will be deprecated in subsequent versions.
|
||||
Version `1.2.0` creates a catalog through properties.
|
||||
```sql
|
||||
CREATE CATALOG hive PROPERTIES (
|
||||
'type'='hms',
|
||||
|
||||
@ -39,14 +39,14 @@ Syntax:
|
||||
```sql
|
||||
CREATE CATALOG [IF NOT EXISTS] catalog_name
|
||||
[WITH RESOURCE resource_name]
|
||||
| [PROPERTIES ("key"="value", ...)];
|
||||
[PROPERTIES ("key"="value", ...)];
|
||||
```
|
||||
|
||||
`RESOURCE` can be created from [CREATE RESOURCE](../../../sql-reference/Data-Definition-Statements/Create/CREATE-RESOURCE.md), current supports：
|
||||
|
||||
* hms：Hive MetaStore
|
||||
* es：Elasticsearch
|
||||
* jdbc：数据库访问的标准接口(JDBC), 当前只支持`jdbc:mysql`
|
||||
* jdbc: Standard interface for database access (JDBC), currently supports MySQL and PostgreSQL
|
||||
|
||||
### Create catalog
|
||||
|
||||
@ -58,12 +58,14 @@ CREATE RESOURCE catalog_resource PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
...
|
||||
);
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource;
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource PROPERTIES (
|
||||
'key' = 'value'
|
||||
);
|
||||
```
|
||||
|
||||
**Create catalog through properties**
|
||||
|
||||
Version `1.2.0` creates a catalog through properties. This method will be deprecated in subsequent versions.
|
||||
Version `1.2.0` creates a catalog through properties.
|
||||
```sql
|
||||
CREATE CATALOG catalog_name PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
|
||||
@ -99,12 +99,16 @@ CREATE RESOURCE hms_resource PROPERTIES (
|
||||
'dfs.namenode.rpc-address.your-nameservice.nn2'='172.21.0.3:4007',
|
||||
'dfs.client.failover.proxy.provider.your-nameservice'='org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider'
|
||||
);
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource;
|
||||
|
||||
// 在 PROERPTIES 中指定的配置，将会覆盖 Resource 中的配置。
|
||||
CREATE CATALOG hive WITH RESOURCE hms_resource PROPERTIES(
|
||||
'key' = 'value'
|
||||
);
|
||||
```
|
||||
|
||||
**通过 properties 创建 catalog**
|
||||
|
||||
`1.2.0` 版本通过 properties 创建 catalog，该方法将在后续版本弃用。
|
||||
`1.2.0` 版本通过 properties 创建 catalog
|
||||
```sql
|
||||
CREATE CATALOG hive PROPERTIES (
|
||||
'type'='hms',
|
||||
|
||||
@ -43,14 +43,14 @@ CREATE CATALOG
|
||||
```sql
|
||||
CREATE CATALOG [IF NOT EXISTS] catalog_name
|
||||
[WITH RESOURCE resource_name]
|
||||
| [PROPERTIES ("key"="value", ...)];
|
||||
[PROPERTIES ("key"="value", ...)];
|
||||
```
|
||||
|
||||
`RESOURCE` 可以通过 [CREATE RESOURCE](../../../sql-reference/Data-Definition-Statements/Create/CREATE-RESOURCE.md) 创建，目前支持三种 Resource，分别连接三种外部数据源：
|
||||
|
||||
* hms：Hive MetaStore
|
||||
* es：Elasticsearch
|
||||
* jdbc：数据库访问的标准接口(JDBC), 当前只支持`jdbc:mysql`
|
||||
* jdbc：数据库访问的标准接口(JDBC), 当前支持 MySQL 和 PostgreSQL
|
||||
|
||||
### 创建 catalog
|
||||
|
||||
@ -62,12 +62,16 @@ CREATE RESOURCE catalog_resource PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
...
|
||||
);
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource;
|
||||
|
||||
// 在 PROERPTIES 中指定的配置，将会覆盖 Resource 中的配置。
|
||||
CREATE CATALOG catalog_name WITH RESOURCE catalog_resource PROPERTIES(
|
||||
'key' = 'value'
|
||||
)
|
||||
```
|
||||
|
||||
**通过 properties 创建 catalog**
|
||||
|
||||
`1.2.0` 版本通过 properties 创建 catalog，该方法将在后续版本弃用。
|
||||
`1.2.0` 版本通过 properties 创建 catalog。
|
||||
```sql
|
||||
CREATE CATALOG catalog_name PROPERTIES (
|
||||
'type'='hms|es|jdbc',
|
||||
|
||||
@ -1740,11 +1740,11 @@ create_stmt ::=
|
||||
/* Catalog */
|
||||
| KW_CREATE KW_CATALOG opt_if_not_exists:ifNotExists ident:catalogName opt_properties:properties
|
||||
{:
|
||||
RESULT = new CreateCatalogStmt(ifNotExists, catalogName, properties);
|
||||
RESULT = new CreateCatalogStmt(ifNotExists, catalogName, null, properties);
|
||||
:}
|
||||
| KW_CREATE KW_CATALOG opt_if_not_exists:ifNotExists ident:catalogName KW_WITH KW_RESOURCE ident:resourceName
|
||||
| KW_CREATE KW_CATALOG opt_if_not_exists:ifNotExists ident:catalogName KW_WITH KW_RESOURCE ident:resourceName opt_properties:properties
|
||||
{:
|
||||
RESULT = new CreateCatalogStmt(ifNotExists, catalogName, resourceName);
|
||||
RESULT = new CreateCatalogStmt(ifNotExists, catalogName, resourceName, properties);
|
||||
:}
|
||||
/* cluster */
|
||||
/* KW_CREATE KW_CLUSTER ident:name opt_properties:properties KW_IDENTIFIED KW_BY STRING_LITERAL:password
|
||||
|
||||
@ -29,7 +29,9 @@ import org.apache.doris.datasource.InternalCatalog;
|
||||
import org.apache.doris.mysql.privilege.PrivPredicate;
|
||||
import org.apache.doris.qe.ConnectContext;
|
||||
|
||||
import java.util.HashMap;
|
||||
import com.google.common.base.Strings;
|
||||
import com.google.common.collect.Maps;
|
||||
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
@ -44,18 +46,11 @@ public class CreateCatalogStmt extends DdlStmt {
|
||||
/**
|
||||
* Statement for create a new catalog.
|
||||
*/
|
||||
public CreateCatalogStmt(boolean ifNotExists, String catalogName, Map<String, String> properties) {
|
||||
public CreateCatalogStmt(boolean ifNotExists, String catalogName, String resource, Map<String, String> properties) {
|
||||
this.ifNotExists = ifNotExists;
|
||||
this.catalogName = catalogName;
|
||||
this.resource = null;
|
||||
this.properties = properties == null ? new HashMap<>() : properties;
|
||||
}
|
||||
|
||||
public CreateCatalogStmt(boolean ifNotExists, String catalogName, String resource) {
|
||||
this.ifNotExists = ifNotExists;
|
||||
this.catalogName = catalogName;
|
||||
this.resource = resource;
|
||||
this.properties = new HashMap<>();
|
||||
this.resource = resource == null ? "" : resource;
|
||||
this.properties = properties == null ? Maps.newHashMap() : properties;
|
||||
}
|
||||
|
||||
public String getCatalogName() {
|
||||
@ -99,12 +94,13 @@ public class CreateCatalogStmt extends DdlStmt {
|
||||
public String toSql() {
|
||||
StringBuilder stringBuilder = new StringBuilder();
|
||||
stringBuilder.append("CREATE CATALOG ").append("`").append(catalogName).append("`");
|
||||
if (!Strings.isNullOrEmpty(resource)) {
|
||||
stringBuilder.append(" WITH RESOURCE `").append(resource).append("`");
|
||||
}
|
||||
if (properties.size() > 0) {
|
||||
stringBuilder.append("\nPROPERTIES (\n");
|
||||
stringBuilder.append(new PrintableMap<>(properties, "=", true, true, false));
|
||||
stringBuilder.append("\n)");
|
||||
} else if (resource != null) {
|
||||
stringBuilder.append(" WITH RESOURCE `").append(resource).append("`");
|
||||
}
|
||||
return stringBuilder.toString();
|
||||
}
|
||||
|
||||
@ -36,7 +36,7 @@ public class ShowCreateCatalogStmt extends ShowStmt {
|
||||
private static final ShowResultSetMetaData META_DATA =
|
||||
ShowResultSetMetaData.builder()
|
||||
.addColumn(new Column("Catalog", ScalarType.createVarchar(20)))
|
||||
.addColumn(new Column("Create Catalog", ScalarType.createVarchar(30)))
|
||||
.addColumn(new Column("CreateCatalog", ScalarType.createVarchar(30)))
|
||||
.build();
|
||||
|
||||
private String catalog;
|
||||
|
||||
@ -95,8 +95,8 @@ public class HMSResource extends Resource {
|
||||
LOG.debug("Get properties from hive-site.xml");
|
||||
}
|
||||
Map<String, String> res = Maps.newHashMap();
|
||||
HiveConf hiveConf = new HiveConf();
|
||||
// read properties from hive-site.xml.
|
||||
HiveConf hiveConf = new HiveConf();
|
||||
String metastoreType = hiveConf.get(HIVE_METASTORE_TYPE);
|
||||
if (!"dlf".equalsIgnoreCase(metastoreType)) {
|
||||
return res;
|
||||
|
||||
@ -24,6 +24,7 @@ import org.apache.doris.common.DdlException;
|
||||
import org.apache.doris.common.FeConstants;
|
||||
import org.apache.doris.common.proc.BaseProcResult;
|
||||
import org.apache.doris.common.util.Util;
|
||||
import org.apache.doris.external.jdbc.JdbcClientException;
|
||||
|
||||
import com.google.common.base.Preconditions;
|
||||
import com.google.common.collect.Lists;
|
||||
@ -60,6 +61,11 @@ import java.util.Map;
|
||||
public class JdbcResource extends Resource {
|
||||
private static final Logger LOG = LogManager.getLogger(JdbcResource.class);
|
||||
|
||||
public static final String MYSQL = "MYSQL";
|
||||
public static final String POSTGRESQL = "POSTGRESQL";
|
||||
// private static final String ORACLE = "ORACLE";
|
||||
// private static final String SQLSERVER = "SQLSERVER";
|
||||
|
||||
public static final String JDBC_PROPERTIES_PREFIX = "jdbc.";
|
||||
public static final String JDBC_URL = "jdbc_url";
|
||||
public static final String USER = "user";
|
||||
@ -108,6 +114,7 @@ public class JdbcResource extends Resource {
|
||||
replaceIfEffectiveValue(this.configs, USER, properties.get(USER));
|
||||
replaceIfEffectiveValue(this.configs, PASSWORD, properties.get(PASSWORD));
|
||||
replaceIfEffectiveValue(this.configs, TYPE, properties.get(TYPE));
|
||||
this.configs.put(JDBC_URL, handleJdbcUrl(getProperty(JDBC_URL)));
|
||||
super.modifyProperties(properties);
|
||||
}
|
||||
|
||||
@ -142,7 +149,8 @@ public class JdbcResource extends Resource {
|
||||
checkProperties(USER);
|
||||
checkProperties(PASSWORD);
|
||||
checkProperties(TYPE);
|
||||
computeObjectChecksum();
|
||||
this.configs.put(JDBC_URL, handleJdbcUrl(getProperty(JDBC_URL)));
|
||||
configs.put(CHECK_SUM, computeObjectChecksum(getProperty(DRIVER_URL)));
|
||||
}
|
||||
|
||||
@Override
|
||||
@ -166,19 +174,18 @@ public class JdbcResource extends Resource {
|
||||
|
||||
public String getProperty(String propertiesKey) {
|
||||
// check the properties key
|
||||
String value = configs.get(propertiesKey);
|
||||
return value;
|
||||
return configs.get(propertiesKey);
|
||||
}
|
||||
|
||||
private void computeObjectChecksum() throws DdlException {
|
||||
public static String computeObjectChecksum(String driverPath) throws DdlException {
|
||||
if (FeConstants.runningUnitTest) {
|
||||
// skip checking checksum when running ut
|
||||
return;
|
||||
return "";
|
||||
}
|
||||
String realDriverPath = getRealDriverPath();
|
||||
String fullDriverPath = getRealDriverPath(driverPath);
|
||||
InputStream inputStream = null;
|
||||
try {
|
||||
inputStream = Util.getInputStreamFromUrl(realDriverPath, null, HTTP_TIMEOUT_MS, HTTP_TIMEOUT_MS);
|
||||
inputStream = Util.getInputStreamFromUrl(fullDriverPath, null, HTTP_TIMEOUT_MS, HTTP_TIMEOUT_MS);
|
||||
MessageDigest digest = MessageDigest.getInstance("MD5");
|
||||
byte[] buf = new byte[4096];
|
||||
int bytesRead = 0;
|
||||
@ -189,29 +196,77 @@ public class JdbcResource extends Resource {
|
||||
}
|
||||
digest.update(buf, 0, bytesRead);
|
||||
} while (true);
|
||||
String checkSum = Hex.encodeHexString(digest.digest());
|
||||
configs.put(CHECK_SUM, checkSum);
|
||||
return Hex.encodeHexString(digest.digest());
|
||||
} catch (IOException e) {
|
||||
throw new DdlException("compute driver checksum from url: " + getProperty(DRIVER_URL)
|
||||
throw new DdlException("compute driver checksum from url: " + driverPath
|
||||
+ " meet an IOException: " + e.getMessage());
|
||||
} catch (NoSuchAlgorithmException e) {
|
||||
throw new DdlException("compute driver checksum from url: " + getProperty(DRIVER_URL)
|
||||
throw new DdlException("compute driver checksum from url: " + driverPath
|
||||
+ " could not find algorithm: " + e.getMessage());
|
||||
}
|
||||
}
|
||||
|
||||
private String getRealDriverPath() {
|
||||
String path = getProperty(DRIVER_URL);
|
||||
private static String getRealDriverPath(String driverUrl) {
|
||||
try {
|
||||
URI uri = new URI(path);
|
||||
URI uri = new URI(driverUrl);
|
||||
String schema = uri.getScheme();
|
||||
if (schema == null && !path.startsWith("/")) {
|
||||
return "file://" + Config.jdbc_drivers_dir + "/" + path;
|
||||
if (schema == null && !driverUrl.startsWith("/")) {
|
||||
return "file://" + Config.jdbc_drivers_dir + "/" + driverUrl;
|
||||
}
|
||||
return path;
|
||||
return driverUrl;
|
||||
} catch (URISyntaxException e) {
|
||||
LOG.warn("invalid jdbc driver url: " + path);
|
||||
return path;
|
||||
LOG.warn("invalid jdbc driver url: " + driverUrl);
|
||||
return driverUrl;
|
||||
}
|
||||
}
|
||||
|
||||
public static String parseDbType(String url) {
|
||||
if (url.startsWith("jdbc:mysql") || url.startsWith("jdbc:mariadb")) {
|
||||
return MYSQL;
|
||||
} else if (url.startsWith("jdbc:postgresql")) {
|
||||
return POSTGRESQL;
|
||||
}
|
||||
// else if (url.startsWith("jdbc:oracle")) {
|
||||
// return ORACLE;
|
||||
// }
|
||||
// else if (url.startsWith("jdbc:sqlserver")) {
|
||||
// return SQLSERVER;
|
||||
throw new JdbcClientException("Unsupported jdbc database type, please check jdbcUrl: " + url);
|
||||
}
|
||||
|
||||
public static String handleJdbcUrl(String jdbcUrl) {
|
||||
// delete all space in jdbcUrl
|
||||
String newJdbcUrl = jdbcUrl.replaceAll(" ", "");
|
||||
String dbType = parseDbType(newJdbcUrl);
|
||||
if (dbType.equals(MYSQL)) {
|
||||
// 1. `yearIsDateType` is a parameter of JDBC, and the default is true.
|
||||
// We force the use of `yearIsDateType=false`
|
||||
// 2. `tinyInt1isBit` is a parameter of JDBC, and the default is true.
|
||||
// We force the use of `tinyInt1isBit=false`, so that for mysql type tinyint,
|
||||
// it will convert to Doris tinyint, not bit.
|
||||
newJdbcUrl = checkJdbcUrlParam(newJdbcUrl, "yearIsDateType", "true", "false");
|
||||
newJdbcUrl = checkJdbcUrlParam(newJdbcUrl, "tinyInt1isBit", "true", "false");
|
||||
}
|
||||
return newJdbcUrl;
|
||||
}
|
||||
|
||||
private static String checkJdbcUrlParam(String jdbcUrl, String params, String unexpectedVal, String expectedVal) {
|
||||
String unexpectedParams = params + "=" + unexpectedVal;
|
||||
String expectedParams = params + "=" + expectedVal;
|
||||
if (jdbcUrl.contains(expectedParams)) {
|
||||
return jdbcUrl;
|
||||
} else if (jdbcUrl.contains(unexpectedParams)) {
|
||||
jdbcUrl = jdbcUrl.replaceAll(unexpectedParams, expectedParams);
|
||||
} else {
|
||||
if (jdbcUrl.contains("?")) {
|
||||
if (jdbcUrl.charAt(jdbcUrl.length() - 1) != '?') {
|
||||
jdbcUrl += "&";
|
||||
}
|
||||
} else {
|
||||
jdbcUrl += "?";
|
||||
}
|
||||
jdbcUrl += expectedParams;
|
||||
}
|
||||
return jdbcUrl;
|
||||
}
|
||||
}
|
||||
|
||||
@ -84,10 +84,10 @@ public class EsExternalTable extends ExternalTable {
|
||||
esTable.setClient(esCatalog.getEsRestClient());
|
||||
esTable.setUserName(esCatalog.getUsername());
|
||||
esTable.setPasswd(esCatalog.getPassword());
|
||||
esTable.setEnableDocValueScan(esCatalog.isEnableDocValueScan());
|
||||
esTable.setEnableKeywordSniff(esCatalog.isEnableKeywordSniff());
|
||||
esTable.setNodesDiscovery(esCatalog.isEnableNodesDiscovery());
|
||||
esTable.setHttpSslEnabled(esCatalog.isEnableSsl());
|
||||
esTable.setEnableDocValueScan(esCatalog.enableDocValueScan());
|
||||
esTable.setEnableKeywordSniff(esCatalog.enableKeywordSniff());
|
||||
esTable.setNodesDiscovery(esCatalog.enableNodesDiscovery());
|
||||
esTable.setHttpSslEnabled(esCatalog.enableSsl());
|
||||
esTable.setSeeds(esCatalog.getNodes());
|
||||
esTable.setHosts(String.join(",", esCatalog.getNodes()));
|
||||
esTable.syncTableMetaData();
|
||||
|
||||
@ -27,6 +27,8 @@ import org.apache.doris.catalog.Env;
|
||||
import org.apache.doris.catalog.Resource;
|
||||
import org.apache.doris.common.DdlException;
|
||||
|
||||
import org.apache.parquet.Strings;
|
||||
|
||||
import java.util.Map;
|
||||
import java.util.Optional;
|
||||
|
||||
@ -71,44 +73,37 @@ public class CatalogFactory {
|
||||
|
||||
private static CatalogIf constructorCatalog(
|
||||
long catalogId, String name, String resource, Map<String, String> props) throws DdlException {
|
||||
CatalogIf catalog;
|
||||
if (resource == null) {
|
||||
String type = props.get("type");
|
||||
if (type == null) {
|
||||
throw new DdlException("Need catalog type in properties");
|
||||
}
|
||||
switch (type) {
|
||||
case "hms":
|
||||
catalog = new HMSExternalCatalog(catalogId, name, null, props);
|
||||
break;
|
||||
case "es":
|
||||
catalog = new EsExternalCatalog(catalogId, name, null, props);
|
||||
break;
|
||||
case "jdbc":
|
||||
catalog = new JdbcExternalCatalog(catalogId, name, null, props);
|
||||
break;
|
||||
default:
|
||||
throw new DdlException("Unknown catalog type: " + type);
|
||||
}
|
||||
} else if (props.size() == 0) {
|
||||
// get catalog type from resource or properties
|
||||
String catalogType;
|
||||
if (!Strings.isNullOrEmpty(resource)) {
|
||||
Resource catalogResource = Optional.ofNullable(Env.getCurrentEnv().getResourceMgr().getResource(resource))
|
||||
.orElseThrow(() -> new DdlException("Resource doesn't exist: " + resource));
|
||||
Resource.ResourceType type = catalogResource.getType();
|
||||
switch (type) {
|
||||
case HMS:
|
||||
catalog = new HMSExternalCatalog(catalogId, name, resource, props);
|
||||
break;
|
||||
case ES:
|
||||
catalog = new EsExternalCatalog(catalogId, name, resource, props);
|
||||
break;
|
||||
case JDBC:
|
||||
catalog = new JdbcExternalCatalog(catalogId, name, resource, props);
|
||||
break;
|
||||
default:
|
||||
throw new DdlException("Unknown catalog type with resource: " + resource + ", type: " + type);
|
||||
catalogType = catalogResource.getType().name().toLowerCase();
|
||||
if (props.containsKey("type")) {
|
||||
throw new DdlException("Can not set 'type' when creating catalog with resource");
|
||||
}
|
||||
} else {
|
||||
throw new DdlException("Can't provide resource and properties for catalog simultaneously");
|
||||
String type = props.get("type");
|
||||
if (Strings.isNullOrEmpty(type)) {
|
||||
throw new DdlException("Missing property 'type' in properties");
|
||||
}
|
||||
catalogType = type.toLowerCase();
|
||||
}
|
||||
|
||||
// create catalog
|
||||
CatalogIf catalog;
|
||||
switch (catalogType) {
|
||||
case "hms":
|
||||
catalog = new HMSExternalCatalog(catalogId, name, resource, props);
|
||||
break;
|
||||
case "es":
|
||||
catalog = new EsExternalCatalog(catalogId, name, resource, props);
|
||||
break;
|
||||
case "jdbc":
|
||||
catalog = new JdbcExternalCatalog(catalogId, name, resource, props);
|
||||
break;
|
||||
default:
|
||||
throw new DdlException("Unknown catalog type: " + catalogType);
|
||||
}
|
||||
return catalog;
|
||||
}
|
||||
|
||||
@ -72,7 +72,11 @@ public interface CatalogIf<T extends DatabaseIf> {
|
||||
return null;
|
||||
}
|
||||
|
||||
default void notifyPropertiesUpdated() { }
|
||||
default void notifyPropertiesUpdated() {
|
||||
if (this instanceof ExternalCatalog) {
|
||||
((ExternalCatalog) this).setUninitialized(false);
|
||||
}
|
||||
}
|
||||
|
||||
void modifyCatalogName(String name);
|
||||
|
||||
|
||||
@ -55,6 +55,7 @@ import com.google.common.collect.Maps;
|
||||
import com.google.gson.annotations.SerializedName;
|
||||
import org.apache.logging.log4j.LogManager;
|
||||
import org.apache.logging.log4j.Logger;
|
||||
import org.apache.parquet.Strings;
|
||||
|
||||
import java.io.DataInput;
|
||||
import java.io.DataOutput;
|
||||
@ -107,7 +108,7 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
|
||||
private void addCatalog(CatalogIf catalog) {
|
||||
nameToCatalog.put(catalog.getName(), catalog);
|
||||
idToCatalog.put(catalog.getId(), catalog);
|
||||
if (catalog.getResource() != null) {
|
||||
if (!Strings.isNullOrEmpty(catalog.getResource())) {
|
||||
Env.getCurrentEnv().getResourceMgr().getResource(catalog.getResource())
|
||||
.addReference(catalog.getName(), ReferenceType.CATALOG);
|
||||
}
|
||||
@ -120,7 +121,7 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
|
||||
nameToCatalog.remove(catalog.getName());
|
||||
lastDBOfCatalog.remove(catalog.getName());
|
||||
Env.getCurrentEnv().getExtMetaCacheMgr().removeCache(catalog.getName());
|
||||
if (catalog.getResource() != null) {
|
||||
if (!Strings.isNullOrEmpty(catalog.getResource())) {
|
||||
Resource catalogResource = Env.getCurrentEnv().getResourceMgr().getResource(catalog.getResource());
|
||||
if (catalogResource != null) {
|
||||
catalogResource.removeReference(catalog.getName(), ReferenceType.CATALOG);
|
||||
@ -313,14 +314,6 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
|
||||
if (catalog == null) {
|
||||
throw new DdlException("No catalog found with name: " + stmt.getCatalogName());
|
||||
}
|
||||
if (catalog instanceof ExternalCatalog) {
|
||||
String resource = ((ExternalCatalog) catalog).getCatalogProperty().getResource();
|
||||
if (resource != null) {
|
||||
throw new DdlException(String.format(
|
||||
"Catalog %s has %s resource, please change the resource properties directly.",
|
||||
stmt.getCatalogName(), resource));
|
||||
}
|
||||
}
|
||||
if (stmt.getNewProperties().containsKey("type") && !catalog.getType()
|
||||
.equalsIgnoreCase(stmt.getNewProperties().get("type"))) {
|
||||
throw new DdlException("Can't modify the type of catalog property with name: " + stmt.getCatalogName());
|
||||
@ -393,7 +386,7 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
|
||||
ErrorReport.reportAnalysisException(ErrorCode.ERR_CATALOG_ACCESS_DENIED,
|
||||
ConnectContext.get().getQualifiedUser(), catalog.getName());
|
||||
}
|
||||
if (catalog.getResource() != null) {
|
||||
if (!Strings.isNullOrEmpty(catalog.getResource())) {
|
||||
rows.add(Arrays.asList("resource", catalog.getResource()));
|
||||
}
|
||||
for (Map.Entry<String, String> elem : catalog.getProperties().entrySet()) {
|
||||
|
||||
@ -24,6 +24,8 @@ import org.apache.doris.common.io.Text;
|
||||
import org.apache.doris.common.io.Writable;
|
||||
import org.apache.doris.persist.gson.GsonUtils;
|
||||
|
||||
import com.google.common.base.Strings;
|
||||
import com.google.common.collect.Maps;
|
||||
import com.google.gson.annotations.SerializedName;
|
||||
import lombok.Data;
|
||||
import org.apache.logging.log4j.LogManager;
|
||||
@ -32,11 +34,11 @@ import org.apache.logging.log4j.Logger;
|
||||
import java.io.DataInput;
|
||||
import java.io.DataOutput;
|
||||
import java.io.IOException;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* CatalogProperty to store the properties for catalog.
|
||||
* the properties in "properties" will overwrite properties in "resource"
|
||||
*/
|
||||
@Data
|
||||
public class CatalogProperty implements Writable {
|
||||
@ -50,8 +52,11 @@ public class CatalogProperty implements Writable {
|
||||
private volatile Resource catalogResource = null;
|
||||
|
||||
public CatalogProperty(String resource, Map<String, String> properties) {
|
||||
this.resource = resource;
|
||||
this.resource = Strings.nullToEmpty(resource);
|
||||
this.properties = properties;
|
||||
if (this.properties == null) {
|
||||
this.properties = Maps.newConcurrentMap();
|
||||
}
|
||||
}
|
||||
|
||||
private Resource catalogResource() {
|
||||
@ -66,35 +71,33 @@ public class CatalogProperty implements Writable {
|
||||
}
|
||||
|
||||
public String getOrDefault(String key, String defaultVal) {
|
||||
if (resource == null) {
|
||||
return properties.getOrDefault(key, defaultVal);
|
||||
} else {
|
||||
return catalogResource().getCopiedProperties().getOrDefault(key, defaultVal);
|
||||
String val = properties.get(key);
|
||||
if (val == null) {
|
||||
Resource res = catalogResource();
|
||||
if (res != null) {
|
||||
val = res.getCopiedProperties().getOrDefault(key, defaultVal);
|
||||
} else {
|
||||
val = defaultVal;
|
||||
}
|
||||
}
|
||||
return val;
|
||||
}
|
||||
|
||||
public Map<String, String> getProperties() {
|
||||
if (resource == null) {
|
||||
return new HashMap<>(properties);
|
||||
} else {
|
||||
return catalogResource().getCopiedProperties();
|
||||
Map<String, String> mergedProperties = Maps.newHashMap();
|
||||
if (!Strings.isNullOrEmpty(resource)) {
|
||||
mergedProperties = catalogResource().getCopiedProperties();
|
||||
}
|
||||
mergedProperties.putAll(properties);
|
||||
return mergedProperties;
|
||||
}
|
||||
|
||||
public void modifyCatalogProps(Map<String, String> props) {
|
||||
if (resource == null) {
|
||||
properties.putAll(props);
|
||||
} else {
|
||||
LOG.error("Please change the resource {} properties directly", resource);
|
||||
}
|
||||
properties.putAll(props);
|
||||
}
|
||||
|
||||
public Map<String, String> getS3HadoopProperties() {
|
||||
if (resource == null) {
|
||||
return S3Resource.getS3HadoopProperties(properties);
|
||||
} else {
|
||||
return S3Resource.getS3HadoopProperties(catalogResource().getCopiedProperties());
|
||||
}
|
||||
return S3Resource.getS3HadoopProperties(getProperties());
|
||||
}
|
||||
|
||||
public Map<String, String> getHadoopProperties() {
|
||||
|
||||
@ -22,7 +22,6 @@ import org.apache.doris.catalog.Column;
|
||||
import org.apache.doris.catalog.Env;
|
||||
import org.apache.doris.catalog.EsResource;
|
||||
import org.apache.doris.catalog.external.EsExternalDatabase;
|
||||
import org.apache.doris.common.DdlException;
|
||||
import org.apache.doris.external.elasticsearch.EsRestClient;
|
||||
import org.apache.doris.external.elasticsearch.EsUtil;
|
||||
|
||||
@ -33,9 +32,7 @@ import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.logging.log4j.LogManager;
|
||||
import org.apache.logging.log4j.Logger;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@ -50,108 +47,86 @@ public class EsExternalCatalog extends ExternalCatalog {
|
||||
|
||||
private EsRestClient esRestClient;
|
||||
|
||||
private String[] nodes;
|
||||
|
||||
private String username = null;
|
||||
|
||||
private String password = null;
|
||||
|
||||
private boolean enableDocValueScan = true;
|
||||
|
||||
private boolean enableKeywordSniff = true;
|
||||
|
||||
private boolean enableSsl = false;
|
||||
|
||||
private boolean enableNodesDiscovery = true;
|
||||
|
||||
/**
|
||||
* Default constructor for EsExternalCatalog.
|
||||
*/
|
||||
public EsExternalCatalog(
|
||||
long catalogId, String name, String resource, Map<String, String> props) throws DdlException {
|
||||
this.id = catalogId;
|
||||
this.name = name;
|
||||
public EsExternalCatalog(long catalogId, String name, String resource, Map<String, String> props) {
|
||||
super(catalogId, name);
|
||||
this.type = "es";
|
||||
if (resource == null) {
|
||||
catalogProperty = new CatalogProperty(null, processCompatibleProperties(props));
|
||||
} else {
|
||||
catalogProperty = new CatalogProperty(resource, Collections.emptyMap());
|
||||
processCompatibleProperties(catalogProperty.getProperties());
|
||||
}
|
||||
this.catalogProperty = new CatalogProperty(resource, processCompatibleProperties(props));
|
||||
}
|
||||
|
||||
private Map<String, String> processCompatibleProperties(Map<String, String> props) throws DdlException {
|
||||
private Map<String, String> processCompatibleProperties(Map<String, String> props) {
|
||||
// Compatible with "Doris On ES" interfaces
|
||||
Map<String, String> properties = Maps.newHashMap();
|
||||
for (Map.Entry<String, String> kv : props.entrySet()) {
|
||||
properties.put(StringUtils.removeStart(kv.getKey(), EsResource.ES_PROPERTIES_PREFIX), kv.getValue());
|
||||
}
|
||||
nodes = properties.get(EsResource.HOSTS).trim().split(",");
|
||||
// nodes = properties.get(EsResource.HOSTS).trim().split(",");
|
||||
if (properties.containsKey("ssl")) {
|
||||
properties.put(EsResource.HTTP_SSL_ENABLED, properties.remove("ssl"));
|
||||
}
|
||||
if (properties.containsKey(EsResource.HTTP_SSL_ENABLED)) {
|
||||
enableSsl = EsUtil.getBoolean(properties, EsResource.HTTP_SSL_ENABLED);
|
||||
} else {
|
||||
properties.put(EsResource.HTTP_SSL_ENABLED, String.valueOf(enableSsl));
|
||||
if (!properties.containsKey(EsResource.HTTP_SSL_ENABLED)) {
|
||||
properties.put(EsResource.HTTP_SSL_ENABLED, String.valueOf(false));
|
||||
}
|
||||
|
||||
if (properties.containsKey("username")) {
|
||||
properties.put(EsResource.USER, properties.remove("username"));
|
||||
}
|
||||
if (StringUtils.isNotBlank(properties.get(EsResource.USER))) {
|
||||
username = properties.get(EsResource.USER).trim();
|
||||
}
|
||||
|
||||
if (StringUtils.isNotBlank(properties.get(EsResource.PASSWORD))) {
|
||||
password = properties.get(EsResource.PASSWORD).trim();
|
||||
}
|
||||
|
||||
if (properties.containsKey("doc_value_scan")) {
|
||||
properties.put(EsResource.DOC_VALUE_SCAN, properties.remove("doc_value_scan"));
|
||||
}
|
||||
if (properties.containsKey(EsResource.DOC_VALUE_SCAN)) {
|
||||
enableDocValueScan = EsUtil.getBoolean(properties, EsResource.DOC_VALUE_SCAN);
|
||||
} else {
|
||||
properties.put(EsResource.DOC_VALUE_SCAN, String.valueOf(enableDocValueScan));
|
||||
if (!properties.containsKey(EsResource.DOC_VALUE_SCAN)) {
|
||||
properties.put(EsResource.DOC_VALUE_SCAN, "true");
|
||||
}
|
||||
|
||||
if (properties.containsKey("keyword_sniff")) {
|
||||
properties.put(EsResource.KEYWORD_SNIFF, properties.remove("keyword_sniff"));
|
||||
}
|
||||
if (properties.containsKey(EsResource.KEYWORD_SNIFF)) {
|
||||
enableKeywordSniff = EsUtil.getBoolean(properties, EsResource.KEYWORD_SNIFF);
|
||||
} else {
|
||||
properties.put(EsResource.KEYWORD_SNIFF, String.valueOf(enableKeywordSniff));
|
||||
if (!properties.containsKey(EsResource.KEYWORD_SNIFF)) {
|
||||
properties.put(EsResource.KEYWORD_SNIFF, "true");
|
||||
}
|
||||
|
||||
if (properties.containsKey(EsResource.NODES_DISCOVERY)) {
|
||||
enableNodesDiscovery = EsUtil.getBoolean(properties, EsResource.NODES_DISCOVERY);
|
||||
} else {
|
||||
properties.put(EsResource.NODES_DISCOVERY, String.valueOf(enableNodesDiscovery));
|
||||
if (!properties.containsKey(EsResource.NODES_DISCOVERY)) {
|
||||
properties.put(EsResource.NODES_DISCOVERY, "true");
|
||||
}
|
||||
return properties;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void notifyPropertiesUpdated() {
|
||||
try {
|
||||
processCompatibleProperties(catalogProperty.getProperties());
|
||||
initLocalObjectsImpl();
|
||||
} catch (DdlException e) {
|
||||
LOG.warn("Failed to notify properties updated to catalog {}", name, e);
|
||||
}
|
||||
public String[] getNodes() {
|
||||
String hosts = catalogProperty.getOrDefault(EsResource.HOSTS, "");
|
||||
return hosts.trim().split(",");
|
||||
}
|
||||
|
||||
public String getUsername() {
|
||||
return catalogProperty.getOrDefault(EsResource.USER, "");
|
||||
}
|
||||
|
||||
public String getPassword() {
|
||||
return catalogProperty.getOrDefault(EsResource.PASSWORD, "");
|
||||
}
|
||||
|
||||
public boolean enableDocValueScan() {
|
||||
return Boolean.valueOf(catalogProperty.getOrDefault(EsResource.DOC_VALUE_SCAN, "true"));
|
||||
}
|
||||
|
||||
public boolean enableKeywordSniff() {
|
||||
return Boolean.valueOf(catalogProperty.getOrDefault(EsResource.KEYWORD_SNIFF, "true"));
|
||||
}
|
||||
|
||||
public boolean enableSsl() {
|
||||
return Boolean.valueOf(catalogProperty.getOrDefault(EsResource.HTTP_SSL_ENABLED, "false"));
|
||||
}
|
||||
|
||||
public boolean enableNodesDiscovery() {
|
||||
return Boolean.valueOf(catalogProperty.getOrDefault(EsResource.NODES_DISCOVERY, "true"));
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void initLocalObjectsImpl() {
|
||||
esRestClient = new EsRestClient(this.nodes, this.username, this.password, this.enableSsl);
|
||||
esRestClient = new EsRestClient(getNodes(), getUsername(), getPassword(), enableSsl());
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void init() {
|
||||
InitCatalogLog initCatalogLog = new InitCatalogLog();
|
||||
this.esRestClient = new EsRestClient(this.nodes, this.username, this.password, this.enableSsl);
|
||||
initCatalogLog.setCatalogId(id);
|
||||
initCatalogLog.setType(InitCatalogLog.Type.ES);
|
||||
if (dbNameToId != null && dbNameToId.containsKey(DEFAULT_DB)) {
|
||||
@ -193,20 +168,6 @@ public class EsExternalCatalog extends ExternalCatalog {
|
||||
return esRestClient.existIndex(this.esRestClient.getClient(), tblName);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void gsonPostProcess() throws IOException {
|
||||
super.gsonPostProcess();
|
||||
try {
|
||||
if (catalogProperty.getResource() == null) {
|
||||
catalogProperty.setProperties(processCompatibleProperties(catalogProperty.getProperties()));
|
||||
} else {
|
||||
processCompatibleProperties(catalogProperty.getProperties());
|
||||
}
|
||||
} catch (DdlException e) {
|
||||
throw new IOException(e);
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Column> getSchema(String dbName, String tblName) {
|
||||
makeSureInitialized();
|
||||
|
||||
@ -73,6 +73,11 @@ public abstract class ExternalCatalog implements CatalogIf<ExternalDatabase>, Wr
|
||||
|
||||
private ExternalSchemaCache schemaCache;
|
||||
|
||||
public ExternalCatalog(long catalogId, String name) {
|
||||
this.id = catalogId;
|
||||
this.name = name;
|
||||
}
|
||||
|
||||
/**
|
||||
* @return names of database in this catalog.
|
||||
*/
|
||||
@ -131,6 +136,7 @@ public abstract class ExternalCatalog implements CatalogIf<ExternalDatabase>, Wr
|
||||
protected abstract void init();
|
||||
|
||||
public void setUninitialized(boolean invalidCache) {
|
||||
this.objectCreated = false;
|
||||
this.initialized = false;
|
||||
this.invalidCacheInInit = invalidCache;
|
||||
if (invalidCache) {
|
||||
|
||||
@ -26,7 +26,6 @@ import org.apache.doris.catalog.HiveMetaStoreClientHelper;
|
||||
import org.apache.doris.catalog.external.ExternalDatabase;
|
||||
import org.apache.doris.catalog.external.HMSExternalDatabase;
|
||||
import org.apache.doris.common.Config;
|
||||
import org.apache.doris.common.DdlException;
|
||||
import org.apache.doris.datasource.hive.event.MetastoreNotificationFetchException;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
@ -58,14 +57,10 @@ public class HMSExternalCatalog extends ExternalCatalog {
|
||||
/**
|
||||
* Default constructor for HMSExternalCatalog.
|
||||
*/
|
||||
public HMSExternalCatalog(
|
||||
long catalogId, String name, String resource, Map<String, String> props) throws DdlException {
|
||||
this.id = catalogId;
|
||||
this.name = name;
|
||||
public HMSExternalCatalog(long catalogId, String name, String resource, Map<String, String> props) {
|
||||
super(catalogId, name);
|
||||
this.type = "hms";
|
||||
if (resource == null) {
|
||||
props.putAll(HMSResource.getPropertiesFromDLF());
|
||||
}
|
||||
props.putAll(HMSResource.getPropertiesFromDLF());
|
||||
catalogProperty = new CatalogProperty(resource, props);
|
||||
}
|
||||
|
||||
@ -104,11 +99,6 @@ public class HMSExternalCatalog extends ExternalCatalog {
|
||||
Env.getCurrentEnv().getEditLog().logInitCatalog(initCatalogLog);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void notifyPropertiesUpdated() {
|
||||
initLocalObjectsImpl();
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void initLocalObjectsImpl() {
|
||||
HiveConf hiveConf = new HiveConf();
|
||||
|
||||
@ -25,6 +25,7 @@ import org.apache.doris.catalog.external.JdbcExternalDatabase;
|
||||
import org.apache.doris.common.DdlException;
|
||||
import org.apache.doris.external.jdbc.JdbcClient;
|
||||
|
||||
import com.google.common.base.Strings;
|
||||
import com.google.common.collect.Lists;
|
||||
import com.google.common.collect.Maps;
|
||||
import lombok.Getter;
|
||||
@ -32,8 +33,6 @@ import org.apache.commons.lang3.StringUtils;
|
||||
import org.apache.logging.log4j.LogManager;
|
||||
import org.apache.logging.log4j.Logger;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.util.Collections;
|
||||
import java.util.List;
|
||||
import java.util.Map;
|
||||
|
||||
@ -44,25 +43,12 @@ public class JdbcExternalCatalog extends ExternalCatalog {
|
||||
// Must add "transient" for Gson to ignore this field,
|
||||
// or Gson will throw exception with HikariCP
|
||||
private transient JdbcClient jdbcClient;
|
||||
private String databaseTypeName;
|
||||
private String jdbcUser;
|
||||
private String jdbcPasswd;
|
||||
private String jdbcUrl;
|
||||
private String driverUrl;
|
||||
private String driverClass;
|
||||
private String checkSum = "";
|
||||
|
||||
public JdbcExternalCatalog(
|
||||
long catalogId, String name, String resource, Map<String, String> props) throws DdlException {
|
||||
this.id = catalogId;
|
||||
this.name = name;
|
||||
public JdbcExternalCatalog(long catalogId, String name, String resource, Map<String, String> props)
|
||||
throws DdlException {
|
||||
super(catalogId, name);
|
||||
this.type = "jdbc";
|
||||
if (resource == null) {
|
||||
catalogProperty = new CatalogProperty(null, processCompatibleProperties(props));
|
||||
} else {
|
||||
catalogProperty = new CatalogProperty(resource, Collections.emptyMap());
|
||||
processCompatibleProperties(catalogProperty.getProperties());
|
||||
}
|
||||
this.catalogProperty = new CatalogProperty(resource, processCompatibleProperties(props));
|
||||
}
|
||||
|
||||
@Override
|
||||
@ -72,57 +58,55 @@ public class JdbcExternalCatalog extends ExternalCatalog {
|
||||
}
|
||||
}
|
||||
|
||||
private Map<String, String> processCompatibleProperties(Map<String, String> props) {
|
||||
private Map<String, String> processCompatibleProperties(Map<String, String> props) throws DdlException {
|
||||
Map<String, String> properties = Maps.newHashMap();
|
||||
for (Map.Entry<String, String> kv : props.entrySet()) {
|
||||
properties.put(StringUtils.removeStart(kv.getKey(), JdbcResource.JDBC_PROPERTIES_PREFIX), kv.getValue());
|
||||
}
|
||||
jdbcUser = properties.getOrDefault(JdbcResource.USER, "");
|
||||
jdbcPasswd = properties.getOrDefault(JdbcResource.PASSWORD, "");
|
||||
jdbcUrl = properties.getOrDefault(JdbcResource.JDBC_URL, "");
|
||||
handleJdbcUrl();
|
||||
properties.put(JdbcResource.JDBC_URL, jdbcUrl);
|
||||
driverUrl = properties.getOrDefault(JdbcResource.DRIVER_URL, "");
|
||||
driverClass = properties.getOrDefault(JdbcResource.DRIVER_CLASS, "");
|
||||
checkSum = properties.getOrDefault(JdbcResource.CHECK_SUM, "");
|
||||
String jdbcUrl = properties.getOrDefault(JdbcResource.JDBC_URL, "");
|
||||
if (!Strings.isNullOrEmpty(jdbcUrl)) {
|
||||
jdbcUrl = JdbcResource.handleJdbcUrl(jdbcUrl);
|
||||
properties.put(JdbcResource.JDBC_URL, jdbcUrl);
|
||||
}
|
||||
|
||||
if (properties.containsKey(JdbcResource.DRIVER_URL)) {
|
||||
properties.put(JdbcResource.CHECK_SUM,
|
||||
JdbcResource.computeObjectChecksum(properties.get(JdbcResource.DRIVER_URL)));
|
||||
}
|
||||
return properties;
|
||||
}
|
||||
|
||||
// `yearIsDateType` is a parameter of JDBC, and the default is `yearIsDateType=true`
|
||||
// We force the use of `yearIsDateType=false`
|
||||
private void handleJdbcUrl() {
|
||||
// delete all space in jdbcUrl
|
||||
jdbcUrl = jdbcUrl.replaceAll(" ", "");
|
||||
if (jdbcUrl.contains("yearIsDateType=false")) {
|
||||
return;
|
||||
} else if (jdbcUrl.contains("yearIsDateType=true")) {
|
||||
jdbcUrl = jdbcUrl.replaceAll("yearIsDateType=true", "yearIsDateType=false");
|
||||
} else {
|
||||
String yearIsDateType = "yearIsDateType=false";
|
||||
if (jdbcUrl.contains("?")) {
|
||||
if (jdbcUrl.charAt(jdbcUrl.length() - 1) != '?') {
|
||||
jdbcUrl += "&";
|
||||
}
|
||||
} else {
|
||||
jdbcUrl += "?";
|
||||
}
|
||||
jdbcUrl += yearIsDateType;
|
||||
}
|
||||
public String getDatabaseTypeName() {
|
||||
return JdbcResource.parseDbType(getJdbcUrl());
|
||||
}
|
||||
|
||||
@Override
|
||||
public void notifyPropertiesUpdated() {
|
||||
processCompatibleProperties(catalogProperty.getProperties());
|
||||
initLocalObjectsImpl();
|
||||
public String getJdbcUser() {
|
||||
return catalogProperty.getOrDefault(JdbcResource.USER, "");
|
||||
}
|
||||
|
||||
public String getJdbcPasswd() {
|
||||
return catalogProperty.getOrDefault(JdbcResource.PASSWORD, "");
|
||||
}
|
||||
|
||||
public String getJdbcUrl() {
|
||||
return catalogProperty.getOrDefault(JdbcResource.JDBC_URL, "");
|
||||
}
|
||||
|
||||
public String getDriverUrl() {
|
||||
return catalogProperty.getOrDefault(JdbcResource.DRIVER_URL, "");
|
||||
}
|
||||
|
||||
public String getDriverClass() {
|
||||
return catalogProperty.getOrDefault(JdbcResource.DRIVER_CLASS, "");
|
||||
}
|
||||
|
||||
public String getCheckSum() {
|
||||
return catalogProperty.getOrDefault(JdbcResource.CHECK_SUM, "");
|
||||
}
|
||||
|
||||
@Override
|
||||
protected void initLocalObjectsImpl() {
|
||||
jdbcClient = new JdbcClient(jdbcUser, jdbcPasswd, jdbcUrl, driverUrl, driverClass);
|
||||
databaseTypeName = jdbcClient.getDbType();
|
||||
if (checkSum.isEmpty()) {
|
||||
checkSum = jdbcClient.getCheckSum();
|
||||
}
|
||||
jdbcClient = new JdbcClient(getJdbcUser(), getJdbcPasswd(), getJdbcUrl(), getDriverUrl(), getDriverClass());
|
||||
}
|
||||
|
||||
@Override
|
||||
@ -180,16 +164,6 @@ public class JdbcExternalCatalog extends ExternalCatalog {
|
||||
return jdbcClient.isTableExist(dbName, tblName);
|
||||
}
|
||||
|
||||
@Override
|
||||
public void gsonPostProcess() throws IOException {
|
||||
super.gsonPostProcess();
|
||||
if (catalogProperty.getResource() == null) {
|
||||
catalogProperty.setProperties(processCompatibleProperties(catalogProperty.getProperties()));
|
||||
} else {
|
||||
processCompatibleProperties(catalogProperty.getProperties());
|
||||
}
|
||||
}
|
||||
|
||||
@Override
|
||||
public List<Column> getSchema(String dbName, String tblName) {
|
||||
makeSureInitialized();
|
||||
|
||||
@ -18,29 +18,23 @@
|
||||
package org.apache.doris.external.jdbc;
|
||||
|
||||
import org.apache.doris.catalog.Column;
|
||||
import org.apache.doris.catalog.JdbcResource;
|
||||
import org.apache.doris.catalog.PrimitiveType;
|
||||
import org.apache.doris.catalog.ScalarType;
|
||||
import org.apache.doris.catalog.Type;
|
||||
import org.apache.doris.common.Config;
|
||||
import org.apache.doris.common.FeConstants;
|
||||
import org.apache.doris.common.util.Util;
|
||||
|
||||
import com.google.common.collect.Lists;
|
||||
import com.zaxxer.hikari.HikariConfig;
|
||||
import com.zaxxer.hikari.HikariDataSource;
|
||||
import lombok.Data;
|
||||
import lombok.Getter;
|
||||
import org.apache.commons.codec.binary.Hex;
|
||||
import org.apache.logging.log4j.LogManager;
|
||||
import org.apache.logging.log4j.Logger;
|
||||
|
||||
import java.io.IOException;
|
||||
import java.io.InputStream;
|
||||
import java.net.MalformedURLException;
|
||||
import java.net.URL;
|
||||
import java.net.URLClassLoader;
|
||||
import java.security.MessageDigest;
|
||||
import java.security.NoSuchAlgorithmException;
|
||||
import java.sql.Connection;
|
||||
import java.sql.DatabaseMetaData;
|
||||
import java.sql.ResultSet;
|
||||
@ -51,10 +45,6 @@ import java.util.List;
|
||||
@Getter
|
||||
public class JdbcClient {
|
||||
private static final Logger LOG = LogManager.getLogger(JdbcClient.class);
|
||||
private static final String MYSQL = "MYSQL";
|
||||
private static final String POSTGRESQL = "POSTGRESQL";
|
||||
// private static final String ORACLE = "ORACLE";
|
||||
// private static final String SQLSERVER = "SQLSERVER";
|
||||
private static final int HTTP_TIMEOUT_MS = 10000;
|
||||
|
||||
private String dbType;
|
||||
@ -63,7 +53,6 @@ public class JdbcClient {
|
||||
private String jdbcUrl;
|
||||
private String driverUrl;
|
||||
private String driverClass;
|
||||
private String checkSum;
|
||||
|
||||
private URLClassLoader classLoader = null;
|
||||
|
||||
@ -76,8 +65,7 @@ public class JdbcClient {
|
||||
this.jdbcUrl = jdbcUrl;
|
||||
this.driverUrl = driverUrl;
|
||||
this.driverClass = driverClass;
|
||||
this.dbType = parseDbType(jdbcUrl);
|
||||
this.checkSum = computeObjectChecksum();
|
||||
this.dbType = JdbcResource.parseDbType(jdbcUrl);
|
||||
|
||||
ClassLoader oldClassLoader = Thread.currentThread().getContextClassLoader();
|
||||
try {
|
||||
@ -105,20 +93,6 @@ public class JdbcClient {
|
||||
dataSource.close();
|
||||
}
|
||||
|
||||
public String parseDbType(String url) {
|
||||
if (url.startsWith("jdbc:mysql") || url.startsWith("jdbc:mariadb")) {
|
||||
return MYSQL;
|
||||
} else if (url.startsWith("jdbc:postgresql")) {
|
||||
return POSTGRESQL;
|
||||
}
|
||||
// else if (url.startsWith("jdbc:oracle")) {
|
||||
// return ORACLE;
|
||||
// }
|
||||
// else if (url.startsWith("jdbc:sqlserver")) {
|
||||
// return SQLSERVER;
|
||||
throw new JdbcClientException("Unsupported jdbc database type, please check jdbcUrl: " + jdbcUrl);
|
||||
}
|
||||
|
||||
public Connection getConnection() throws JdbcClientException {
|
||||
Connection conn = null;
|
||||
try {
|
||||
@ -181,10 +155,10 @@ public class JdbcClient {
|
||||
try {
|
||||
stmt = conn.createStatement();
|
||||
switch (dbType) {
|
||||
case MYSQL:
|
||||
case JdbcResource.MYSQL:
|
||||
rs = stmt.executeQuery("SHOW DATABASES");
|
||||
break;
|
||||
case POSTGRESQL:
|
||||
case JdbcResource.POSTGRESQL:
|
||||
rs = stmt.executeQuery("SELECT schema_name FROM information_schema.schemata "
|
||||
+ "where schema_owner='" + jdbcUser + "';");
|
||||
break;
|
||||
@ -214,10 +188,10 @@ public class JdbcClient {
|
||||
try {
|
||||
DatabaseMetaData databaseMetaData = conn.getMetaData();
|
||||
switch (dbType) {
|
||||
case MYSQL:
|
||||
case JdbcResource.MYSQL:
|
||||
rs = databaseMetaData.getTables(dbName, null, null, types);
|
||||
break;
|
||||
case POSTGRESQL:
|
||||
case JdbcResource.POSTGRESQL:
|
||||
rs = databaseMetaData.getTables(null, dbName, null, types);
|
||||
break;
|
||||
default:
|
||||
@ -241,11 +215,14 @@ public class JdbcClient {
|
||||
try {
|
||||
DatabaseMetaData databaseMetaData = conn.getMetaData();
|
||||
switch (dbType) {
|
||||
case MYSQL:
|
||||
case JdbcResource.MYSQL:
|
||||
rs = databaseMetaData.getTables(dbName, null, tableName, types);
|
||||
break;
|
||||
case JdbcResource.POSTGRESQL:
|
||||
rs = databaseMetaData.getTables(null, dbName, null, types);
|
||||
break;
|
||||
default:
|
||||
throw new JdbcClientException("Unknown database type");
|
||||
throw new JdbcClientException("Unknown database type: " + dbType);
|
||||
}
|
||||
if (rs.next()) {
|
||||
return true;
|
||||
@ -306,10 +283,10 @@ public class JdbcClient {
|
||||
// columnNamePattern - column name, `null` means get all columns
|
||||
// Can contain single-character wildcards ("_"), or multi-character wildcards ("%")
|
||||
switch (dbType) {
|
||||
case MYSQL:
|
||||
case JdbcResource.MYSQL:
|
||||
rs = databaseMetaData.getColumns(dbName, null, tableName, null);
|
||||
break;
|
||||
case POSTGRESQL:
|
||||
case JdbcResource.POSTGRESQL:
|
||||
rs = databaseMetaData.getColumns(null, dbName, tableName, null);
|
||||
break;
|
||||
default:
|
||||
@ -338,9 +315,9 @@ public class JdbcClient {
|
||||
|
||||
public Type jdbcTypeToDoris(JdbcFieldSchema fieldSchema) {
|
||||
switch (dbType) {
|
||||
case MYSQL:
|
||||
case JdbcResource.MYSQL:
|
||||
return mysqlTypeToDoris(fieldSchema);
|
||||
case POSTGRESQL:
|
||||
case JdbcResource.POSTGRESQL:
|
||||
return postgresqlTypeToDoris(fieldSchema);
|
||||
default:
|
||||
throw new JdbcClientException("Unknown database type");
|
||||
@ -519,32 +496,4 @@ public class JdbcClient {
|
||||
}
|
||||
return dorisTableSchema;
|
||||
}
|
||||
|
||||
private String computeObjectChecksum() {
|
||||
if (FeConstants.runningUnitTest) {
|
||||
// skip checking checksum when running ut
|
||||
return "";
|
||||
}
|
||||
|
||||
InputStream inputStream = null;
|
||||
try {
|
||||
inputStream = Util.getInputStreamFromUrl(driverUrl, null, HTTP_TIMEOUT_MS, HTTP_TIMEOUT_MS);
|
||||
MessageDigest digest = MessageDigest.getInstance("MD5");
|
||||
byte[] buf = new byte[4096];
|
||||
int bytesRead = 0;
|
||||
do {
|
||||
bytesRead = inputStream.read(buf);
|
||||
if (bytesRead < 0) {
|
||||
break;
|
||||
}
|
||||
digest.update(buf, 0, bytesRead);
|
||||
} while (true);
|
||||
return Hex.encodeHexString(digest.digest());
|
||||
} catch (IOException e) {
|
||||
throw new JdbcClientException("compute driver checksum from url: " + driverUrl + " meet an IOException.");
|
||||
} catch (NoSuchAlgorithmException e) {
|
||||
throw new JdbcClientException(
|
||||
"compute driver checksum from url: " + driverUrl + " could not find algorithm.");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@ -53,7 +53,7 @@ public class CreateCatalogStmtTest {
|
||||
Map<String, String> props = Maps.newHashMap();
|
||||
props.put("type", "hms");
|
||||
props.put("hive.metastore.uris", "thrift://localhost:9083");
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, "testCatalog", props);
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, "testCatalog", null, props);
|
||||
stmt.analyze(analyzer);
|
||||
Assert.assertEquals("testCatalog", stmt.getCatalogName());
|
||||
Assert.assertNotNull(stmt.getProperties());
|
||||
@ -65,7 +65,7 @@ public class CreateCatalogStmtTest {
|
||||
Map<String, String> props = Maps.newHashMap();
|
||||
props.put("type", "hms");
|
||||
props.put("hive.metastore.uris", "thrift://localhost:9083");
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, "", props);
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, "", null, props);
|
||||
stmt.analyze(analyzer);
|
||||
Assert.fail("no exception");
|
||||
}
|
||||
@ -75,7 +75,7 @@ public class CreateCatalogStmtTest {
|
||||
Map<String, String> props = Maps.newHashMap();
|
||||
props.put("type", "hms");
|
||||
props.put("hive.metastore.uris", "thrift://localhost:9083");
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, InternalCatalog.INTERNAL_CATALOG_NAME, props);
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, InternalCatalog.INTERNAL_CATALOG_NAME, null, props);
|
||||
stmt.analyze(analyzer);
|
||||
Assert.fail("no exception");
|
||||
}
|
||||
@ -84,7 +84,7 @@ public class CreateCatalogStmtTest {
|
||||
public void testPropsTypeException() throws UserException {
|
||||
Map<String, String> props = Maps.newHashMap();
|
||||
props.put("hive.metastore.uris", "thrift://localhost:9083");
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, InternalCatalog.INTERNAL_CATALOG_NAME, props);
|
||||
CreateCatalogStmt stmt = new CreateCatalogStmt(false, InternalCatalog.INTERNAL_CATALOG_NAME, null, props);
|
||||
stmt.analyze(analyzer);
|
||||
Assert.fail("no exception");
|
||||
}
|
||||
|
||||
@ -33,6 +33,7 @@ import org.apache.doris.catalog.Column;
|
||||
import org.apache.doris.catalog.Env;
|
||||
import org.apache.doris.catalog.EsResource;
|
||||
import org.apache.doris.catalog.PrimitiveType;
|
||||
import org.apache.doris.catalog.ResourceMgr;
|
||||
import org.apache.doris.catalog.external.EsExternalDatabase;
|
||||
import org.apache.doris.catalog.external.EsExternalTable;
|
||||
import org.apache.doris.catalog.external.HMSExternalDatabase;
|
||||
@ -40,7 +41,6 @@ import org.apache.doris.catalog.external.HMSExternalTable;
|
||||
import org.apache.doris.common.AnalysisException;
|
||||
import org.apache.doris.common.DdlException;
|
||||
import org.apache.doris.common.FeConstants;
|
||||
import org.apache.doris.common.UserException;
|
||||
import org.apache.doris.mysql.privilege.PaloAuth;
|
||||
import org.apache.doris.qe.ConnectContext;
|
||||
import org.apache.doris.qe.ShowResultSet;
|
||||
@ -68,11 +68,13 @@ public class CatalogMgrTest extends TestWithFeService {
|
||||
private static UserIdentity user1;
|
||||
private static UserIdentity user2;
|
||||
private CatalogMgr mgr;
|
||||
private ResourceMgr resourceMgr;
|
||||
|
||||
@Override
|
||||
protected void runBeforeAll() throws Exception {
|
||||
FeConstants.runningUnitTest = true;
|
||||
mgr = Env.getCurrentEnv().getCatalogMgr();
|
||||
resourceMgr = Env.getCurrentEnv().getResourceMgr();
|
||||
|
||||
ConnectContext rootCtx = createDefaultCtx();
|
||||
env = Env.getCurrentEnv();
|
||||
@ -188,13 +190,9 @@ public class CatalogMgrTest extends TestWithFeService {
|
||||
// Can't alter catalog with resource directly
|
||||
String alterCltWithResource = "ALTER CATALOG hive SET PROPERTIES"
|
||||
+ " ('hive.metastore.uris' = 'thrift://192.168.0.2:9084');";
|
||||
try {
|
||||
mgr.alterCatalogProps((AlterCatalogPropertyStmt) parseAndAnalyzeStmt(alterCltWithResource));
|
||||
Assert.fail("Can't alter catalog with resource directly");
|
||||
} catch (UserException e) {
|
||||
Assert.assertEquals(e.getMessage(),
|
||||
"errCode = 2, detailMessage = Catalog hive has hms_resource resource, please change the resource properties directly.");
|
||||
}
|
||||
mgr.alterCatalogProps((AlterCatalogPropertyStmt) parseAndAnalyzeStmt(alterCltWithResource));
|
||||
Assertions.assertEquals("thrift://192.168.0.2:9084",
|
||||
mgr.getCatalog("hive").getProperties().get("hive.metastore.uris"));
|
||||
|
||||
showCatalogSql = "SHOW CATALOGS LIKE 'hms%'";
|
||||
showStmt = (ShowCatalogStmt) parseAndAnalyzeStmt(showCatalogSql);
|
||||
|
||||
@ -0,0 +1,31 @@
|
||||
-- This file is automatically generated. You should know what you did if you want to edit this
|
||||
-- !sql11 --
|
||||
jdbc1_catalog CREATE CATALOG `jdbc1_catalog` PROPERTIES (\n"password" = "*XXX",\n"driver_class" = "com.mysql.cj.jdbc.Driver",\n"checksum" = "fdf55dcef04b09f2eaf42b75e61ccc9a",\n"driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",\n"type" = "jdbc",\n"user" = "root",\n"jdbc_url" = "jdbc:mysql://127.0.0.1:3336/doris_test?useSSL=false&yearIsDateType=false&tinyInt1isBit=false"\n);
|
||||
|
||||
-- !sql12 --
|
||||
jdbc1_catalog CREATE CATALOG `jdbc1_catalog` PROPERTIES (\n"password" = "*XXX",\n"driver_class" = "com.mysql.cj.jdbc.Driver",\n"checksum" = "fdf55dcef04b09f2eaf42b75e61ccc9a",\n"driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",\n"type" = "jdbc",\n"user" = "root2",\n"jdbc_url" = "jdbc:mysql://127.0.0.1:3336/doris_test?useSSL=false&yearIsDateType=false&tinyInt1isBit=false"\n);
|
||||
|
||||
-- !sql13 --
|
||||
jdbc1_catalog CREATE CATALOG `jdbc1_catalog` PROPERTIES (\n"password" = "*XXX",\n"driver_class" = "com.mysql.cj.jdbc.Driver",\n"checksum" = "fdf55dcef04b09f2eaf42b75e61ccc9a",\n"driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",\n"type" = "jdbc",\n"user" = "root2",\n"jdbc_url" = "jdbc:mysql://127.0.0.1:3336/doris_test?useSSL=false&yearIsDateType=false&tinyInt1isBit=false"\n);
|
||||
|
||||
-- !sql14 --
|
||||
jdbc1_catalog CREATE CATALOG `jdbc1_catalog` PROPERTIES (\n"password" = "*XXX",\n"driver_class" = "com.mysql.jdbc.Driver",\n"checksum" = "fdf55dcef04b09f2eaf42b75e61ccc9a",\n"driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",\n"type" = "jdbc",\n"user" = "root2",\n"jdbc_url" = "jdbc:mysql://127.0.0.1:3336/doris_test?useSSL=false&yearIsDateType=false&tinyInt1isBit=false"\n);
|
||||
|
||||
-- !sql21 --
|
||||
hive1_catalog CREATE CATALOG `hive1_catalog` PROPERTIES (\n"type" = "hms",\n"hive.metastore.uris" = "thrift://127.0.0.1:9184"\n);
|
||||
|
||||
-- !sql22 --
|
||||
hive1_catalog CREATE CATALOG `hive1_catalog` PROPERTIES (\n"type" = "hms",\n"new_config" = "value1",\n"hive.metastore.uris" = "thrift://127.0.0.1:9184"\n);
|
||||
|
||||
-- !sql23 --
|
||||
hive1_catalog CREATE CATALOG `hive1_catalog` PROPERTIES (\n"type" = "hms",\n"new_config" = "value1",\n"hive.metastore.uris" = "thrift://127.0.0.2:9184"\n);
|
||||
|
||||
-- !sql31 --
|
||||
es1_catalog CREATE CATALOG `es1_catalog` PROPERTIES (\n"nodes_discovery" = "true",\n"enable_docvalue_scan" = "true",\n"http_ssl_enabled" = "false",\n"type" = "es",\n"enable_keyword_sniff" = "true",\n"hosts" = "http://127.0.0.1:29222"\n);
|
||||
|
||||
-- !sql22 --
|
||||
es1_catalog CREATE CATALOG `es1_catalog` PROPERTIES (\n"nodes_discovery" = "true",\n"enable_docvalue_scan" = "true",\n"http_ssl_enabled" = "false",\n"type" = "es",\n"enable_keyword_sniff" = "false",\n"hosts" = "http://127.0.0.1:29222"\n);
|
||||
|
||||
-- !sql23 --
|
||||
es1_catalog CREATE CATALOG `es1_catalog` PROPERTIES (\n"nodes_discovery" = "true",\n"enable_docvalue_scan" = "true",\n"http_ssl_enabled" = "false",\n"type" = "es",\n"enable_keyword_sniff" = "false",\n"hosts" = "http://127.0.0.2:29222"\n);
|
||||
|
||||
@ -80,14 +80,14 @@ b 1
|
||||
c 1
|
||||
|
||||
-- !ex_tb13 --
|
||||
张三0 11 1234567 123 321312 1999-02-13T00:00 中国 男 false
|
||||
张三1 11 12345678 123 321312 1999-02-13T00:00 中国 男 false
|
||||
张三2 11 12345671 123 321312 1999-02-13T00:00 中国 男 false
|
||||
张三3 11 12345673 123 321312 1999-02-13T00:00 中国 男 false
|
||||
张三4 11 123456711 123 321312 1999-02-13T00:00 中国 男 false
|
||||
张三5 11 1232134567 123 321312 1999-02-13T00:00 中国 男 false
|
||||
张三6 11 124314567 123 321312 1999-02-13T00:00 中国 男 false
|
||||
张三7 11 123445167 123 321312 1998-02-13T00:00 中国 男 false
|
||||
张三0 11 1234567 123 321312 1999-02-13T00:00 中国 男 0
|
||||
张三1 11 12345678 123 321312 1999-02-13T00:00 中国 男 0
|
||||
张三2 11 12345671 123 321312 1999-02-13T00:00 中国 男 0
|
||||
张三3 11 12345673 123 321312 1999-02-13T00:00 中国 男 0
|
||||
张三4 11 123456711 123 321312 1999-02-13T00:00 中国 男 0
|
||||
张三5 11 1232134567 123 321312 1999-02-13T00:00 中国 男 0
|
||||
张三6 11 124314567 123 321312 1999-02-13T00:00 中国 男 0
|
||||
张三7 11 123445167 123 321312 1998-02-13T00:00 中国 男 0
|
||||
|
||||
-- !ex_tb14 --
|
||||
123 2022-11-02 2022-11-02 8011 oppo
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@ -1,4 +1,4 @@
|
||||
-- This file is automatically generated. You should know what you did if you want to edit this
|
||||
-- !select --
|
||||
test_show_create_mysql_jdbc_catalog CREATE CATALOG `test_show_create_mysql_jdbc_catalog` PROPERTIES (\n"jdbc.password" = "*XXX",\n"jdbc.driver_class" = "com.mysql.cj.jdbc.Driver",\n"jdbc.user" = "root",\n"jdbc.jdbc_url" = "jdbc:mysql://127.0.0.1:3316/doris_test?useSSL=false",\n"jdbc.driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",\n"type" = "jdbc"\n);
|
||||
test_show_create_mysql_jdbc_catalog CREATE CATALOG `test_show_create_mysql_jdbc_catalog` PROPERTIES (\n"password" = "*XXX",\n"driver_class" = "com.mysql.cj.jdbc.Driver",\n"checksum" = "fdf55dcef04b09f2eaf42b75e61ccc9a",\n"driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",\n"type" = "jdbc",\n"user" = "root",\n"jdbc_url" = "jdbc:mysql://127.0.0.1:3336/doris_test?useSSL=false&yearIsDateType=false&tinyInt1isBit=false"\n);
|
||||
|
||||
|
||||
@ -22,9 +22,27 @@ doris_test
|
||||
|
||||
-- !select --
|
||||
ex_tb0
|
||||
|
||||
-- !select --
|
||||
ex_tb0
|
||||
ex_tb1
|
||||
ex_tb10
|
||||
ex_tb11
|
||||
ex_tb12
|
||||
ex_tb13
|
||||
ex_tb14
|
||||
ex_tb15
|
||||
ex_tb16
|
||||
ex_tb17
|
||||
ex_tb18
|
||||
ex_tb19
|
||||
ex_tb2
|
||||
ex_tb20
|
||||
ex_tb3
|
||||
ex_tb4
|
||||
ex_tb5
|
||||
ex_tb6
|
||||
ex_tb7
|
||||
ex_tb8
|
||||
ex_tb9
|
||||
test1
|
||||
|
||||
-- !select --
|
||||
ex_tb0
|
||||
@ -32,9 +50,54 @@ ex_tb0
|
||||
-- !select --
|
||||
ex_tb0
|
||||
ex_tb1
|
||||
ex_tb10
|
||||
ex_tb11
|
||||
ex_tb12
|
||||
ex_tb13
|
||||
ex_tb14
|
||||
ex_tb15
|
||||
ex_tb16
|
||||
ex_tb17
|
||||
ex_tb18
|
||||
ex_tb19
|
||||
ex_tb2
|
||||
ex_tb20
|
||||
ex_tb3
|
||||
ex_tb4
|
||||
ex_tb5
|
||||
ex_tb6
|
||||
ex_tb7
|
||||
ex_tb8
|
||||
ex_tb9
|
||||
test1
|
||||
|
||||
-- !select --
|
||||
ex_tb0
|
||||
ex_tb1
|
||||
|
||||
-- !select --
|
||||
ex_tb0
|
||||
ex_tb1
|
||||
ex_tb10
|
||||
ex_tb11
|
||||
ex_tb12
|
||||
ex_tb13
|
||||
ex_tb14
|
||||
ex_tb15
|
||||
ex_tb16
|
||||
ex_tb17
|
||||
ex_tb18
|
||||
ex_tb19
|
||||
ex_tb2
|
||||
ex_tb20
|
||||
ex_tb3
|
||||
ex_tb4
|
||||
ex_tb5
|
||||
ex_tb6
|
||||
ex_tb7
|
||||
ex_tb8
|
||||
ex_tb9
|
||||
test1
|
||||
|
||||
-- !select --
|
||||
doris_test
|
||||
@ -52,4 +115,25 @@ ex_tb1
|
||||
|
||||
-- !select --
|
||||
ex_tb0
|
||||
ex_tb1
|
||||
ex_tb10
|
||||
ex_tb11
|
||||
ex_tb12
|
||||
ex_tb13
|
||||
ex_tb14
|
||||
ex_tb15
|
||||
ex_tb16
|
||||
ex_tb17
|
||||
ex_tb18
|
||||
ex_tb19
|
||||
ex_tb2
|
||||
ex_tb20
|
||||
ex_tb3
|
||||
ex_tb4
|
||||
ex_tb5
|
||||
ex_tb6
|
||||
ex_tb7
|
||||
ex_tb8
|
||||
ex_tb9
|
||||
test1
|
||||
|
||||
|
||||
@ -0,0 +1,98 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
suite("test_catalog_with_resource", "p0") {
|
||||
String enabledJdbc = context.config.otherConfigs.get("enableJdbcTest")
|
||||
String enabledHive = context.config.otherConfigs.get("enableHiveTest")
|
||||
String enabledEs = context.config.otherConfigs.get("enableEsTest")
|
||||
if (enabledJdbc != null && enabledJdbc.equalsIgnoreCase("true")
|
||||
&& enabledHive != null && enabledHive.equalsIgnoreCase("true")
|
||||
&& enabledEs != null && enabledEs.equalsIgnoreCase("true")) {
|
||||
|
||||
String jdbc_resource_name = "jdbc1_resource"
|
||||
String hive_resource_name = "hive1_resource"
|
||||
String es_resource_name = "es1_resource"
|
||||
|
||||
String jdbc_catalog_name = "jdbc1_catalog";
|
||||
String hive_catalog_name = "hive1_catalog";
|
||||
String es_catalog_name = "es1_catalog";
|
||||
|
||||
String mysql_port = context.config.otherConfigs.get("mysql_57_port");
|
||||
String hms_port = context.config.otherConfigs.get("hms_port")
|
||||
String es_7_port = context.config.otherConfigs.get("es_7_port")
|
||||
|
||||
sql """drop catalog if exists ${jdbc_catalog_name} """
|
||||
sql """drop resource if exists ${jdbc_resource_name} """
|
||||
sql """drop catalog if exists ${hive_catalog_name} """
|
||||
sql """drop resource if exists ${hive_resource_name} """
|
||||
sql """drop catalog if exists ${es_catalog_name} """
|
||||
sql """drop resource if exists ${es_resource_name} """
|
||||
|
||||
// jdbc
|
||||
sql """create resource if not exists ${jdbc_resource_name} properties(
|
||||
"type"="jdbc",
|
||||
"user"="root",
|
||||
"password"="123456",
|
||||
"jdbc_url" = "jdbc:mysql://127.0.0.1:${mysql_port}/doris_test?useSSL=false",
|
||||
"driver_url" = "https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/jdbc_driver/mysql-connector-java-8.0.25.jar",
|
||||
"driver_class" = "com.mysql.cj.jdbc.Driver"
|
||||
);"""
|
||||
sql """CREATE CATALOG ${jdbc_catalog_name} WITH RESOURCE ${jdbc_resource_name}"""
|
||||
|
||||
qt_sql11 """show create catalog ${jdbc_catalog_name}"""
|
||||
sql """alter catalog ${jdbc_catalog_name} set properties("user"="root2")"""
|
||||
qt_sql12 """show create catalog ${jdbc_catalog_name}"""
|
||||
sql """alter resource ${jdbc_resource_name} properties("user"="root3")"""
|
||||
qt_sql13 """show create catalog ${jdbc_catalog_name}"""
|
||||
sql """alter resource ${jdbc_resource_name} properties("driver_class"="com.mysql.jdbc.Driver")"""
|
||||
qt_sql14 """show create catalog ${jdbc_catalog_name}"""
|
||||
|
||||
// hive
|
||||
sql """drop catalog if exists ${hive_catalog_name} """
|
||||
sql """drop resource if exists ${hive_resource_name} """
|
||||
sql """create resource if not exists ${hive_resource_name} properties (
|
||||
"type"="hms",
|
||||
'hive.metastore.uris' = 'thrift://127.0.0.1:${hms_port}'
|
||||
);"""
|
||||
sql """create catalog if not exists ${hive_catalog_name} with resource ${hive_resource_name};"""
|
||||
qt_sql21 """show create catalog ${hive_catalog_name}"""
|
||||
sql """alter catalog ${hive_catalog_name} set properties("new_config"="value1")"""
|
||||
qt_sql22 """show create catalog ${hive_catalog_name}"""
|
||||
sql """alter resource ${hive_resource_name} properties('hive.metastore.uris' = 'thrift://127.0.0.2:${hms_port}')"""
|
||||
qt_sql23 """show create catalog ${hive_catalog_name}"""
|
||||
|
||||
// es
|
||||
sql """drop catalog if exists ${es_catalog_name} """
|
||||
sql """drop resource if exists ${es_resource_name} """
|
||||
|
||||
sql """create resource if not exists ${es_resource_name} properties(
|
||||
"type"="es",
|
||||
"hosts"="http://127.0.0.1:$es_7_port",
|
||||
"nodes_discovery"="false",
|
||||
"enable_keyword_sniff"="true"
|
||||
);
|
||||
"""
|
||||
sql """create catalog ${es_catalog_name} with resource ${es_resource_name} properties
|
||||
("nodes_discovery"="true");
|
||||
"""
|
||||
qt_sql31 """show create catalog ${es_catalog_name}"""
|
||||
sql """alter catalog ${es_catalog_name} set properties("enable_keyword_sniff"="false")"""
|
||||
qt_sql22 """show create catalog ${es_catalog_name}"""
|
||||
sql """alter resource ${es_resource_name} properties('hosts' = 'http://127.0.0.2:${es_7_port}')"""
|
||||
qt_sql23 """show create catalog ${es_catalog_name}"""
|
||||
}
|
||||
}
|
||||
@ -46,7 +46,7 @@ suite("test_jdbc_query_mysql", "p0") {
|
||||
sql """drop table if exists $jdbcMysql57Table1"""
|
||||
sql """
|
||||
CREATE EXTERNAL TABLE `$jdbcMysql57Table1` (
|
||||
k1 boolean,
|
||||
k1 tinyint,
|
||||
k2 char(100),
|
||||
k3 varchar(128),
|
||||
k4 date,
|
||||
@ -541,7 +541,7 @@ suite("test_jdbc_query_mysql", "p0") {
|
||||
birthday DATETIME,
|
||||
country varchar(128),
|
||||
gender varchar(128),
|
||||
covid BOOLEAN
|
||||
covid tinyint
|
||||
) ENGINE=JDBC
|
||||
COMMENT "JDBC Mysql 外部表"
|
||||
PROPERTIES (
|
||||
|
||||
Reference in New Issue
Block a user