[Docs] Added some missing documents (#4710)
1. Add ODBC resource in resource manage doc and add english of resource manage doc 2. Add miss part in sidebar
This commit is contained in:
HappenLee
@ -166,6 +166,7 @@ module.exports = [
|
||||
"export_manual",
|
||||
"outfile",
|
||||
"privilege",
|
||||
"resource-management",
|
||||
"running-profile",
|
||||
"small-file-mgr",
|
||||
"sql-mode",
|
||||
@ -180,6 +181,7 @@ module.exports = [
|
||||
children: [
|
||||
"audit-plugin",
|
||||
"doris-on-es",
|
||||
"odbc-of-doris",
|
||||
"plugin-development-manual",
|
||||
"spark-doris-connector",
|
||||
"logstash",
|
||||
|
||||
@ -184,6 +184,7 @@ module.exports = [
|
||||
children: [
|
||||
"audit-plugin",
|
||||
"doris-on-es",
|
||||
"odbc-of-doris",
|
||||
"plugin-development-manual",
|
||||
"spark-doris-connector",
|
||||
"logstash",
|
||||
|
||||
169
docs/en/administrator-guide/resource-management.md
Normal file
169
docs/en/administrator-guide/resource-management.md
Normal file
@ -0,0 +1,169 @@
|
||||
---
|
||||
{
|
||||
"title": "Resource management",
|
||||
"language": "en"
|
||||
}
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Resource Management
|
||||
|
||||
In order to save the compute and storage resources in the Doris cluster, Doris needs to reference to some other external resources to do the related work. such as spark/GPU for query, HDFS/S3 for external storage, spark/MapReduce for ETL, connect to external storage by ODBC driver. Therefore, Doris need a resource management mechanism to manage these external resources.
|
||||
|
||||
## Fundamental Concept
|
||||
|
||||
A resource contains basic information such as name and type. The name is globally unique. Different types of resources contain different attributes. Please refer to the introduction of each resource for details.
|
||||
|
||||
The creation and deletion of resources can only be performed by users own `admin` permission. One resource belongs to the entire Doris cluster. Users with `admin` permission can assign permission of resource to other users. Please refer to `HELP GRANT` or doris document.
|
||||
|
||||
|
||||
## Operation Of Resource
|
||||
|
||||
There are three main commands for resource management: `create resource`, `drop resource` and `show resources`. They are to create, delete and check resources. The specific syntax of these three commands can be viewed by executing `help CMD` after MySQL client connects to Doris.
|
||||
|
||||
1. CREATE RESOURCE
|
||||
|
||||
|
||||
```sql
|
||||
CREATE [EXTERNAL] RESOURCE "resource_name"
|
||||
PROPERTIES ("key"="value", ...);
|
||||
```
|
||||
|
||||
In the command to create a resource, the user must provide the following information:
|
||||
|
||||
* `resource_name` name of the resource
|
||||
* `PROPERTIES` related parameters, as follows：
|
||||
* `type`：resource type, required. Currently, only spark and odbc_catalog are supported.
|
||||
* For other parameters, see the resource introduction
|
||||
|
||||
|
||||
|
||||
2. DROP RESOURCE
|
||||
|
||||
This command can delete an existing resource. For details, please refer to: `HELP DROP RESOURCE`
|
||||
|
||||
3. SHOW RESOURCES
|
||||
|
||||
This command can view the resources that the user has permission to use. Please refer to：`HELP SHOW RESOURCES`
|
||||
|
||||
|
||||
|
||||
## Resources Supported
|
||||
|
||||
Currently, Doris can support
|
||||
|
||||
* Spark resource : do ETL work
|
||||
* ODBC resource : query and import data from external tables
|
||||
|
||||
The following shows how the two resources are used.
|
||||
|
||||
### Spark
|
||||
|
||||
#### Parameter
|
||||
|
||||
##### Spark Parameters：
|
||||
|
||||
`spark.master`: required, currently supported yarn，spark://host:port。
|
||||
|
||||
`spark.submit.deployMode`: The deployment mode of spark. required. It supports cluster and client.
|
||||
|
||||
`spark.hadoop.yarn.resourcemanager.address`: required when master is yarn.
|
||||
|
||||
`spark.hadoop.fs.defaultFS`: required when master is yarn.
|
||||
|
||||
Other parameters are optional, refer to: http://spark.apache.org/docs/latest/configuration.html.
|
||||
|
||||
##### If spark is used for ETL, also need to specify the following parameters：
|
||||
|
||||
`working_dir`: Directory used by ETL. Spark is required when used as an ETL resource. For example: hdfs://host:port/tmp/doris.
|
||||
|
||||
`broker`: The name of broker. Is required when spark be used as ETL resource. You need to use the `ALTER SYSTEM ADD BROKER` command to complete the configuration in advance.。
|
||||
|
||||
* `broker.property_key`: When the broker reads the intermediate file generated by ETL, it needs the specified authentication information.
|
||||
|
||||
|
||||
|
||||
#### Example
|
||||
|
||||
Create a spark resource named `spark0 `in the yarn cluster mode.
|
||||
|
||||
|
||||
```sql
|
||||
CREATE EXTERNAL RESOURCE "spark0"
|
||||
PROPERTIES
|
||||
(
|
||||
"type" = "spark",
|
||||
"spark.master" = "yarn",
|
||||
"spark.submit.deployMode" = "cluster",
|
||||
"spark.jars" = "xxx.jar,yyy.jar",
|
||||
"spark.files" = "/tmp/aaa,/tmp/bbb",
|
||||
"spark.executor.memory" = "1g",
|
||||
"spark.yarn.queue" = "queue0",
|
||||
"spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999",
|
||||
"spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000",
|
||||
"working_dir" = "hdfs://127.0.0.1:10000/tmp/doris",
|
||||
"broker" = "broker0",
|
||||
"broker.username" = "user0",
|
||||
"broker.password" = "password0"
|
||||
);
|
||||
```
|
||||
|
||||
### ODBC
|
||||
|
||||
#### Parameter
|
||||
|
||||
##### ODBC Parameters：
|
||||
|
||||
`type`: Required，must be `odbc_catalog`. As the type identifier of resource.
|
||||
|
||||
`user`: The user name of the external table, required.
|
||||
|
||||
`password`: The user password of the external table, required.
|
||||
|
||||
`host`: The ip address of the external table, required.
|
||||
|
||||
`port`: The port of the external table, required.
|
||||
|
||||
`odbc_type`: Indicates the type of external table. Currently, Doris supports `MySQL` and `Oracle`. In the future, it may support more databases. The ODBC exteranl table referring to the resource is required. The old MySQL exteranl table referring to the resource is optional.
|
||||
|
||||
`driver`: Indicates the driver dynamic library used by the ODBC external table.
|
||||
The ODBC exteranl table referring to the resource is required. The old MySQL exteranl table referring to the resource is optional.
|
||||
|
||||
For the usage of ODBC resource, please refer to [ODBC of Doris](../extending-doris/odbc-of-doris.html)
|
||||
|
||||
|
||||
#### Example
|
||||
|
||||
Create the ODBC resource of Oracle, named `oracle_odbc`.
|
||||
|
||||
```sql
|
||||
CREATE EXTERNAL RESOURCE `oracle_odbc`
|
||||
PROPERTIES (
|
||||
"type" = "odbc_catalog",
|
||||
"host" = "192.168.0.1",
|
||||
"port" = "8086",
|
||||
"user" = "test",
|
||||
"password" = "test",
|
||||
"database" = "test",
|
||||
"odbc_type" = "oracle",
|
||||
"driver" = "Oracle 19 ODBC driver"
|
||||
);
|
||||
```
|
||||
@ -26,7 +26,7 @@ under the License.
|
||||
|
||||
# 资源管理
|
||||
|
||||
为了节省Doris集群内的计算、存储资源，Doris需要引入一些其他外部资源来完成相关的工作，如Spark/GPU用于查询，HDFS/S3用于外部存储，Spark/MapReduce用于ETL等，因此我们引入资源管理机制来管理Doris使用的这些外部资源。
|
||||
为了节省Doris集群内的计算、存储资源，Doris需要引入一些其他外部资源来完成相关的工作，如Spark/GPU用于查询，HDFS/S3用于外部存储，Spark/MapReduce用于ETL, 通过ODBC连接外部存储等，因此我们引入资源管理机制来管理Doris使用的这些外部资源。
|
||||
|
||||
|
||||
|
||||
@ -55,7 +55,7 @@ under the License.
|
||||
|
||||
* `resource_name` 为 Doris 中配置的资源的名字。
|
||||
* `PROPERTIES` 是资源相关参数，如下：
|
||||
* `type`：资源类型，必填，目前仅支持 spark。
|
||||
* `type`：资源类型，必填，目前仅支持 spark与odbc_catalog。
|
||||
* 其他参数见各资源介绍。
|
||||
|
||||
2. DROP RESOURCE
|
||||
@ -70,7 +70,11 @@ under the License.
|
||||
|
||||
## 支持的资源
|
||||
|
||||
目前仅支持Spark资源，完成ETL工作。下面的示例都以Spark资源为例。
|
||||
目前Dorisn能够支持
|
||||
* Spark资源 : 完成ETL工作。
|
||||
* ODBC资源：查询和导入外部表的数据
|
||||
|
||||
下面将分别展示两种资源的使用方式。
|
||||
|
||||
### Spark
|
||||
|
||||
@ -122,4 +126,45 @@ PROPERTIES
|
||||
"broker.username" = "user0",
|
||||
"broker.password" = "password0"
|
||||
);
|
||||
```
|
||||
|
||||
### ODBC
|
||||
|
||||
#### 参数
|
||||
|
||||
##### ODBC 相关参数如下：
|
||||
|
||||
`type`: 必填，且必须为`odbc_catalog`。作为resource的类型标识。
|
||||
|
||||
`user`: 外部表的账号，必填。
|
||||
|
||||
`password`: 外部表的密码，必填。
|
||||
|
||||
`host`: 外部表的连接ip地址，必填。
|
||||
|
||||
`port`: 外部表的连接端口，必填。
|
||||
|
||||
`odbc_type`: 标示外部表的类型，当前doris支持`mysql`与`oracle`，未来可能支持更多的数据库。引用该resource的ODBC外表必填，旧的mysql外表选填。
|
||||
|
||||
`driver`: 标示外部表使用的driver动态库，引用该resource的ODBC外表必填，旧的mysql外表选填。
|
||||
|
||||
|
||||
具体如何使用可以，可以参考[ODBC of Doris](../extending-doris/odbc-of-doris.html)
|
||||
|
||||
#### 示例
|
||||
|
||||
创建oracle的odbc resource，名为 odbc_oracle 的 odbc_catalog的 资源。
|
||||
|
||||
```sql
|
||||
CREATE EXTERNAL RESOURCE `oracle_odbc`
|
||||
PROPERTIES (
|
||||
"type" = "odbc_catalog",
|
||||
"host" = "192.168.0.1",
|
||||
"port" = "8086",
|
||||
"user" = "test",
|
||||
"password" = "test",
|
||||
"database" = "test",
|
||||
"odbc_type" = "oracle",
|
||||
"driver" = "Oracle 19 ODBC driver"
|
||||
);
|
||||
```
|
||||
Reference in New Issue
Block a user