Files

wyb 4978bd6c81 [Spark load] Add resource manager (#3418 )

1. User interface:

1.1 Spark resource management

Spark is used as an external computing resource in Doris to do ETL work. In the future, there may be other external resources that will be used in Doris, for example, MapReduce is used for ETL, Spark/GPU is used for queries, HDFS/S3  is used for external storage. We introduced resource management to manage these external resources used by Doris.

```sql
-- create spark resource
CREATE EXTERNAL RESOURCE resource_name
PROPERTIES 
(                 
  type = spark,
  spark_conf_key = spark_conf_value,
  working_dir = path,
  broker = broker_name,
  broker.property_key = property_value
)

-- drop spark resource
DROP RESOURCE resource_name

-- show resources
SHOW RESOURCES
SHOW PROC "/resources"

-- privileges
GRANT USAGE_PRIV ON RESOURCE resource_name TO user_identity
GRANT USAGE_PRIV ON RESOURCE resource_name TO ROLE role_name

REVOKE USAGE_PRIV ON RESOURCE resource_name FROM user_identity
REVOKE USAGE_PRIV ON RESOURCE resource_name FROM ROLE role_name
```



- CREATE EXTERNAL RESOURCE:

FOR user_name is optional. If there has, the external resource belongs to this user. If not, the external resource belongs to the system and all users are available.

PROPERTIES：
1. type: resource type. Only support spark now.
2. spark configuration: follow the standard writing of Spark configurations, refer to: https://spark.apache.org/docs/latest/configuration.html.
3. working_dir: optional, used to store ETL intermediate results in spark ETL.
4. broker: optional, used in spark ETL. The ETL intermediate results need to be read with the broker when pushed into BE.

Example: 

```sql
CREATE EXTERNAL RESOURCE "spark0"
PROPERTIES 
(                                                                             
  "type" = "spark",                   
  "spark.master" = "yarn",
  "spark.submit.deployMode" = "cluster",
  "spark.jars" = "xxx.jar,yyy.jar",
  "spark.files" = "/tmp/aaa,/tmp/bbb",
  "spark.yarn.queue" = "queue0",
  "spark.executor.memory" = "1g",
  "spark.hadoop.yarn.resourcemanager.address" = "127.0.0.1:9999",
  "spark.hadoop.fs.defaultFS" = "hdfs://127.0.0.1:10000",
  "working_dir" = "hdfs://127.0.0.1:10000/tmp/doris",
  "broker" = "broker0",
  "broker.username" = "user0",
  "broker.password" = "password0"
)
```



- SHOW RESOURCES:
General users can only see their own resources.
Admin and root users can show all resources.




1.2 Create spark load job

```sql
LOAD LABEL db_name.label_name 
(
  DATA INFILE ("/tmp/file1") INTO TABLE table_name, ...
)
WITH RESOURCE resource_name
[(key1 = value1, ...)]
[PROPERTIES (key2 = value2, ... )]
```

Example:

```sql
LOAD LABEL example_db.test_label 
(
  DATA INFILE ("hdfs:/127.0.0.1:10000/tmp/file1") INTO TABLE example_table
)
WITH RESOURCE "spark0"
(
  "spark.executor.memory" = "1g",
  "spark.files" = "/tmp/aaa,/tmp/bbb"
)
PROPERTIES ("timeout" = "3600")
```

The spark configurations in load stmt can override the existing configuration in the resource for temporary use.

#3010

2020-05-26 18:21:21 +08:00

3.6 KiB

Raw Blame History

title, language

title	language
GRANT	zh-CN

GRANT

description

GRANT 命令用于赋予指定用户或角色指定的权限。

Syntax:

GRANT privilege_list ON db_name[.tbl_name] TO user_identity [ROLE role_name]

GRANT privilege_list ON RESOURCE resource_name TO user_identity [ROLE role_name]

privilege_list 是需要赋予的权限列表，以逗号分隔。当前 Doris 支持如下权限：

NODE_PRIV：集群节点操作权限，包括节点上下线等操作，只有 root 用户有该权限，不可赋予其他用户。
ADMIN_PRIV：除 NODE_PRIV 以外的所有权限。
GRANT_PRIV: 操作权限的权限。包括创建删除用户、角色，授权和撤权，设置密码等。
SELECT_PRIV：对指定的库或表的读取权限
LOAD_PRIV：对指定的库或表的导入权限
ALTER_PRIV：对指定的库或表的schema变更权限
CREATE_PRIV：对指定的库或表的创建权限
DROP_PRIV：对指定的库或表的删除权限
USAGE_PRIV: 对指定资源的使用权限

旧版权限中的 ALL 和 READ_WRITE 会被转换成：SELECT_PRIV,LOAD_PRIV,ALTER_PRIV,CREATE_PRIV,DROP_PRIV；
READ_ONLY 会被转换为 SELECT_PRIV。

权限分类：

1. 节点权限：NODE_PRIV
2. 库表权限：SELECT_PRIV,LOAD_PRIV,ALTER_PRIV,CREATE_PRIV,DROP_PRIV
3. 资源权限：USAGE_PRIV

db_name[.tbl_name] 支持以下三种形式：

1. *.* 权限可以应用于所有库及其中所有表
2. db.* 权限可以应用于指定库下的所有表
3. db.tbl 权限可以应用于指定库下的指定表

这里指定的库或表可以是不存在的库和表。

resource_name 支持以下两种形式：

1. * 权限应用于所有资源
2. resource 权限应用于指定资源

这里指定的资源可以是不存在的资源。

user_identity：

这里的 user_identity 语法同 CREATE USER。且必须为使用 CREATE USER 创建过的 user_identity。user_identity 中的host可以是域名，如果是域名的话，权限的生效时间可能会有1分钟左右的延迟。

也可以将权限赋予指定的 ROLE，如果指定的 ROLE 不存在，则会自动创建。

example

1. 授予所有库和表的权限给用户

    GRANT SELECT_PRIV ON *.* TO 'jack'@'%';

2. 授予指定库表的权限给用户

    GRANT SELECT_PRIV,ALTER_PRIV,LOAD_PRIV ON db1.tbl1 TO 'jack'@'192.8.%';
    
3. 授予指定库表的权限给角色

    GRANT LOAD_PRIV ON db1.* TO ROLE 'my_role';

4. 授予所有资源的使用权限给用户

    GRANT USAGE_PRIV ON RESOURCE * TO 'jack'@'%';

5. 授予指定资源的使用权限给用户

    GRANT USAGE_PRIV ON RESOURCE 'spark_resource' TO 'jack'@'%';

6. 授予指定资源的使用权限给角色

    GRANT USAGE_PRIV ON RESOURCE 'spark_resource' TO ROLE 'my_role';

keyword

GRANT

3.6 KiB Raw Blame History

GRANT

description

example

keyword

3.6 KiB

Raw Blame History