Add ldap, multi-tenant, sql-block-rule documentation (#9044)

Add ldap, multi-tenant, sql-block-rule documentation
This commit is contained in:
smallhibiscus
2022-04-15 16:30:56 +08:00
committed by GitHub
parent d1d834694f
commit 415695f84e
15 changed files with 1003 additions and 25 deletions

View File

@ -528,6 +528,7 @@ module.exports = [
initialOpenGroupIndex: -1,
children: [
"ALTER-DATABASE",
"ALTER-SQL-BLOCK-RULE",
"ALTER-TABLE-COLUMN",
"ALTER-TABLE-PARTITION",
"ALTER-TABLE-PROPERTY",
@ -563,6 +564,7 @@ module.exports = [
"CREATE-INDEX",
"CREATE-MATERIALIZED-VIEW",
"CREATE-RESOURCE",
"CREATE-SQL-BLOCK-RULE",
"CREATE-TABLE-LIKE",
"CREATE-TABLE",
"CREATE-VIEW",
@ -581,6 +583,7 @@ module.exports = [
"DROP-INDEX",
"DROP-MATERIALIZED-VIEW",
"DROP-RESOURCE",
"DROP-SQL-BLOCK-RULE",
"DROP-TABLE",
"TRUNCATE-TABLE",
],
@ -696,6 +699,7 @@ module.exports = [
"SHOW-ROUTINE-LOAD",
"SHOW-SMALL-FILES",
"SHOW-SNAPSHOT",
"SHOW-SQL-BLOCK-RULE",
"SHOW-STATUS",
"SHOW-STREAM-LOAD",
"SHOW-TABLE-ID",

View File

@ -528,6 +528,7 @@ module.exports = [
initialOpenGroupIndex: -1,
children: [
"ALTER-DATABASE",
"ALTER-SQL-BLOCK-RULE",
"ALTER-TABLE-COLUMN",
"ALTER-TABLE-PARTITION",
"ALTER-TABLE-PROPERTY",
@ -563,6 +564,7 @@ module.exports = [
"CREATE-INDEX",
"CREATE-MATERIALIZED-VIEW",
"CREATE-RESOURCE",
"CREATE-SQL-BLOCK-RULE",
"CREATE-TABLE-LIKE",
"CREATE-TABLE",
"CREATE-VIEW",
@ -581,6 +583,7 @@ module.exports = [
"DROP-INDEX",
"DROP-MATERIALIZED-VIEW",
"DROP-RESOURCE",
"DROP-SQL-BLOCK-RULE",
"DROP-TABLE",
"TRUNCATE-TABLE",
],
@ -696,6 +699,7 @@ module.exports = [
"SHOW-ROUTINE-LOAD",
"SHOW-SMALL-FILES",
"SHOW-SNAPSHOT",
"SHOW-SQL-BLOCK-RULE",
"SHOW-STATUS",
"SHOW-STREAM-LOAD",
"SHOW-TABLE-ID",

View File

@ -73,7 +73,7 @@ You need to configure the LDAP basic information in the fe/conf/ldap.conf file,
#### Set the LDAP administrator password:
After configuring the ldap.conf file, start fe, log in to Doris with the root or admin account, and execute sql:
```
```sql
set ldap_admin_password = 'ldap_admin_password';
```
@ -82,14 +82,14 @@ Client-side LDAP authentication requires the mysql client-side explicit authenti
* Set the environment variable LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN to value 1.
For example, in a linux or max environment you can use the command:
```
```bash
echo "export LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN=1" >> ~/.bash_profile && source ~/.bash_profile
```
* Add the parameter "--enable-cleartext-plugin" each time you log in to Doris.
```
```sql
mysql -hDORIS_HOST -PDORIS_PORT -u user -p --enable-cleartext-plugin
Enter ldap password
```
@ -118,12 +118,12 @@ For example:
Doris account exists: jack@'172.10.1.10', password: 123456
LDAP user node presence attribute: uid: jack user password: abcdef
The jack@'172.10.1.10' account can be logged into by logging into Doris using the following command:
```
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p abcdef
```
Login will fail with the following command:
```
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p 123456
```
@ -131,7 +131,7 @@ mysql -hDoris_HOST -PDoris_PORT -ujack -p 123456
LDAP user node presence attribute: uid: jack User password: abcdef
Use the following command to create a temporary user and log in to jack@'%', the temporary user has basic privileges DatabasePrivs: Select_priv, Doris will delete the temporary user after the user logs out and logs in:
```
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p abcdef
```
@ -139,7 +139,7 @@ mysql -hDoris_HOST -PDoris_PORT -ujack -p abcdef
Doris account exists: jack@'172.10.1.10', password: 123456
Login to the account using the Doris password, successfully:
```
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p 123456
```
@ -159,7 +159,7 @@ If the logged-in user is a temporary user and no group permission exists, the us
Example:
LDAP user dn is the "member" attribute of the LDAP group node then the user is considered to belong to the group, Doris will intercept the first Rdn of group dn as the group name.
For example, if user dn is "uid=jack,ou=aidp,dc=domain,dc=com", the group information is as follows:
```
```text
dn: cn=doris_rd,ou=group,dc=domain,dc=com
objectClass: groupOfNames
member: uid=jack,ou=aidp,dc=domain,dc=com

View File

@ -72,9 +72,7 @@ Node resource division refers to setting tags for BE nodes in a Doris cluster, a
(k1 int, k2 int)
distributed by hash(k1) buckets 1
properties(
"replication_allocation"
=
"tag.location.group_a:1, tag.location.group_b:1, tag.location.group_c:1"
"replication_allocation"="tag.location.group_a:1, tag.location.group_b:1, tag.location.group_c:1"
)
```
@ -148,12 +146,12 @@ At present, Doris's resource restrictions on single queries are mainly divided i
Doris can limit the maximum memory overhead that a query is allowed to use. To ensure that the memory resources of the cluster will not be fully occupied by a query. We can set the memory limit in the following ways:
```
// Set the session variable exec_mem_limit. Then all subsequent queries in the session (within the connection) use this memory limit.
```sql
# Set the session variable exec_mem_limit. Then all subsequent queries in the session (within the connection) use this memory limit.
set exec_mem_limit=1G;
// Set the global variable exec_mem_limit. Then all subsequent queries of all new sessions (new connections) use this memory limit.
# Set the global variable exec_mem_limit. Then all subsequent queries of all new sessions (new connections) use this memory limit.
set global exec_mem_limit=1G;
// Set the variable exec_mem_limit in SQL. Then the variable only affects this SQL.
# Set the variable exec_mem_limit in SQL. Then the variable only affects this SQL.
select /*+ SET_VAR(exec_mem_limit=1G) */ id, name from tbl where xxx;
```
@ -163,15 +161,16 @@ At present, Doris's resource restrictions on single queries are mainly divided i
Users can limit the CPU resources of the query in the following ways:
```
// Set the session variable cpu_resource_limit. Then all queries in the session (within the connection) will use this CPU limit.
```sql
# Set the session variable cpu_resource_limit. Then all queries in the session (within the connection) will use this CPU limit.
set cpu_resource_limit = 2
// Set the user's attribute cpu_resource_limit, then all queries of this user will use this CPU limit. The priority of this attribute is higher than the session variable cpu_resource_limit
# Set the user's attribute cpu_resource_limit, then all queries of this user will use this CPU limit. The priority of this attribute is higher than the session variable cpu_resource_limit
set property for'user1''cpu_resource_limit' = '3';
```
The value of `cpu_resource_limit` is a relative value. The larger the value, the more CPU resources can be used. However, the upper limit of the CPU that can be used by a query also depends on the number of partitions and buckets of the table. In principle, the maximum CPU usage of a query is positively related to the number of tablets involved in the query. In extreme cases, assuming that a query involves only one tablet, even if `cpu_resource_limit` is set to a larger value, only 1 CPU resource can be used.
Through memory and CPU resource limits. We can divide user queries into more fine-grained resources within a resource group. For example, we can make some offline tasks with low timeliness requirements, but with a large amount of calculation, use less CPU resources and more memory resources. Some delay-sensitive online tasks use more CPU resources and reasonable memory resources.
## Best practices and forward compatibility
@ -190,7 +189,7 @@ Here we give an example of the steps to start using the resource division functi
After the upgrade, the default Tag of BE is `"tag.location": "default"`, and the default copy distribution of the table is: `"tag.location.default:xx`. So if you directly modify the Tag of BE, the system will Automatically detect changes in the distribution of copies, and start data redistribution. This may occupy some system resources. So we can turn off the data repair and balance logic before modifying the tag to ensure that there will be no copies when we plan resources Redistribution operation.
```
```sql
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
ADMIN SET FRONTEND CONFIG ("disable_tablet_scheduler" = "true");
```
@ -199,7 +198,7 @@ Here we give an example of the steps to start using the resource division functi
Next, you can use the `alter system modify backend` statement to set the BE Tag. And through the `alter table` statement to modify the copy distribution strategy of the table. Examples are as follows:
```
```sql
alter system modify backend "host1:9050, 1212:9050" set ("tag.location" = "group_a");
alter table my_table modify partition p1 set ("replication_allocation" = "tag.location.group_a:2");
```
@ -208,7 +207,7 @@ Here we give an example of the steps to start using the resource division functi
After the tag and copy distribution are set, we can turn on the data repair and equalization logic to trigger data redistribution.
```
```sql
ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");
ADMIN SET FRONTEND CONFIG ("disable_tablet_scheduler" = "false");
```
@ -216,7 +215,8 @@ Here we give an example of the steps to start using the resource division functi
This process will continue for a period of time depending on the amount of data involved. And it will cause some colocation tables to fail colocation planning (because the copy is being migrated). You can view the progress by `show proc "/cluster_balance/"`. You can also judge the progress by the number of `UnhealthyTabletNum` in `show proc "/statistic"`. When `UnhealthyTabletNum` drops to 0, it means that the data redistribution is completed. .
4. Set the user's resource label permissions.
After the data is redistributed. We can start to set the user's resource label permissions. Because by default, the user's `resource_tags.location` attribute is empty, that is, the BE of any tag can be accessed. Therefore, in the previous steps, the normal query of existing users will not be affected. When the `resource_tags.location` property is not empty, the user will be restricted from accessing the BE of the specified Tag.
Through the above 4 steps, we can smoothly use the resource division function after the original cluster is upgraded.

View File

@ -0,0 +1,69 @@
---
{
"title": "ALTER-SQL-BLOCK-RULE",
"language": "en"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## ALTER-SQL-BLOCK-RULE
### Name
ALTER SQL BLOCK RULE
### Description
Modify SQL blocking rules to allow modification of each item such as sql/sqlHash/partition_num/tablet_num/cardinality/global/enable.
grammar:
```sql
ALTER SQL_BLOCK_RULE rule_name
[PROPERTIES ("key"="value", ...)];
````
illustrate:
- sql and sqlHash cannot be set at the same time. This means that if a rule sets sql or sqlHash, the other attribute cannot be modified;
- sql/sqlHash and partition_num/tablet_num/cardinality cannot be set at the same time. For example, if a rule sets partition_num, then sql or sqlHash cannot be modified;
### Example
1. Modify according to SQL properties
```sql
ALTER SQL_BLOCK_RULE test_rule PROPERTIES("sql"="select \\* from test_table","enable"="true")
````
2. If a rule sets partition_num, then sql or sqlHash cannot be modified
```sql
ALTER SQL_BLOCK_RULE test_rule2 PROPERTIES("partition_num" = "10","tablet_num"="300","enable"="true")
````
### Keywords
````text
ALTER,SQL_BLOCK_RULE
````
### Best Practice

View File

@ -0,0 +1,101 @@
---
{
"title": "CREATE-SQL-BLOCK-RULE",
"language": "en"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## CREATE-SQL-BLOCK-RULE
### Name
CREATE SQL BLOCK RULE
### Description
This statement creates a SQL blocking rule, which is only used to restrict query statements, not to restrict the execution of explian statements.
Supports configuring SQL blacklists by user:
- Refuse to specify SQL by regular matching
- Check if a query reaches one of these limits by setting partition_num, tablet_num, cardinality
- partition_num, tablet_num, cardinality can be set together, once a query reaches one of these limits, the query will be intercepted
grammar:
```sql
CREATE SQL_BLOCK_RULE rule_name
[PROPERTIES ("key"="value", ...)];
````
Parameter Description:
- sql: matching rule (based on regular matching, special characters need to be translated), optional, the default value is "NULL"
- sqlHash: sql hash value, used for exact matching, we will print this value in `fe.audit.log`, optional, this parameter and sql can only be selected one, the default value is "NULL"
- partition_num: the maximum number of partitions a scan node will scan, the default value is 0L
- tablet_num: The maximum number of tablets that a scanning node will scan, the default value is 0L
- cardinality: the rough scan line number of a scan node, the default value is 0L
- global: Whether to take effect globally (all users), the default is false
- enable: whether to enable blocking rules, the default is true
### Example
1. Create a block rule named test_rule
```sql
mysql> CREATE SQL_BLOCK_RULE test_rule
-> PROPERTIES(
-> "sql"="select * from order_analysis;",
-> "global"="false",
-> "enable"="true"
-> );
Query OK, 0 rows affected (0.01 sec)
````
When we execute the sql we just defined in the rule, an exception error will be returned. The example is as follows:
```sql
mysql> select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
````
2. Create test_rule2, limit the maximum number of scanned partitions to 30, and limit the maximum scan base to 10 billion rows. The example is as follows:
```sql
mysql> CREATE SQL_BLOCK_RULE test_rule2
-> PROPERTIES (
-> "partition_num" = "30",
-> "cardinality" = "10000000000",
-> "global" = "false",
-> "enable" = "true"
-> );
Query OK, 0 rows affected (0.01 sec)
````
### Keywords
````text
CREATE, SQL_BLCOK_RULE
````
### Best Practice

View File

@ -0,0 +1,58 @@
---
{
"title": "DROP-DATABASE",
"language": "en"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## DROP-SQL-BLOCK-RULE
### Name
DROP SQL BLOCK RULE
### Description
Delete SQL blocking rules, support multiple rules, separated by ,
grammar:
```sql
DROP SQL_BLOCK_RULE test_rule1,...
````
### Example
1. Delete the test_rule1 and test_rule2 blocking rules
```sql
mysql> DROP SQL_BLOCK_RULE test_rule1,test_rule2;
Query OK, 0 rows affected (0.00 sec)
````
### Keywords
````text
DROP, SQL_BLOCK_RULE
````
### Best Practice

View File

@ -0,0 +1,76 @@
---
{
"title": "SHOW-SQL-BLOCK-RULE",
"language": "en"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## SQL-BLOCK-RULE
### Name
SHOW SQL BLOCK RULE
### Description
View the configured SQL blocking rules. If you do not specify a rule name, you will view all rules.
grammar:
```sql
SHOW SQL_BLOCK_RULE [FOR RULE_NAME];
````
### Example
1. View all rules.
```sql
mysql> SHOW SQL_BLOCK_RULE;
+------------+----------------------------+---------+- -------------+------------+-------------+--------+- -------+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
+------------+----------------------------+---------+- -------------+------------+-------------+--------+- -------+
| test_rule | select * from order_analysis | NULL | 0 | 0 | 0 | true | true |
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
+------------+----------------------------+---------+- -------------+------------+-------------+--------+- -------+
2 rows in set (0.01 sec)
````
2. Make a rule name query
```sql
mysql> SHOW SQL_BLOCK_RULE FOR test_rule2;
+------------+------+---------+---------------+---- -------+-------------+--------+--------+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
+------------+------+---------+---------------+---- -------+-------------+--------+--------+
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
+------------+------+---------+---------------+---- -------+-------------+--------+--------+
1 row in set (0.00 sec)
````
### Keywords
SHOW, SQL_BLOCK_RULE
### Best Practice

View File

@ -24,4 +24,169 @@ specific language governing permissions and limitations
under the License.
-->
# LDAP
# LDAP
接入第三方LDAP服务为Doris提供验证登录和组授权服务。
LDAP验证登录指的是接入LDAP服务的密码验证来补充Doris的验证登录。Doris优先使用LDAP验证用户密码,如果LDAP服务中不存在该用户则继续使用Doris验证密码,如果LDAP密码正确但是Doris中没有对应账户则创建临时用户登录Doris。
LDAP组授权是将LDAP中的group映射到Doris中的Role,如果用户在LDAP中属于多个用户组,登录Doris后用户将获得所有组对应Role的权限,要求组名与Role名字相同。
## 名词解释
- LDAP: 轻量级目录访问协议,能够实现账号密码的集中管理。
- 权限 Privilege:权限作用的对象是节点、数据库或表。不同的权限代表不同的操作许可。
- 角色 Role:Doris可以创建自定义命名的角色。角色可以被看做是一组权限的集合。
## 启用LDAP认证
### server端配置
需要在fe/conf/ldap.conf文件中配置LDAP基本信息,另有LDAP管理员密码需要使用sql语句进行设置。
#### 配置fe/conf/ldap.conf文件:
- ldap_authentication_enabled = false
设置值为“true”启用LDAP验证;当值为“false”时,不启用LDAP验证,该配置文件的其他配置项都无效。
- ldap_host = 127.0.0.1
LDAP服务ip。
- ldap_port = 389
LDAP服务端口,默认明文传输端口为389,目前Doris的LDAP功能仅支持明文密码传输。
- ldap_admin_name = cn=admin,dc=domain,dc=com
LDAP管理员账户“Distinguished Name”。当用户使用LDAP验证登录Doris时,Doris会绑定该管理员账户在LDAP中搜索用户信息。
- ldap_user_basedn = ou=people,dc=domain,dc=com
Doris在LDAP中搜索用户信息时的base dn。
- ldap_user_filter = (&(uid={login}))
- Doris在LDAP中搜索用户信息时的过滤条件,占位符“{login}”会被替换为登录用户名。必须保证通过该过滤条件搜索的用户唯一,否则Doris无法通过LDAP验证密码,登录时会出现“ERROR 5081 (42000): user is not unique in LDAP server.”的错误信息。
例如使用LDAP用户节点uid属性作为登录Doris的用户名可以配置该项为:
ldap_user_filter = (&(uid={login}));
使用LDAP用户邮箱前缀作为用户名可配置该项:
ldap_user_filter = (&(mail={login}@baidu.com))。
- ldap_group_basedn = ou=group,dc=domain,dc=com
Doris在LDAP中搜索组信息时的base dn。如果不配置该项,将不启用LDAP组授权。
#### 设置LDAP管理员密码:
配置好ldap.conf文件后启动fe,使用root或admin账号登录Doris,执行sql:
```sql
set ldap_admin_password = 'ldap_admin_password';
```
### Client端配置
客户端使用LDAP验证需要启用mysql客户端明文验证插件,使用命令行登录Doris可以使用下面两种方式之一启用mysql明文验证插件:
- 设置环境变量LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN值1。
例如在linux或者mac环境中可以使用:
```bash
echo "export LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN=1" >> ~/.bash_profile && source ~/.bash_profile
```
- 每次登录Doris时添加参数“--enable-cleartext-plugin”:
```bash
mysql -hDORIS_HOST -PDORIS_PORT -u user -p --enable-cleartext-plugin
输入ldap密码
```
## LDAP认证详解
LDAP密码验证和组授权是Doris密码验证和授权的补充,开启LDAP功能并不能完全替代Doris的密码验证和授权,而是与Doris密码验证和授权并存。
### LDAP验证登录详解
开启LDAP后,用户在Doris和DLAP中存在以下几种情况:
| LDAP用户 | Doris用户 | 密码 | 登录情况 | 登录Doris的用户 |
| -------- | --------- | --------- | -------- | --------------- |
| 存在 | 存在 | LDAP密码 | 登录成功 | Doris用户 |
| 存在 | 存在 | Doris密码 | 登录失败 | 无 |
| 不存在 | 存在 | Doris密码 | 登录成功 | Doris用户 |
| 存在 | 不存在 | LDAP密码 | 登录成功 | Ldap临时用户 |
开启LDAP后,用户使用mysql client登录时,Doris会先通过LDAP服务验证用户密码,如果LDAP存在用户且密码正确,Doris则使用该用户登录;此时Doris若存在对应账户则直接登录该账户,如果不存在对应账户则为用户创建临时账户并登录该账户。临时账户具有具有相应对权限(参见LDAP组授权),仅对当前连接有效,doris不会创建该用户,也不会产生创建用户对元数据。
如果LDAP服务中不存在登录用户,则使用Doris进行密码认证。
以下假设已开启LDAP认证,配置ldap_user_filter = (&(uid={login})),且其他配置项都正确,客户端设置环境变量LIBMYSQL_ENABLE_CLEARTEXT_PLUGIN=1
例如:
#### 1:Doris和LDAP中都存在账户:
存在Doris账户:jack@'172.10.1.10',密码:123456
LDAP用户节点存在属性:uid: jack 用户密码:abcdef
使用以下命令登录Doris可以登录jack@'172.10.1.10'账户:
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p abcdef
```
使用以下命令将登录失败:
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p 123456
```
#### 2:LDAP中存在用户,Doris中不存在对应账户:
LDAP用户节点存在属性:uid: jack 用户密码:abcdef
使用以下命令创建临时用户并登录jack@'%',临时用户具有基本权限 DatabasePrivs:Select_priv, 用户退出登录后Doris将删除该临时用户:
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p abcdef
```
#### 3:LDAP不存在用户:
存在Doris账户:jack@'172.10.1.10',密码:123456
使用Doris密码登录账户,成功:
```bash
mysql -hDoris_HOST -PDoris_PORT -ujack -p 123456
```
### LDAP组授权详解
DLAP用户dn是LDAP组节点的“member”属性则Doris认为用户属于该组。LDAP组授权是将LDAP中的group映射到Doris中的role,并将所有对应的role权限授予登录用户,用户退出登录后Doris会撤销对应的role权限。在使用LDAP组授权前应该在Doris中创建相应对role,并为role授权。
登录用户权限跟Doris用户和组权限有关,见下表:
| LDAP用户 | Doris用户 | 登录用户的权限 |
| -------- | --------- | -------------------------- |
| 存在 | 存在 | LDAP组权限 + Doris用户权限 |
| 不存在 | 存在 | Doris用户权限 |
| 存在 | 不存在 | LDAP组权限 |
如果登录的用户为临时用户,且不存在组权限,则该用户默认具有information_schema的select_priv权限
举例:
LDAP用户dn是LDAP组节点的“member”属性则认为用户属于该组,Doris会截取组dn的第一个Rdn作为组名。
例如用户dn为“uid=jack,ou=aidp,dc=domain,dc=com”, 组信息如下:
```text
dn: cn=doris_rd,ou=group,dc=domain,dc=com
objectClass: groupOfNames
member: uid=jack,ou=aidp,dc=domain,dc=com
```
则组名为doris_rd。
假如jack还属于LDAP组doris_qa、doris_pm;Doris存在role:doris_rd、doris_qa、doris_pm,在使用LDAP验证登录后,用户不但具有该账户原有的权限,还将获得role doris_rd、doris_qa和doris_pm的权限。
## LDAP验证的局限
- 目前Doris的LDAP功能只支持明文密码验证,即用户登录时,密码在client与fe之间、fe与LDAP服务之间以明文的形式传输。
- 当前的LDAP验证只支持在mysql协议下进行密码验证,如果使用Http接口则无法使用LDAP用户进行验证。
- 临时用户不具有用户属性。

View File

@ -24,4 +24,197 @@ specific language governing permissions and limitations
under the License.
-->
# 多租户和资源划分
# 多租户和资源划分
Doris 的多租户和资源隔离方案,主要目的是为了多用户在同一 Doris 集群内进行数据操作时,减少相互之间的干扰,能够将集群资源更合理的分配给各用户。
该方案主要分为两部分,一是集群内节点级别的资源组划分,二是针对单个查询的资源限制。
## Doris 中的节点
首先先简单介绍一下 Doris 的节点组成。一个 Doris 集群中有两类节点:Frontend(FE) 和 Backend(BE)。
FE 主要负责元数据管理、集群管理、用户请求的接入和查询计划的解析等工作。
BE 主要负责数据存储、查询计划的执行等工作。
FE 不参与用户数据的处理计算等工作,因此是一个资源消耗较低的节点。而 BE 负责所有的数据计算、任务处理,属于资源消耗型的节点。因此,本文所介绍的资源划分及资源限制方案,都是针对 BE 节点的。FE 节点因为资源消耗相对较低,并且还可以横向扩展,因此通常无需做资源上的隔离和限制,FE 节点由所有用户共享即可。
## 节点资源划分
节点资源划分,是指将一个 Doris 集群内的 BE 节点设置标签(Tag),标签相同的 BE 节点组成一个资源组(Resource Group)。资源组可以看作是数据存储和计算的一个管理单元。下面我们通过一个具体示例,来介绍资源组的使用方式。
1. 为 BE 节点设置标签
假设当前 Doris 集群有 6 个 BE 节点。分别为 host[1-6]。在初始情况下,所有节点都属于一个默认资源组(Default)。
我们可以使用以下命令将这6个节点划分成3个资源组:group_a、group_b、group_c:
```sql
alter system modify backend "host1:9050" set ("tag.location" = "group_a");
alter system modify backend "host2:9050" set ("tag.location" = "group_a");
alter system modify backend "host3:9050" set ("tag.location" = "group_b");
alter system modify backend "host4:9050" set ("tag.location" = "group_b");
alter system modify backend "host5:9050" set ("tag.location" = "group_c");
alter system modify backend "host6:9050" set ("tag.location" = "group_c");
```
这里我们将 `host[1-2]` 组成资源组 `group_a`,`host[3-4]` 组成资源组 `group_b`,`host[5-6]` 组成资源组 `group_c`。
> 注:一个 BE 只支持设置一个 Tag。
2. 按照资源组分配数据分布
资源组划分好后。我们可以将用户数据的不同副本分布在不同资源组内。假设一张用户表 UserTable。我们希望在3个资源组内各存放一个副本,则可以通过如下建表语句实现:
```sql
create table UserTable
(k1 int, k2 int)
distributed by hash(k1) buckets 1
properties(
"replication_allocation"="tag.location.group_a:1, tag.location.group_b:1, tag.location.group_c:1"
)
```
这样一来,表 UserTable 中的数据,将会以3副本的形式,分别存储在资源组 group_a、group_b、group_c所在的节点中。
下图展示了当前的节点划分和数据分布:
```text
┌────────────────────────────────────────────────────┐
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ host1 │ │ host2 │ │
│ │ ┌─────────────┐ │ │ │ │
│ group_a │ │ replica1 │ │ │ │ │
│ │ └─────────────┘ │ │ │ │
│ │ │ │ │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
├────────────────────────────────────────────────────┤
├────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ host3 │ │ host4 │ │
│ │ │ │ ┌─────────────┐ │ │
│ group_b │ │ │ │ replica2 │ │ │
│ │ │ │ └─────────────┘ │ │
│ │ │ │ │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
├────────────────────────────────────────────────────┤
├────────────────────────────────────────────────────┤
│ │
│ ┌──────────────────┐ ┌──────────────────┐ │
│ │ host5 │ │ host6 │ │
│ │ │ │ ┌─────────────┐ │ │
│ group_c │ │ │ │ replica3 │ │ │
│ │ │ │ └─────────────┘ │ │
│ │ │ │ │ │
│ └──────────────────┘ └──────────────────┘ │
│ │
└────────────────────────────────────────────────────┘
```
3. 使用不同资源组进行数据查询
在前两步执行完成后,我们就可以通过设置用户的资源使用权限,来限制某一用户的查询,只能使用指定资源组中的节点来执行。
比如我们可以通过以下语句,限制 user1 只能使用 `group_a` 资源组中的节点进行数据查询,user2 只能使用 `group_b` 资源组,而 user3 可以同时使用 3 个资源组:
```sql
set property for 'user1' 'resource_tags.location' = 'group_a';
set property for 'user2' 'resource_tags.location' = 'group_b';
set property for 'user3' 'resource_tags.location' = 'group_a, group_b, group_c';
```
设置完成后,user1 在发起对 UserTable 表的查询时,只会访问 `group_a` 资源组内节点上的数据副本,并且查询仅会使用 `group_a` 资源组内的节点计算资源。而 user3 的查询可以使用任意资源组内的副本和计算资源。
这样,我们通过对节点的划分,以及对用户的资源使用限制,实现了不同用户查询上的物理资源隔离。更进一步,我们可以给不同的业务部门创建不同的用户,并限制每个用户使用不同的资源组。以避免不同业务部分之间使用资源干扰。比如集群内有一张业务表需要共享给所有9个业务部门使用,但是希望能够尽量避免不同部门之间的资源抢占。则我们可以为这张表创建3个副本,分别存储在3个资源组中。接下来,我们为9个业务部门创建9个用户,每3个用户限制使用一个资源组。这样,资源的竞争程度就由9降低到了3。
另一方面,针对在线和离线任务的隔离。我们可以利用资源组的方式实现。比如我们可以将节点划分为 Online 和 Offline 两个资源组。表数据依然以3副本的方式存储,其中 2 个副本存放在 Online 资源组,1 个副本存放在 Offline 资源组。Online 资源组主要用于高并发低延迟的在线数据服务,而一些大查询或离线ETL操作,则可以使用 Offline 资源组中的节点执行。从而实现在统一集群内同时提供在线和离线服务的能力。
## 单查询资源限制
前面提到的资源组方法是节点级别的资源隔离和限制。而在资源组内,依然可能发生资源抢占问题。比如前文提到的将3个业务部门安排在同一资源组内。虽然降低了资源竞争程度,但是这3个部门的查询依然有可能相互影响。
因此,除了资源组方案外,Doris 还提供了对单查询的资源限制功能。
目前 Doris 对单查询的资源限制主要分为 CPU 和 内存限制两方面。
1. 内存限制
Doris 可以限制一个查询被允许使用的最大内存开销。以保证集群的内存资源不会被某一个查询全部占用。我们可以通过以下方式设置内存限制:
```sql
# 设置会话变量 exec_mem_limit。则之后该会话内(连接内)的所有查询都使用这个内存限制。
set exec_mem_limit=1G;
# 设置全局变量 exec_mem_limit。则之后所有新会话(新连接)的所有查询都使用这个内存限制。
set global exec_mem_limit=1G;
# 在 SQL 中设置变量 exec_mem_limit。则该变量仅影响这个 SQL。
select /*+ SET_VAR(exec_mem_limit=1G) */ id, name from tbl where xxx;
```
因为 Doris 的查询引擎是基于全内存的 MPP 查询框架。因此当一个查询的内存使用超过限制后,查询会被终止。因此,当一个查询无法在合理的内存限制下运行时,我们就需要通过一些 SQL 优化手段,或者集群扩容的方式来解决了。
2. CPU 限制
用户可以通过以下方式限制查询的 CPU 资源:
```sql
# 设置会话变量 cpu_resource_limit。则之后该会话内(连接内)的所有查询都使用这个CPU限制。
set cpu_resource_limit = 2
# 设置用户的属性 cpu_resource_limit,则所有该用户的查询情况都使用这个CPU限制。该属性的优先级高于会话变量 cpu_resource_limit
set property for 'user1' 'cpu_resource_limit' = '3';
```
`cpu_resource_limit` 的取值是一个相对值,取值越大则能够使用的 CPU 资源越多。但一个查询能使用的CPU上限也取决于表的分区分桶数。原则上,一个查询的最大 CPU 使用量和查询涉及到的 tablet 数量正相关。极端情况下,假设一个查询仅涉及到一个 tablet,则即使 `cpu_resource_limit` 设置一个较大值,也仅能使用 1 个 CPU 资源。
通过内存和CPU的资源限制。我们可以在一个资源组内,将用户的查询进行更细粒度的资源划分。比如我们可以让部分时效性要求不高,但是计算量很大的离线任务使用更少的CPU资源和更多的内存资源。而部分延迟敏感的在线任务,使用更多的CPU资源以及合理的内存资源。
## 最佳实践和向前兼容
Tag 划分和 CPU 限制是 0.15 版本中的新功能。为了保证可以从老版本平滑升级,Doris 做了如下的向前兼容:
1. 每个 BE 节点会有一个默认的 Tag:`"tag.location": "default"`。
2. 通过 `alter system add backend` 语句新增的 BE 节点也会默认设置 Tag:`"tag.location": "default"`。
3. 所有表的副本分布默认修改为:`"tag.location.default:xx`。其中 xx 为原副本数量。
4. 用户依然可以通过 `"replication_num" = "xx"` 在建表语句中指定副本数,这种属性将会自动转换成:`"tag.location.default:xx`。从而保证无需修改原建表语句。
5. 默认情况下,单查询的内存限制为单节点2GB,CPU资源无限制,和原有行为保持一致。且用户的 `resource_tags.location` 属性为空,即默认情况下,用户可以访问任意 Tag 的 BE,和原有行为保持一致。
这里我们给出一个从原集群升级到 0.15 版本后,开始使用资源划分功能的步骤示例:
1. 关闭数据修复与均衡逻辑
因为升级后,BE的默认Tag为 `"tag.location": "default"`,而表的默认副本分布为:`"tag.location.default:xx`。所以如果直接修改 BE 的 Tag,系统会自动检测到副本分布的变化,从而开始数据重分布。这可能会占用部分系统资源。所以我们可以在修改 Tag 前,先关闭数据修复与均衡逻辑,以保证我们在规划资源时,不会有副本重分布的操作。
```sql
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
ADMIN SET FRONTEND CONFIG ("disable_tablet_scheduler" = "true");
```
2. 设置 Tag 和表副本分布
接下来可以通过 `alter system modify backend` 语句进行 BE 的 Tag 设置。以及通过 `alter table` 语句修改表的副本分布策略。示例如下:
```sql
alter system modify backend "host1:9050, 1212:9050" set ("tag.location" = "group_a");
alter table my_table modify partition p1 set ("replication_allocation" = "tag.location.group_a:2");
```
3. 开启数据修复与均衡逻辑
在 Tag 和副本分布都设置完毕后,我们可以开启数据修复与均衡逻辑来触发数据的重分布了。
```sql
ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");
ADMIN SET FRONTEND CONFIG ("disable_tablet_scheduler" = "false");
```
该过程根据涉及到的数据量会持续一段时间。并且会导致部分 colocation table 无法进行 colocation 规划(因为副本在迁移中)。可以通过 `show proc "/cluster_balance/"` 来查看进度。也可以通过 `show proc "/statistic"` 中 `UnhealthyTabletNum` 的数量来判断进度。当 `UnhealthyTabletNum` 降为 0 时,则代表数据重分布完毕。
4. 设置用户的资源标签权限。
等数据重分布完毕后。我们就可以开始设置用户的资源标签权限了。因为默认情况下,用户的 `resource_tags.location` 属性为空,即可以访问任意 Tag 的 BE。所以在前面步骤中,不会影响到已有用户的正常查询。当 `resource_tags.location` 属性非空时,用户将被限制访问指定 Tag 的 BE。
通过以上4步,我们可以较为平滑的在原有集群升级后,使用资源划分功能。

View File

@ -0,0 +1,69 @@
---
{
"title": "ALTER-SQL-BLOCK-RULE",
"language": "zh-CN"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## ALTER-SQL-BLOCK-RULE
### Name
ALTER SQL BLOCK RULE
### Description
修改SQL阻止规则,允许对sql/sqlHash/partition_num/tablet_num/cardinality/global/enable等每一项进行修改。
语法:
```sql
ALTER SQL_BLOCK_RULE rule_name
[PROPERTIES ("key"="value", ...)];
```
说明:
- sql 和 sqlHash 不能同时被设置。这意味着,如果一个rule设置了sql或者sqlHash,则另一个属性将无法被修改;
- sql/sqlHash 和 partition_num/tablet_num/cardinality 不能同时被设置。举个例子,如果一个rule设置了partition_num,那么sql或者sqlHash将无法被修改;
### Example
1. 根据SQL属性进行修改
```sql
ALTER SQL_BLOCK_RULE test_rule PROPERTIES("sql"="select \\* from test_table","enable"="true")
```
2. 如果一个rule设置了partition_num,那么sql或者sqlHash将无法被修改
```sql
ALTER SQL_BLOCK_RULE test_rule2 PROPERTIES("partition_num" = "10","tablet_num"="300","enable"="true")
```
### Keywords
```text
ALTER,SQL_BLOCK_RULE
```
### Best Practice

View File

@ -0,0 +1,101 @@
---
{
"title": "CREATE-SQL-BLOCK-RULE",
"language": "zh-CN"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## CREATE-SQL-BLOCK-RULE
### Name
CREATE SQL BLOCK RULE
### Description
该语句创建SQL阻止规则,该功能仅用于限制查询语句,不会限制explian语句的执行。
支持按用户配置SQL黑名单:
- 通过正则匹配的方式拒绝指定SQL
- 通过设置partition_num, tablet_num, cardinality, 检查一个查询是否达到其中一个限制
- partition_num, tablet_num, cardinality 可以一起设置,一旦一个查询达到其中一个限制,查询将会被拦截
语法:
```sql
CREATE SQL_BLOCK_RULE rule_name
[PROPERTIES ("key"="value", ...)];
```
参数说明:
- sql:匹配规则(基于正则匹配,特殊字符需要转译),可选,默认值为 "NULL"
- sqlHash: sql hash值,用于完全匹配,我们会在`fe.audit.log`打印这个值,可选,这个参数和sql只能二选一,默认值为 "NULL"
- partition_num: 一个扫描节点会扫描的最大partition数量,默认值为0L
- tablet_num: 一个扫描节点会扫描的最大tablet数量,默认值为0L
- cardinality: 一个扫描节点粗略的扫描行数,默认值为0L
- global:是否全局(所有用户)生效,默认为false
- enable:是否开启阻止规则,默认为true
### Example
1. 创建一个名称为 test_rule 的阻止规则
```sql
mysql> CREATE SQL_BLOCK_RULE test_rule
-> PROPERTIES(
-> "sql"="select * from order_analysis;",
-> "global"="false",
-> "enable"="true"
-> );
Query OK, 0 rows affected (0.01 sec)
```
当我们去执行刚才我们定义在规则里的sql时就会返回异常错误,示例如下:
```sql
mysql> select * from order_analysis;
ERROR 1064 (HY000): errCode = 2, detailMessage = sql match regex sql block rule: order_analysis_rule
```
2. 创建 test_rule2,将最大扫描的分区数量限制在30个,最大扫描基数限制在100亿行,示例如下:
```sql
mysql> CREATE SQL_BLOCK_RULE test_rule2
-> PROPERTIES (
-> "partition_num" = "30",
-> "cardinality" = "10000000000",
-> "global" = "false",
-> "enable" = "true"
-> );
Query OK, 0 rows affected (0.01 sec)
```
### Keywords
```text
CREATE, SQL_BLCOK_RULE
```
### Best Practice

View File

@ -0,0 +1,59 @@
---
{
"title": "DROP-SQL-BLOCK-RULE",
"language": "zh-CN"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## DROP-SQL-BLOCK-RULE
### Name
DROP SQL BLOCK RULE
### Description
删除SQL阻止规则,支持多规则,以,隔开
语法:
```sql
DROP SQL_BLOCK_RULE test_rule1,...
```
### Example
1. 删除test_rule1、test_rule2阻止规则
```sql
mysql> DROP SQL_BLOCK_RULE test_rule1,test_rule2;
Query OK, 0 rows affected (0.00 sec)
```
### Keywords
```text
DROP, SQL_BLOCK_RULE
```
### Best Practice

View File

@ -1,7 +1,9 @@
---
{
"title": "SHOW-EXPORT",
"language": "zh-CN"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one

View File

@ -0,0 +1,77 @@
---
{
"title": "SHOW-SQL-BLOCK-RULE",
"language": "zh-CN"
}
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## SHOW-SQL-BLOCK-RULE
### Name
SHOW SQL BLOCK RULE
### Description
查看已配置的SQL阻止规则,不指定规则名则为查看所有规则。
语法:
```sql
SHOW SQL_BLOCK_RULE [FOR RULE_NAME];
```
### Example
1. 查看所有规则。
```sql
mysql> SHOW SQL_BLOCK_RULE;
+------------+------------------------+---------+--------------+-----------+-------------+--------+--------+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
+------------+------------------------+---------+--------------+-----------+-------------+--------+--------+
| test_rule | select * from order_analysis | NULL | 0 | 0 | 0 | true | true |
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
+------------+------------------------+---------+--------------+-----------+-------------+--------+--------+
2 rows in set (0.01 sec)
```
2. 制定规则名查询
```sql
mysql> SHOW SQL_BLOCK_RULE FOR test_rule2;
+------------+------+---------+--------------+-----------+-------------+--------+--------+
| Name | Sql | SqlHash | PartitionNum | TabletNum | Cardinality | Global | Enable |
+------------+------+---------+--------------+-----------+-------------+--------+--------+
| test_rule2 | NULL | NULL | 30 | 0 | 10000000000 | false | true |
+------------+------+---------+--------------+-----------+-------------+--------+--------+
1 row in set (0.00 sec)
```
### Keywords
SHOW, SQL_BLOCK_RULE
### Best Practice