From a06baad7d7f41cdcb2cebfbb2af52b3c59783714 Mon Sep 17 00:00:00 2001 From: FreeOnePlus <54164178+FreeOnePlus@users.noreply.github.com> Date: Fri, 10 Feb 2023 14:07:07 +0800 Subject: [PATCH] [docs](docker) Add Run Docker cluster docs (#16520) --- .../construct-docker-compose.md | 30 --- .../construct-docker/run-docker-cluster.md | 215 ++++++++++++++++++ docs/sidebars.json | 2 +- .../construct-docker-compose.md | 30 --- .../construct-docker/run-docker-cluster.md | 215 ++++++++++++++++++ 5 files changed, 431 insertions(+), 61 deletions(-) delete mode 100644 docs/en/docs/install/construct-docker/construct-docker-compose.md create mode 100644 docs/en/docs/install/construct-docker/run-docker-cluster.md delete mode 100644 docs/zh-CN/docs/install/construct-docker/construct-docker-compose.md create mode 100644 docs/zh-CN/docs/install/construct-docker/run-docker-cluster.md diff --git a/docs/en/docs/install/construct-docker/construct-docker-compose.md b/docs/en/docs/install/construct-docker/construct-docker-compose.md deleted file mode 100644 index 2719af733b..0000000000 --- a/docs/en/docs/install/construct-docker/construct-docker-compose.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -{ -"title": "Build Docker Compose", -"language": "en" -} ---- - - - -# Build Docker Compose - -*Todo* - diff --git a/docs/en/docs/install/construct-docker/run-docker-cluster.md b/docs/en/docs/install/construct-docker/run-docker-cluster.md new file mode 100644 index 0000000000..a6ca26aae9 --- /dev/null +++ b/docs/en/docs/install/construct-docker/run-docker-cluster.md @@ -0,0 +1,215 @@ +--- +{ +"title": "Deploy Docker cluster", +"language": "zh-CN" +} +--- + + +# Deploy the Docker cluster + +## Background description + +This article will briefly describe how to quickly build a complete Doris test cluster through `docker run` or `docker-compose up` commands. + +## Applicable scene + +It is recommended to use Doris Docker in SIT or DEV environment to simplify the deployment process. + +If you want to test a certain function point in the new version, you can use Doris Docker to deploy a Playground environment. Or when you want to reproduce a certain problem during debugging, you can also use the docker environment to simulate. + +In the production environment, currently try to avoid using containerized solutions for Doris deployment. + +## Software Environment + +| Software | Version | +| -------------- | ----------- | +| Docker | 20.0 and above | +| docker-compose | 2.10 and above | + +## Hardware environment + +| Configuration Type | Hardware Information | Maximum Running Cluster Size | +| -------- | -------- | ---------------- | +| Minimum configuration | 2C 4G | 1FE 1BE | +| Recommended configuration | 4C 16G | 3FE 3BE | + +## Pre-environment preparation + +The following command needs to be executed on the host machine + +```shell +sysctl -w vm.max_map_count=2000000 +``` + +## Docker Compose + +Different platforms need to use different Image images. This article takes the `X86_64` platform as an example. + +### Network Mode Description + +There are two network modes applicable to Doris Docker. + +1. HOST mode suitable for deployment across multiple nodes, this mode is suitable for deploying 1FE 1BE on each node. +2. The subnet bridge mode is suitable for deploying multiple Doris processes on a single node. This mode is suitable for single-node deployment (recommended). If you want to deploy multiple nodes, you need to deploy more components (not recommended). + +For the sake of presentation, this chapter only demonstrates scripts written in subnet bridge mode. + +### Interface Description + +From the version of `Apache Doris 1.2.1 Docker Image`, the interface list of each process image is as follows: + +| process name | interface name | interface definition | interface example | +| -------------- |-------------| ------------------- | -------------------------------------------------------------- | +| FE\BE\BROKER | FE_SERVERS | FE node main information | fe1:172.20.80.2:9010,fe2:172.20.80.3:9010,fe3:172.20.80.4:9010 | +| FE | FE_ID | FE node ID | 1 | +| BE | BE_ADDR | BE node main information | 172.20.80.5:9050 | +| BROKER | BROKER_ADDR | Main information of BROKER node | 172.20.80.6:8000 | + +Note that the above interface must fill in the information, otherwise the process cannot be started. + +> FE_SERVERS interface rules are: `FE_NAME:FE_HOST:FE_EDIT_LOG_PORT[,FE_NAME:FE_HOST:FE_EDIT_LOG_PORT]` +> +> The FE_ID interface rule is: an integer of `1-9`, where the FE number `1` is the Master node. +> +> BE_ADDR interface rule is: `BE_HOST:BE_HEARTBEAT_SERVICE_PORT` +> +> BROKER_ADDR interface rule is: `BROKER_HOST:BROKER_IPC_PORT` + +### Script Template + +#### Docker Run command + +Create a subnet bridge + +``` shell +docker network create --driver bridge --subnet=172.20.80.0/24 doris-network +``` + +1FE & 1BE Command Templates + +```shell +docker run -itd \ +--name=fe \ +--env FE_SERVERS="fe1:172.20.80.2:9010" \ +--env FE_ID=1 \ +-p 8030:8030\ +-p 9030:9030 \ +-v /data/fe/doris-meta:/opt/apache-doris/fe/doris-meta \ +-v /data/fe/conf:/opt/apache-doris/fe/conf \ +-v /data/fe/log:/opt/apache-doris/fe/log \ +--network=doris-network \ +--ip=172.20.80.2\ +apache/doris:1.2.1-fe-x86_64 + +docker run -itd \ +--name=be\ +--env FE_SERVERS="fe1:172.20.80.2:9010" \ +--env BE_ADDR="172.20.80.3:9050" \ +-p 8040:8040 \ +-v /data/be/storage:/opt/apache-doris/be/storage \ +-v /data/be/conf:/opt/apache-doris/be/conf \ +-v /data/be/log:/opt/apache-doris/be/log \ +--network=doris-network \ +--ip=172.20.80.3\ +apache/doris:1.2.1-be-x86_64 +``` + +3FE & 3BE Run command template if needed [click here](https://github.com/apache/doris/tree/master/docker/runtime/docker-compose-demo/build-cluster/rum-command/3fe_3be.sh ) to access Downloads. + +#### Docker Compose script + +1FE & 1BE template + +```yaml +version: '3' +services: + docker-fe: + image: "apache/doris:1.2.1-fe-x86_64" + container_name: "doris-fe" + hostname: "fe" + environment: + - FE_SERVERS=fe1:172.20.80.2:9010 + - FE_ID=1 + ports: + - 8030:8030 + - 9030:9030 + volumes: + - /data/fe/doris-meta:/opt/apache-doris/fe/doris-meta + - /data/fe/conf:/opt/apache-doris/fe/conf + - /data/fe/log:/opt/apache-doris/fe/log + networks: + doris_net: + ipv4_address: 172.20.80.2 + docker-be: + image: "apache/doris:1.2.1-be-x86_64" + container_name: "doris-be" + hostname: "be" + depends_on: + - docker-fe + environment: + - FE_SERVERS=fe1:172.20.80.2:9010 + - BE_ADDR=172.20.80.3:9050 + ports: + - 8040:8040 + volumes: + - /data/be/storage:/opt/apache-doris/be/storage + - /data/be/conf:/opt/apache-doris/be/conf + - /data/be/script:/docker-entrypoint-initdb.d + - /data/be/log:/opt/apache-doris/be/log + networks: + doris_net: + ipv4_address: 172.20.80.3 +networks: + doris_net: + ipam: + config: + - subnet: 172.20.80.0/16 +``` + +3FE & 3BE Docker Compose script template if needed [click here](https://github.com/apache/doris/tree/master/docker/runtime/docker-compose-demo/build-cluster/docker-compose/ 3fe_3be/docker-compose.yaml) access to download. + +## Deploy Doris Docker + +You can choose one of the two deployment methods: + +1. Execute the `docker run` command to create a cluster +2. Save the `docker-compose.yaml` script and execute the `docker-compose up -d` command in the same directory to create a cluster + +### Special case description + +Due to the different ways of implementing containers internally on MacOS, it may not be possible to directly modify the value of `max_map_count` on the host during deployment. You need to create the following containers first: + +```shel +docker run -it --privileged --pid=host --name=change_count debian nsenter -t 1 -m -u -n -i sh +``` + +The container was created successfully executing the following command: + +```shell +sysctl -w vm.max_map_count=2000000 +``` + +Then `exit` exits and creates the Doris Docker cluster. + +## Unfinished business + +1. The Computer Node interface has not yet been adapted, and the next version will adapt +2. Compose Demo List diff --git a/docs/sidebars.json b/docs/sidebars.json index 565f5e46d0..3a36f83e84 100644 --- a/docs/sidebars.json +++ b/docs/sidebars.json @@ -35,7 +35,7 @@ "label": "Construct Docker", "items": [ "install/construct-docker/construct-docker-image", - "install/construct-docker/construct-docker-compose" + "install/construct-docker/run-docker-cluster" ] } ] diff --git a/docs/zh-CN/docs/install/construct-docker/construct-docker-compose.md b/docs/zh-CN/docs/install/construct-docker/construct-docker-compose.md deleted file mode 100644 index 9c6d8d1dbe..0000000000 --- a/docs/zh-CN/docs/install/construct-docker/construct-docker-compose.md +++ /dev/null @@ -1,30 +0,0 @@ ---- -{ -"title": "构建 Docker Compose", -"language": "zh-CN" -} ---- - - - -# 构建 Docker Compose - -*Todo* - diff --git a/docs/zh-CN/docs/install/construct-docker/run-docker-cluster.md b/docs/zh-CN/docs/install/construct-docker/run-docker-cluster.md new file mode 100644 index 0000000000..963f91a94f --- /dev/null +++ b/docs/zh-CN/docs/install/construct-docker/run-docker-cluster.md @@ -0,0 +1,215 @@ +--- +{ +"title": "部署 Docker 集群", +"language": "zh-CN" +} +--- + + +# 部署 Docker 集群 + +## 背景说明 + +本篇将简述如何通过 `docker run` 或 `docker-compose up` 命令快速构建一套完整的 Doris 测试集群。 + +## 适用场景 + +建议在 SIT 或者 DEV 环境中使用 Doris Docker 来简化部署的流程。 + +如在新版本中想测试某一个功能点，可以使用 Doris Docker 部署一个 Playground 环境。或者在调试的过程中要复现某个问题时，也可以使用 docker 环境来模拟。 + +在生产环境上，当前暂时尽量避免使用容器化的方案进行 Doris 部署。 + +## 软件环境 + +| 软件 | 版本 | +| -------------- | ----------- | +| Docker | 20.0 及以上 | +| docker-compose | 2.10 及以上 | + +## 硬件环境 + +| 配置类型 | 硬件信息 | 最大运行集群规模 | +| -------- | -------- | ---------------- | +| 最低配置 | 2C 4G | 1FE 1BE | +| 推荐配置 | 4C 16G | 3FE 3BE | + +## 前期环境准备 + +需在宿主机执行如下命令 + +```shell +sysctl -w vm.max_map_count=2000000 +``` + +## Docker Compose + +不同平台需要使用不同 Image 镜像，本篇以 `X86_64` 平台为例。 + +### 网络模式说明 + +Doris Docker 适用的网络模式有两种。 + +1. 适合跨多节点部署的 HOST 模式，这种模式适合每个节点部署 1FE 1BE。 +2. 适合单节点部署多 Doris 进程的子网网桥模式，这种模式适合单节点部署（推荐），若要多节点混部需要做更多组件部署（不推荐）。 + +为便于展示，本章节仅演示子网网桥模式编写的脚本。 + +### 接口说明 + +从 `Apache Doris 1.2.1 Docker Image` 版本起，各个进程镜像接口列表如下： + +| 进程名 | 接口名 | 接口定义 | 接口示例 | +| -------------- | ----------- | ------------------- | ------------------------------------------------------------ | +| FE\|BE\|BROKER | FE_SERVERS | FE 节点主要信息 | fe1:172.20.80.2:9010,fe2:172.20.80.3:9010,fe3:172.20.80.4:9010 | +| FE | FE_ID | FE 节点 ID | 1 | +| BE | BE_ADDR | BE 节点主要信息 | 172.20.80.5:9050 | +| BROKER | BROKER_ADDR | BROKER 节点主要信息 | 172.20.80.6:8000 | + +注意，以上接口必须填写信息，否则进程无法启动。 + +> FE_SERVERS 接口规则为：`FE_NAME:FE_HOST:FE_EDIT_LOG_PORT[,FE_NAME:FE_HOST:FE_EDIT_LOG_PORT]` +> +> FE_ID 接口规则为：`1-9` 的整数，其中 `1` 号 FE 为 Master 节点。 +> +> BE_ADDR 接口规则为：`BE_HOST:BE_HEARTBEAT_SERVICE_PORT` +> +> BROKER_ADDR 接口规则为：`BROKER_HOST:BROKER_IPC_PORT` + +### 脚本模板 + +#### Docker Run 命令 + +创建子网网桥 + +``` shell +docker network create --driver bridge --subnet=172.20.80.0/24 doris-network +``` + +1FE & 1BE 命令模板 + +```shell +docker run -itd \ +--name=fe \ +--env FE_SERVERS="fe1:172.20.80.2:9010" \ +--env FE_ID=1 \ +-p 8030:8030 \ +-p 9030:9030 \ +-v /data/fe/doris-meta:/opt/apache-doris/fe/doris-meta \ +-v /data/fe/conf:/opt/apache-doris/fe/conf \ +-v /data/fe/log:/opt/apache-doris/fe/log \ +--network=doris-network \ +--ip=172.20.80.2 \ +apache/doris:1.2.1-fe-x86_64 + +docker run -itd \ +--name=be \ +--env FE_SERVERS="fe1:172.20.80.2:9010" \ +--env BE_ADDR="172.20.80.3:9050" \ +-p 8040:8040 \ +-v /data/be/storage:/opt/apache-doris/be/storage \ +-v /data/be/conf:/opt/apache-doris/be/conf \ +-v /data/be/log:/opt/apache-doris/be/log \ +--network=doris-network \ +--ip=172.20.80.3 \ +apache/doris:1.2.1-be-x86_64 +``` + +3FE & 3BE Run 命令模板如有需要[点击此处](https://github.com/apache/doris/tree/master/docker/runtime/docker-compose-demo/build-cluster/rum-command/3fe_3be.sh)访问下载。 + +#### Docker Compose 脚本 + +1FE & 1BE 模板 + +``` yaml +version: '3' +services: + docker-fe: + image: "apache/doris:1.2.1-fe-x86_64" + container_name: "doris-fe" + hostname: "fe" + environment: + - FE_SERVERS=fe1:172.20.80.2:9010 + - FE_ID=1 + ports: + - 8030:8030 + - 9030:9030 + volumes: + - /data/fe/doris-meta:/opt/apache-doris/fe/doris-meta + - /data/fe/conf:/opt/apache-doris/fe/conf + - /data/fe/log:/opt/apache-doris/fe/log + networks: + doris_net: + ipv4_address: 172.20.80.2 + docker-be: + image: "apache/doris:1.2.1-be-x86_64" + container_name: "doris-be" + hostname: "be" + depends_on: + - docker-fe + environment: + - FE_SERVERS=fe1:172.20.80.2:9010 + - BE_ADDR=172.20.80.3:9050 + ports: + - 8040:8040 + volumes: + - /data/be/storage:/opt/apache-doris/be/storage + - /data/be/conf:/opt/apache-doris/be/conf + - /data/be/script:/docker-entrypoint-initdb.d + - /data/be/log:/opt/apache-doris/be/log + networks: + doris_net: + ipv4_address: 172.20.80.3 +networks: + doris_net: + ipam: + config: + - subnet: 172.20.80.0/16 +``` + +3FE & 3BE Docker Compose 脚本模板如有需要[点击此处](https://github.com/apache/doris/tree/master/docker/runtime/docker-compose-demo/build-cluster/docker-compose/3fe_3be/docker-compose.yaml)访问下载。 + +## 部署 Doris Docker + +部署方式二选一即可： + +1. 执行 `docker run` 命令创建集群 +2. 保存 `docker-compose.yaml` 脚本，同目录下执行 `docker-compose up -d` 命令创建集群 + +### 特例说明 + +MacOS 由于内部实现容器的方式不同，在部署时宿主机直接修改 `max_map_count` 值可能无法成功，需要先创建以下容器： + +```shel +docker run -it --privileged --pid=host --name=change_count debian nsenter -t 1 -m -u -n -i sh +``` + +容器创建成功执行以下命令： + +```shell +sysctl -w vm.max_map_count=2000000 +``` + +然后 `exit` 退出，创建 Doris Docker 集群。 + +## 未尽事项 + +1. Computer Node 接口尚未适配，下个版本适配 +2. Compose Demo List