database/doris
2
0
Fork 0
Code Issues Pull Requests Actions 3 Packages Projects Releases Wiki Activity

[community] modify the doc of verifying apache release (#8084)

Browse Source
This commit is contained in:
Mingyu Chen
2022-02-17 10:53:31 +08:00
committed by GitHub
parent d9535c29f6
commit 486a0586ac
6 changed files with 57 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files

30
docs/en/community/verify-apache-release.md
View File

@ -30,7 +30,7 @@ To verify the release, following checklist can used to reference:
1. [ ] Download links are valid.
2. [ ] Checksums and PGP signatures are valid.
3. [ ] DISCLAIMER-WIP is included.
3. [ ] DISCLAIMER or DISCLAIMER-WIP is included.
4. [ ] Source code artifacts have correct names matching the current release.
5. [ ] LICENSE and NOTICE files are correct for the repository.
6. [ ] All files have license headers if necessary.
@ -55,6 +55,8 @@ wget https://www.apache.org/dist/incubator/doris/KEYS
GnuPG is recommended, which can install by yum install gnupg or apt-get install gnupg.
Here we use Doris main code release as an example. Other releases are similar.
``` shell
gpg --import KEYS
gpg --verify apache-doris-a.b.c-incubating-src.tar.gz.asc apache-doris-a.b.c-incubating-src.tar.gz
@ -63,20 +65,28 @@ sha512sum --check apache-doris-a.b.c-incubating-src.tar.gz.sha512
## 3. Verify license header
Apache RAT is recommended to verify license header, which can download as following command.
Here we use [apache/skywalking-eyes](https://github.com/apache/skywalking-eyes) for source license header validation.
``` shell
wget http://mirrors.tuna.tsinghua.edu.cn/apache/creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz
tar zxvf apache-rat-0.13-bin.tar.gz
Go to the source directory and execute the following command (requires a Docker environment).
```
docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header check
```
Given your source dir is apache-doris-a.b.c-incubating-src, you can check with following command.
It will output a file list which don't include ASF license header, and these files used other licenses.
The output is similar to the following:
``` shell
/usr/java/jdk/bin/java -jar apache-rat-0.13/apache-rat-0.13.jar -a -d apache-doris-a.b.c-incubating-src -E apache-doris-a.b.c-incubating-src/.rat-excludes
```
INFO GITHUB_TOKEN is not set, license-eye won't comment on the pull request
INFO Loading configuration from file: .licenserc.yaml
INFO Totally checked 5611 files, valid: 3926, invalid: 0, ignored: 1685, fixed: 0
```
where an invalid of 0 means the check passed.
> Some non-Apache License header files are documented in `.licenserc.yaml`.
## 4. Verify building
To compile the Doris, please read [Compilation](../installing/compilation.html)
* For Doris main code compilation, see [compilation documentation](../installing/compilation.html)
* Flink Doris Connector compilation, see [compilation documentation](../extending-doris/flink-doris-connector.md)
* Spark Doris Connector compilation, see [compilation documentation](../extending-doris/spark-doris-connector.md)

21
docs/en/extending-doris/flink-doris-connector.md
View File

@ -28,14 +28,14 @@ under the License.
- The Flink Doris Connector can support operations (read, insert, modify, delete) data stored in Doris through Flink.
Github: https://github.com/apache/incubator-doris-connectors
Github: https://github.com/apache/incubator-doris-flink-connector
* `Doris` table can be mapped to `DataStream` or `Table`.
* `Doris` table can be mapped to `DataStream` or `Table`.
>**Note:**
>
>1. Modification and deletion are only supported on the Unique Key model
>2. The current deletion is to support Flink CDC to access data to achieve automatic deletion. If it is to delete other data access methods, you need to implement it yourself. For the data deletion usage of Flink CDC, please refer to the last section of this document
>**Note:**
>
>1. Modification and deletion are only supported on the Unique Key model
>2. The current deletion is to support Flink CDC to access data to achieve automatic deletion. If it is to delete other data access methods, you need to implement it yourself. For the data deletion usage of Flink CDC, please refer to the last section of this document
## Version Compatibility
@ -63,18 +63,13 @@ Just change the `flink.version` here to be the same as your Flink cluster versio
## Build and Install
Execute following command in dir `extension/flink-doris-connector/`:
**Notice:**
1. If you have not compiled the doris source code as a whole, you need to compile the Doris source code first, otherwise the thrift command will not be found, and you need to execute `sh build.sh` in the `incubator-doris` directory.
2. It is recommended to compile under the docker compile environment `apache/incubator-doris:build-env-1.2` of doris, because the JDK version below 1.3 is 11, there will be compilation problems.
Execute following command in source dir:
```bash
sh build.sh 1.11.6 2.12 # flink 1.11.6 scala 2.12
```
After successful compilation, the file `doris-flink-1.0.0-SNAPSHOT.jar` will be generated in the `output/` directory. Copy this file to `ClassPath` in `Flink` to use `Flink-Doris-Connector`. For example, `Flink` running in `Local` mode, put this file in the `jars/` folder. `Flink` running in `Yarn` cluster mode, put this file in the pre-deployment package.
After successful compilation, the file `doris-flink-1.11.6-2.12-1.0.0-SNAPSHOT.jar` will be generated in the `output/` directory. Copy this file to `ClassPath` in `Flink` to use `Flink-Doris-Connector`. For example, `Flink` running in `Local` mode, put this file in the `jars/` folder. `Flink` running in `Yarn` cluster mode, put this file in the pre-deployment package.
**Remarks:**

13
docs/en/extending-doris/spark-doris-connector.md
View File

@ -28,7 +28,7 @@ under the License.
Spark Doris Connector can support reading data stored in Doris and writing data to Doris through Spark.
Github: https://github.com/apache/incubator-doris-connectors
Github: https://github.com/apache/incubator-doris-spark-connector
- Support reading data from `Doris`.
- Support `Spark DataFrame` batch/stream writing data to `Doris`
@ -42,23 +42,16 @@ Github: https://github.com/apache/incubator-doris-connectors
| 2.3.4-2.11.xx | 2.x | 0.12+ | 8 | 2.11 |
| 3.1.2-2.12.xx | 3.x | 0.12.+ | 8 | 2.12 |
## Build and Install
Execute following command in dir `extension/spark-doris-connector/`:
**Notice:**
1. If you have not compiled the doris source code as a whole, you need to compile the Doris source code first, otherwise the thrift command will not be found, and you need to execute `sh build.sh` in the `incubator-doris` directory.
2. It is recommended to compile under the docker compile environment `apache/incubator-doris:build-env-1.2` of doris, because the JDK version below 1.3 is 11, there will be compilation problems.
Execute following command in source dir
```bash
sh build.sh 2.3.4 2.11 ## spark 2.3.4 version, and scala 2.11
sh build.sh 3.1.2 2.12 ## spark 3.1.2 version, and scala 2.12
```
After successful compilation, the file `doris-spark-1.0.0-SNAPSHOT.jar` will be generated in the `output/` directory. Copy this file to `ClassPath` in `Spark` to use `Spark-Doris-Connector`. For example, `Spark` running in `Local` mode, put this file in the `jars/` folder. `Spark` running in `Yarn` cluster mode, put this file in the pre-deployment package.
After successful compilation, the file `doris-spark-2.3.4-2.11-1.0.0-SNAPSHOT.jar` will be generated in the `output/` directory. Copy this file to `ClassPath` in `Spark` to use `Spark-Doris-Connector`. For example, `Spark` running in `Local` mode, put this file in the `jars/` folder. `Spark` running in `Yarn` cluster mode, put this file in the pre-deployment package.
## Using Maven

30
docs/zh-CN/community/verify-apache-release.md
View File

@ -30,7 +30,7 @@ under the License.
1. [ ] 下载链接是否合法。
2. [ ] 校验值和 PGP 签名是否合法。
3. [ ] 是否包含 DISCLAIMER-WIP。
3. [ ] 是否包含 DISCLAIMER 或 DISCLAIMER-WIP 文件。
4. [ ] 代码是否和当前发布版本相匹配。
5. [ ] LICENSE 和 NOTICE 文件是否正确。
6. [ ] 所有文件都携带必要的协议说明。
@ -60,6 +60,8 @@ CentOS: yum install gnupg
Ubuntu: apt-get install gnupg
```
这里以 Doris 主代码 release 为例。其他 release 类似。
``` shell
gpg --import KEYS
gpg --verify apache-doris-a.b.c-incubating-src.tar.gz.asc apache-doris-a.b.c-incubating-src.tar.gz
@ -68,20 +70,28 @@ sha512sum --check apache-doris-a.b.c-incubating-src.tar.gz.sha512
## 3. 验证源码协议头
推荐使用 Apache RAT 验证源码协议，可以从以下链接下载：
这里我们使用 [apache/skywalking-eyes](https://github.com/apache/skywalking-eyes) 进行源码协议头验证。
``` shell
wget http://mirrors.tuna.tsinghua.edu.cn/apache/creadur/apache-rat-0.13/apache-rat-0.13-bin.tar.gz
tar zxvf apache-rat-0.13-bin.tar.gz
进入源码目录，执行以下命令（需要 Docker 环境）：
```
docker run -it --rm -v $(pwd):/github/workspace apache/skywalking-eyes header check
```
假设源码目录名称为 apache-doris-a.b.c-incubating-src，可以使用以下命令进行验证。
这个命令会产生一个文件，其中列举了所有非 ASF 协议的文件。
输出类似如下结果:
``` shell
/usr/java/jdk/bin/java -jar apache-rat-0.13/apache-rat-0.13.jar -a -d apache-doris-a.b.c-incubating-src -E apache-doris-a.b.c-incubating-src/.rat-excludes
```
INFO GITHUB_TOKEN is not set, license-eye won't comment on the pull request
INFO Loading configuration from file: .licenserc.yaml
INFO Totally checked 5611 files, valid: 3926, invalid: 0, ignored: 1685, fixed: 0
```
其中 invalid 为 0 表示检查通过。
> 部分非 Apache License 协议头的文件记录在 `.licenserc.yaml` 中。
## 4. 验证编译
详细的编译步骤，请参阅 [编译文档](../installing/compilation.html)
* Doris 主代码编译，请参阅 [编译文档](../installing/compilation.html)
* Flink Doris Connector 编译，请参阅 [编译文档](../extending-doris/flink-doris-connector.md)
* Spark Doris Connector 编译，请参阅 [编译文档](../extending-doris/spark-doris-connector.md)

13
docs/zh-CN/extending-doris/flink-doris-connector.md
View File

@ -30,7 +30,7 @@ under the License.
Flink Doris Connector 可以支持通过 Flink 操作（读取、插入、修改、删除） Doris 中存储的数据。
代码库地址：https://github.com/apache/incubator-doris-connectors
代码库地址：https://github.com/apache/incubator-doris-flink-connector
* 可以将`Doris`表映射为`DataStream`或者`Table`。
@ -65,18 +65,13 @@ Flink Doris Connector 可以支持通过 Flink 操作（读取、插入、修改
## 编译与安装
在 `extension/flink-doris-connector/` 源码目录下执行：
**注意**
1. 这里如果你没有整体编译过 doris 源码，需要首先编译一次 Doris 源码，不然会出现 thrift 命令找不到的情况，需要到 `incubator-doris` 目录下执行 `sh build.sh`
2. 建议在 doris 的 docker 编译环境 `apache/incubator-doris:build-env-1.2` 下进行编译，因为 1.3 下面的JDK 版本是 11，会存在编译问题。
在源码目录下执行：
```bash
sh build.sh 1.11.6 2.12 # flink 1.11.6 scala 2.12
```
编译成功后，会在 `output/` 目录下生成文件 `doris-flink-1.0.0-SNAPSHOT.jar` 。将此文件复制到 `Flink` 的 `ClassPath` 中即可使用 `Flink-Doris-Connector` 。例如， `Local` 模式运行的 `Flink` ，将此文件放入 `jars/` 文件夹下。 `Yarn` 集群模式运行的 `Flink` ，则将此文件放入预部署包中。
编译成功后，会在 `output/` 目录下生成文件 `doris-flink-1.11.6-2.12-1.0.0-SNAPSHOT.jar` 。将此文件复制到 `Flink` 的 `ClassPath` 中即可使用 `Flink-Doris-Connector` 。例如， `Local` 模式运行的 `Flink` ，将此文件放入 `jars/` 文件夹下。 `Yarn` 集群模式运行的 `Flink` ，则将此文件放入预部署包中。
**备注**
@ -105,8 +100,6 @@ enable_http_server_v2 = true
`1.11.6 ` 可以根据flink 版本替换成替换成 `1.12.7` 或者 `1.13.5`
## 使用方法
Flink读写Doris数据主要有三种方式

12
docs/zh-CN/extending-doris/spark-doris-connector.md
View File

@ -28,7 +28,7 @@ under the License.
Spark Doris Connector 可以支持通过 Spark 读取 Doris 中存储的数据，也支持通过Spark写入数据到Doris。
代码库地址：https://github.com/apache/incubator-doris-connectors
代码库地址：https://github.com/apache/incubator-doris-spark-connector
- 支持从`Doris`中读取数据
- 支持`Spark DataFrame`批量/流式 写入`Doris`
@ -42,15 +42,9 @@ Spark Doris Connector 可以支持通过 Spark 读取 Doris 中存储的数据
| 2.3.4-2.11.xx | 2.x | 0.12+ | 8 | 2.11 |
| 3.1.2-2.12.xx | 3.x | 0.12.+ | 8 | 2.12 |
## 编译与安装
在 `extension/spark-doris-connector/` 源码目录下执行：
**注意：**
1. 这里如果你没有整体编译过 doris 源码，需要首先编译一次 Doris 源码，不然会出现 thrift 命令找不到的情况，需要到 `incubator-doris` 目录下执行 `sh build.sh`
2. 建议在 doris 的 docker 编译环境 `apache/incubator-doris:build-env-1.2` 下进行编译，因为 1.3 下面的JDK 版本是 11，会存在编译问题。
在源码目录下执行：
```bash
sh build.sh 2.3.4 2.11 ## spark 2.3.4, scala 2.11
@ -58,7 +52,7 @@ sh build.sh 3.1.2 2.12 ## spark 3.1.2, scala 2.12
```
编译成功后，会在 `output/` 目录下生成文件 `doris-spark-1.0.0-SNAPSHOT.jar`。将此文件复制到 `Spark` 的 `ClassPath` 中即可使用 `Spark-Doris-Connector`。例如，`Local` 模式运行的 `Spark`，将此文件放入 `jars/` 文件夹下。`Yarn`集群模式运行的`Spark`，则将此文件放入预部署包中。
编译成功后，会在 `output/` 目录下生成文件 `doris-spark-2.3.4-2.11-1.0.0-SNAPSHOT.jar`。将此文件复制到 `Spark` 的 `ClassPath` 中即可使用 `Spark-Doris-Connector`。例如，`Local` 模式运行的 `Spark`，将此文件放入 `jars/` 文件夹下。`Yarn`集群模式运行的`Spark`，则将此文件放入预部署包中。
## 使用Maven管理