2d39cffa5c
In order to cooperate with Doris's successful graduation from Apache, the Doris official website also needs a new look and more powerful feature, so we decided to redesign the Doris official website. The code and documents of the new official website are included in this PR. Since the new website is completely rewritten, the content and structure of the project are different from the previous one. In particular, the directory structure of documents has changed, and the number of documents is large, so the number of files in this PR is very large. In the old website，all English documents are in the en/ directory, and Chinese documents in the zh-CN/ directory, but in the new website，the documents are split into multiple directories according to the nav. The document's directory structure changes as follows： ``` docs (old website) | |—— .vuepress (library) | |—— en | | |—— admin-manual │ │ |—— advanced | | |—— article | | |—— benchmark | | |—— case-user | | |—— community | | |—— data-operate | | |—— data-table | | |—— design | | |—— developer-guide | | |—— downloads | | |—— ecosystem | | |—— faq | | |—— get-starting | | |—— install | | |—— sql-manual | | |—— summary | | |___ README.md | |—— zh-CN ... docs (new website) | |—— .vuepress (library) | |—— en | | |—— community (unchanged, community nav) │ │ |—— developer (new directory, developer nav) │ │ | |—— design (moved from en/design) │ │ | |__ developer-guide (moved from en/developer-guide) | | |—— docs (new directory, all children directories moved from en/, document nav) │ │ | |—— admin-manual │ │ | |—— advanced │ │ | |—— benchmark │ │ | |—— data-operate │ │ | |—— data-table │ │ | |—— ecosystem │ │ | |—— faq │ │ | |—— get-starting │ │ | |—— install │ │ | |—— sql-manual │ │ | |—— summary | | |—— downloads (unchanged, downloads nav) | | |—— userCase (moved from en/case-user, user nav) | | |___ README.md | |—— zh-CN ... ```
592 lines
21 KiB
Markdown
592 lines
21 KiB
Markdown
---
|
|
{
|
|
"title": "回归测试",
|
|
"language": "zh-CN"
|
|
}
|
|
|
|
---
|
|
|
|
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one
|
|
or more contributor license agreements. See the NOTICE file
|
|
distributed with this work for additional information
|
|
regarding copyright ownership. The ASF licenses this file
|
|
to you under the Apache License, Version 2.0 (the
|
|
"License"); you may not use this file except in compliance
|
|
with the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing,
|
|
software distributed under the License is distributed on an
|
|
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
|
KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations
|
|
under the License.
|
|
-->
|
|
|
|
# 回归测试
|
|
|
|
## 概念
|
|
1. `Suite`: 一个测试用例，目前仅用来指代测试用例文件名
|
|
2. `Group`: 一个测试集，目前仅用于指代测试用例所属的目录
|
|
3. `Action`: 一个封装好的具体测试行为，比如用于执行sql的`sql` Action，用于校验结果的`test` Action，用于导入数据的`streamLoad` Action等
|
|
|
|
## 测试步骤
|
|
1. 需要预先安装好集群
|
|
2. 修改配置文件`${DORIS_HOME}/regression-test/conf/regression-conf.groovy`，设置jdbc url、用户等配置项
|
|
3. 创建测试用例文件并编写用例
|
|
4. 如果用例文件包含`qt` Action，则需要创建关联的data文件，比如`suites/demo/qt_action.groovy`这个例子，需要用到`data/demo/qt_action.out`这个TSV文件来校验输出是否一致
|
|
5. 运行`${DORIS_HOME}/run-regression-test.sh`测试全部用例,或运行`${DORIS_HOME}/run-regression-test.sh --run <suiteName>` 测试若干用例，更多例子见"启动脚本例子"章节
|
|
|
|
## 目录结构
|
|
开发时需要关注的重要文件/目录
|
|
1. `run-regression-test.sh`: 启动脚本
|
|
2. `regression-conf.groovy`: 回归测试的默认配置
|
|
3. `data`: 存放输入数据和输出校验数据
|
|
4. `suites`: 存放用例
|
|
|
|
```
|
|
./${DORIS_HOME}
|
|
|-- **run-regression-test.sh** 回归测试启动脚本
|
|
|-- regression-test
|
|
| |-- plugins 插件目录
|
|
| |-- conf
|
|
| | |-- logback.xml 日志配置文件
|
|
| | |-- **regression-conf.groovy** 默认配置文件
|
|
| |
|
|
| |-- framework 回归测试框架源码
|
|
| |-- **data** 用例的输入输出文件
|
|
| | |-- demo 存放demo的输入输出文件
|
|
| | |-- correctness 存放正确性测试用例的输入输出文件
|
|
| | |-- performance 存放性能测试用例的输入输出文件
|
|
| | |-- utils 存放其他工具的输入输出文件
|
|
| |
|
|
| |-- **suites** 回归测试用例
|
|
| |-- demo 存放测试用例的demo
|
|
| |-- correctness 存放正确性测试用例
|
|
| |-- performance 存放性能测试用例
|
|
| |-- utils 其他工具
|
|
|
|
|
|-- output
|
|
|-- regression-test
|
|
|-- log 回归测试日志
|
|
```
|
|
|
|
|
|
## 框架默认配置
|
|
测试时需要实际情况修改jdbc和fe的配置
|
|
```groovy
|
|
|
|
/* ============ 一般只需要关注下面这部分 ============ */
|
|
// 默认DB,如果未创建,则会尝试创建这个db
|
|
defaultDb = "regression_test"
|
|
|
|
// Jdbc配置
|
|
jdbcUrl = "jdbc:mysql://127.0.0.1:9030/?"
|
|
jdbcUser = "root"
|
|
jdbcPassword = ""
|
|
|
|
// fe地址配置, 用于stream load
|
|
feHttpAddress = "127.0.0.1:8030"
|
|
feHttpUser = "root"
|
|
feHttpPassword = ""
|
|
|
|
/* ============ 一般不需要修改下面的部分 ============ */
|
|
|
|
// DORIS_HOME变量是通过run-regression-test.sh传入的
|
|
// 即 java -DDORIS_HOME=./
|
|
|
|
// 设置回归测试用例的目录
|
|
suitePath = "${DORIS_HOME}/regression-test/suites"
|
|
// 设置输入输出数据的目录
|
|
dataPath = "${DORIS_HOME}/regression-test/data"
|
|
// 设置插件的目录
|
|
pluginPath = "${DORIS_HOME}/regression-test/plugins"
|
|
|
|
// 默认会读所有的组,读多个组可以用半角逗号隔开，如: "demo,performance"
|
|
// 一般不需要在配置文件中修改，而是通过run-regression-test.sh --run -g来动态指定和覆盖
|
|
testGroups = ""
|
|
// 默认会读所有的用例, 同样可以使用run-regression-test.sh --run -s来动态指定和覆盖
|
|
testSuites = ""
|
|
// 默认会加载的用例目录, 可以通过run-regression-test.sh --run -d来动态指定和覆盖
|
|
testDirectories = ""
|
|
|
|
// 排除这些组的用例，可通过run-regression-test.sh --run -xg来动态指定和覆盖
|
|
excludeGroups = ""
|
|
// 排除这些suite，可通过run-regression-test.sh --run -xs来动态指定和覆盖
|
|
excludeSuites = ""
|
|
// 排除这些目录，可通过run-regression-test.sh --run -xd来动态指定和覆盖
|
|
excludeDirectories = ""
|
|
|
|
// 其他自定义配置
|
|
customConf1 = "test_custom_conf_value"
|
|
```
|
|
|
|
## 编写用例的步骤
|
|
1. 进入`${DORIS_HOME}/regression-test`目录
|
|
2. 根据测试的目的来选择用例的目录，正确性测试存在`suites/correctness`，而性能测试存在`suites/performance`
|
|
3. 新建一个groovy用例文件，增加若干`Action`用于测试，Action将在后续章节具体说明
|
|
|
|
## Action
|
|
Action是一个测试框架默认提供的测试行为，使用DSL来定义。
|
|
|
|
### sql action
|
|
sql action用于提交sql并获取结果，如果查询失败则会抛出异常
|
|
|
|
参数如下
|
|
- String sql: 输入的sql字符串
|
|
- return List<List<Object>>: 查询结果，如果是DDL/DML，则返回一行一列，唯一的值是updateRowCount
|
|
|
|
下面的样例代码存放于`${DORIS_HOME}/regression-test/suites/demo/sql_action.groovy`:
|
|
```groovy
|
|
suite("sql_action", "demo") {
|
|
// execute sql and ignore result
|
|
sql "show databases"
|
|
|
|
// execute sql and get result, outer List denote rows, inner List denote columns in a single row
|
|
List<List<Object>> tables = sql "show tables"
|
|
|
|
// assertXxx() will invoke junit5's Assertions.assertXxx() dynamically
|
|
assertTrue(tables.size() >= 0) // test rowCount >= 0
|
|
|
|
// syntax error
|
|
try {
|
|
sql "a b c d e"
|
|
throw new IllegalStateException("Should be syntax error")
|
|
} catch (java.sql.SQLException t) {
|
|
assertTrue(true)
|
|
}
|
|
|
|
def testTable = "test_sql_action1"
|
|
|
|
try {
|
|
sql "DROP TABLE IF EXISTS ${testTable}"
|
|
|
|
// multi-line sql
|
|
def result1 = sql """
|
|
CREATE TABLE IF NOT EXISTS ${testTable} (
|
|
id int
|
|
)
|
|
DISTRIBUTED BY HASH(id) BUCKETS 1
|
|
PROPERTIES (
|
|
"replication_num" = "1"
|
|
)
|
|
"""
|
|
|
|
// DDL/DML return 1 row and 1 column, the only value is update row count
|
|
assertTrue(result1.size() == 1)
|
|
assertTrue(result1[0].size() == 1)
|
|
assertTrue(result1[0][0] == 0, "Create table should update 0 rows")
|
|
|
|
def result2 = sql "INSERT INTO test_sql_action1 values(1), (2), (3)"
|
|
assertTrue(result2.size() == 1)
|
|
assertTrue(result2[0].size() == 1)
|
|
assertTrue(result2[0][0] == 3, "Insert should update 3 rows")
|
|
} finally {
|
|
/**
|
|
* try_xxx(args) means:
|
|
*
|
|
* try {
|
|
* return xxx(args)
|
|
* } catch (Throwable t) {
|
|
* // do nothing
|
|
* return null
|
|
* }
|
|
*/
|
|
try_sql("DROP TABLE IF EXISTS ${testTable}")
|
|
|
|
// you can see the error sql will not throw exception and return
|
|
try {
|
|
def errorSqlResult = try_sql("a b c d e f g")
|
|
assertTrue(errorSqlResult == null)
|
|
} catch (Throwable t) {
|
|
assertTrue(false, "Never catch exception")
|
|
}
|
|
}
|
|
|
|
// order_sql(sqlStr) equals to sql(sqlStr, isOrder=true)
|
|
// sort result by string dict
|
|
def list = order_sql """
|
|
select 2
|
|
union all
|
|
select 1
|
|
union all
|
|
select null
|
|
union all
|
|
select 15
|
|
union all
|
|
select 3
|
|
"""
|
|
|
|
assertEquals(null, list[0][0])
|
|
assertEquals(1, list[1][0])
|
|
assertEquals(15, list[2][0])
|
|
assertEquals(2, list[3][0])
|
|
assertEquals(3, list[4][0])
|
|
}
|
|
```
|
|
|
|
### qt action
|
|
qt action用于提交sql，并使用对应的.out TSV文件来校验结果
|
|
- String sql: 输入sql字符串
|
|
- return void
|
|
|
|
下面的样例代码存放于`${DORIS_HOME}/regression-test/suites/demo/qt_action.groovy`:
|
|
```groovy
|
|
suite("qt_action", "demo") {
|
|
/**
|
|
* qt_xxx sql equals to quickTest(xxx, sql) witch xxx is tag.
|
|
* the result will be compare to the relate file: ${DORIS_HOME}/regression_test/data/qt_action.out.
|
|
*
|
|
* if you want to generate .out tsv file for real execute result. you can run with -genOut or -forceGenOut option.
|
|
* e.g
|
|
* ${DORIS_HOME}/run-regression-test.sh --run qt_action -genOut
|
|
* ${DORIS_HOME}/run-regression-test.sh --run qt_action -forceGenOut
|
|
*/
|
|
qt_select "select 1, 'beijing' union all select 2, 'shanghai'"
|
|
|
|
qt_select2 "select 2"
|
|
|
|
// order result by string dict then compare to .out file.
|
|
// order_qt_xxx sql equals to quickTest(xxx, sql, true).
|
|
order_qt_union_all """
|
|
select 2
|
|
union all
|
|
select 1
|
|
union all
|
|
select null
|
|
union all
|
|
select 15
|
|
union all
|
|
select 3
|
|
"""
|
|
}
|
|
```
|
|
|
|
### test action
|
|
test action可以使用更复杂的校验规则来测试，比如验证行数、执行时间、是否抛出异常
|
|
|
|
可用参数
|
|
- String sql: 输入的sql字符串
|
|
- List<List<Object>> result: 提供一个List对象，用于比较真实查询结果与List对象是否相等
|
|
- Iterator<Object> resultIterator: 提供一个Iterator对象，用于比较真实查询结果与Iterator是否相等
|
|
- String resultFile: 提供一个文件Uri(可以是本地文件相对路径，或http(s)路径)，用于比较真实查询结果与http响应流是否相等，格式与.out文件格式类似，但没有块头和注释
|
|
- String exception: 校验抛出的异常是否包含某些字符串
|
|
- long rowNum: 验证结果行数
|
|
- long time: 验证执行时间是否小于这个值，单位是毫秒
|
|
- Closure<List<List<Object>>, Throwable, Long, Long> check: 自定义回调校验，可传入结果、异常、时间。存在回调函数时，其他校验方式会失效。
|
|
|
|
下面的样例代码存放于`${DORIS_HOME}/regression-test/suites/demo/test_action.groovy`:
|
|
```groovy
|
|
suite("test_action", "demo") {
|
|
test {
|
|
sql "abcdefg"
|
|
// check exception message contains
|
|
exception "errCode = 2, detailMessage = Syntax error"
|
|
}
|
|
|
|
test {
|
|
sql """
|
|
select *
|
|
from (
|
|
select 1 id
|
|
union all
|
|
select 2
|
|
) a
|
|
order by id"""
|
|
|
|
// multi check condition
|
|
|
|
// check return 2 rows
|
|
rowNum 2
|
|
// execute time must <= 5000 millisecond
|
|
time 5000
|
|
// check result, must be 2 rows and 1 column, the first row is 1, second is 2
|
|
result(
|
|
[[1], [2]]
|
|
)
|
|
}
|
|
|
|
test {
|
|
sql "a b c d e f g"
|
|
|
|
// other check will not work because already declared a check callback
|
|
exception "aaaaaaaaa"
|
|
|
|
// callback
|
|
check { result, exception, startTime, endTime ->
|
|
// assertXxx() will invoke junit5's Assertions.assertXxx() dynamically
|
|
assertTrue(exception != null)
|
|
}
|
|
}
|
|
|
|
test {
|
|
sql """
|
|
select 2
|
|
union all
|
|
select 1
|
|
union all
|
|
select null
|
|
union all
|
|
select 15
|
|
union all
|
|
select 3
|
|
"""
|
|
|
|
check { result, ex, startTime, endTime ->
|
|
// same as order_sql(sqlStr)
|
|
result = sortRows(result)
|
|
|
|
assertEquals(null, result[0][0])
|
|
assertEquals(1, result[1][0])
|
|
assertEquals(15, result[2][0])
|
|
assertEquals(2, result[3][0])
|
|
assertEquals(3, result[4][0])
|
|
}
|
|
}
|
|
|
|
// execute sql and order query result, then compare to iterator
|
|
def selectValues = [1, 2, 3, 4]
|
|
test {
|
|
order true
|
|
sql selectUnionAll(selectValues)
|
|
resultIterator(selectValues.iterator())
|
|
}
|
|
|
|
// compare to data/demo/test_action.csv
|
|
test {
|
|
order true
|
|
sql selectUnionAll(selectValues)
|
|
|
|
// you can set to http://xxx or https://xxx
|
|
// and compare to http response body
|
|
resultFile "test_action.csv"
|
|
}
|
|
}
|
|
```
|
|
|
|
### explain action
|
|
explain action用来校验explain返回的字符串是否包含某些字符串
|
|
|
|
可用参数:
|
|
- String sql: 查询的sql，需要去掉sql中的explain
|
|
- String contains: 校验explain是否包含某些字符串，可多次调用校验同时多个结果
|
|
- String notContains: 校验explain是否不含某些字符串，可多次调用校验同时多个结果
|
|
- Closure<String> check: 自定义校验回调函数，可以获取返回的字符串，存在校验函数时，其他校验方式会失效
|
|
- Closure<String, Throwable, Long, Long> check: 自定义校验回调函数，可以额外获取异常和时间
|
|
|
|
下面的样例代码存放于`${DORIS_HOME}/regression-test/suites/demo/explain_action.groovy`:
|
|
```groovy
|
|
suite("explain_action", "demo") {
|
|
explain {
|
|
sql("select 100")
|
|
|
|
// contains("OUTPUT EXPRS:<slot 0> 100\n") && contains("PARTITION: UNPARTITIONED\n")
|
|
contains "OUTPUT EXPRS:<slot 0> 100\n"
|
|
contains "PARTITION: UNPARTITIONED\n"
|
|
}
|
|
|
|
explain {
|
|
sql("select 100")
|
|
|
|
// contains(" 100\n") && !contains("abcdefg") && !("1234567")
|
|
contains " 100\n"
|
|
notContains "abcdefg"
|
|
notContains "1234567"
|
|
}
|
|
|
|
explain {
|
|
sql("select 100")
|
|
// simple callback
|
|
check { explainStr -> explainStr.contains("abcdefg") || explainStr.contains(" 100\n") }
|
|
}
|
|
|
|
explain {
|
|
sql("a b c d e")
|
|
// callback with exception and time
|
|
check { explainStr, exception, startTime, endTime ->
|
|
// assertXxx() will invoke junit5's Assertions.assertXxx() dynamically
|
|
assertTrue(exception != null)
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### streamLoad action
|
|
streamLoad action用于导入数据
|
|
可用参数为
|
|
- String db: db，默认值为regression-conf.groovy中的defaultDb
|
|
- String table: 表名
|
|
- String file: 要导入的文件路径，可以写data目录下的相对路径，或者写http url来导入网络文件
|
|
- Iterator<List<Object>> inputIterator: 要导入的迭代器
|
|
- String inputText: 要导入的文本, 较为少用
|
|
- InputStream inputStream: 要导入的字节流，较为少用
|
|
- long time: 验证执行时间是否小于这个值，单位是毫秒
|
|
- void set(String key, String value): 设置stream load的http请求的header，如label、columnSeparator
|
|
- Closure<String, Throwable, Long, Long> check: 自定义校验回调函数，可以获取返回结果、异常和超时时间。当存在回调函数时，其他校验项会失效。
|
|
|
|
下面的样例代码存放于`${DORIS_HOME}/regression-test/suites/demo/streamLoad_action.groovy`:
|
|
```groovy
|
|
suite("streamLoad_action", "demo") {
|
|
|
|
def tableName = "test_streamload_action1"
|
|
|
|
sql """
|
|
CREATE TABLE IF NOT EXISTS ${tableName} (
|
|
id int,
|
|
name varchar(255)
|
|
)
|
|
DISTRIBUTED BY HASH(id) BUCKETS 1
|
|
PROPERTIES (
|
|
"replication_num" = "1"
|
|
)
|
|
"""
|
|
|
|
streamLoad {
|
|
// you can skip declare db, because a default db already specify in ${DORIS_HOME}/conf/regression-conf.groovy
|
|
// db 'regression_test'
|
|
table tableName
|
|
|
|
// default label is UUID:
|
|
// set 'label' UUID.randomUUID().toString()
|
|
|
|
// default column_separator is specify in doris fe config, usually is '\t'.
|
|
// this line change to ','
|
|
set 'column_separator', ','
|
|
|
|
// relate to ${DORIS_HOME}/regression-test/data/demo/streamload_input.csv.
|
|
// also, you can stream load a http stream, e.g. http://xxx/some.csv
|
|
file 'streamload_input.csv'
|
|
|
|
time 10000 // limit inflight 10s
|
|
|
|
// stream load action will check result, include Success status, and NumberTotalRows == NumberLoadedRows
|
|
}
|
|
|
|
|
|
// stream load 100 rows
|
|
def rowCount = 100
|
|
// range: [0, rowCount)
|
|
// or rangeClosed: [0, rowCount]
|
|
def rowIt = range(0, rowCount)
|
|
.mapToObj({i -> [i, "a_" + i]}) // change Long to List<Long, String>
|
|
.iterator()
|
|
|
|
streamLoad {
|
|
table tableName
|
|
// also, you can upload a memory iterator
|
|
inputIterator rowIt
|
|
|
|
// if declared a check callback, the default check condition will ignore.
|
|
// So you must check all condition
|
|
check { result, exception, startTime, endTime ->
|
|
if (exception != null) {
|
|
throw exception
|
|
}
|
|
log.info("Stream load result: ${result}".toString())
|
|
def json = parseJson(result)
|
|
assertEquals("success", json.Status.toLowerCase())
|
|
assertEquals(json.NumberTotalRows, json.NumberLoadedRows)
|
|
assertTrue(json.NumberLoadedRows > 0 && json.LoadBytes > 0)
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### 其他Action
|
|
thread, lazyCheck, events, connect, selectUnionAll
|
|
具体可以在这个目录找到例子: `${DORIS_HOME}/regression-test/suites/demo`
|
|
|
|
## 启动脚本例子
|
|
```shell
|
|
# 查看脚本参数说明
|
|
./run-regression-test.sh h
|
|
|
|
# 查看框架参数说明
|
|
./run-regression-test.sh --run -h
|
|
|
|
# 测试所有用例
|
|
./run-regression-test.sh
|
|
|
|
# 删除测试框架编译结果和测试日志
|
|
./run-regression-test.sh --clean
|
|
|
|
# 测试suiteName为sql_action的用例, 目前suiteName等于文件名前缀，例子对应的用例文件是sql_action.groovy
|
|
./run-regression-test.sh --run sql_action
|
|
|
|
# 测试suiteName包含'sql'的用例，**注意需要用单引号括起来**
|
|
./run-regression-test.sh --run '*sql*'
|
|
|
|
# 测试demo和perfomance group
|
|
./run-regression-test.sh --run -g 'demo,performance'
|
|
|
|
# 测试demo group下的sql_action
|
|
./run-regression-test.sh --run -g demo -s sql_action
|
|
|
|
# 测试demo目录下的sql_action
|
|
./run-regression-test.sh --run -d demo -s sql_action
|
|
|
|
# 测试demo目录下用例，排除sql_action用例
|
|
./run-regression-test.sh --run -d demo -xs sql_action
|
|
|
|
# 排除demo目录的用例
|
|
./run-regression-test.sh --run -xd demo
|
|
|
|
# 排除demo group的用例
|
|
./run-regression-test.sh --run -xg demo
|
|
|
|
# 自定义配置
|
|
./run-regression-test.sh --run -conf a=b
|
|
|
|
# 并发执行
|
|
./run-regression-test.sh --run -parallel 5 -suiteParallel 10 -actionParallel 20
|
|
```
|
|
|
|
## 使用查询结果自动生成.out文件
|
|
```shell
|
|
# 使用查询结果自动生成sql_action用例的.out文件，如果.out文件存在则忽略
|
|
./run-regression-test.sh --run sql_action -genOut
|
|
|
|
# 使用查询结果自动生成sql_action用例的.out文件，如果.out文件存在则覆盖
|
|
./run-regression-test.sh --run sql_action -forceGenOut
|
|
```
|
|
|
|
## Suite插件
|
|
有的时候我们需要拓展Suite类，但不便于修改Suite类的源码，则可以通过插件来进行拓展。默认插件目录为`${DORIS_HOME}/regression-test/plugins`，在其中可以通过groovy脚本定义拓展方法，以`plugin_example.groovy`为例，为Suite类增加了testPlugin函数用于打印日志：
|
|
```groovy
|
|
import org.apache.doris.regression.suite.Suite
|
|
|
|
// register `testPlugin` function to Suite,
|
|
// and invoke in ${DORIS_HOME}/regression-test/suites/demo/test_plugin.groovy
|
|
Suite.metaClass.testPlugin = { String info /* param */ ->
|
|
|
|
// which suite invoke current function?
|
|
Suite suite = delegate as Suite
|
|
|
|
// function body
|
|
suite.getLogger().info("Test plugin: suiteName: ${suite.name}, info: ${info}".toString())
|
|
|
|
// optional return value
|
|
return "OK"
|
|
}
|
|
|
|
logger.info("Added 'testPlugin' function to Suite")
|
|
```
|
|
|
|
增加了testPlugin函数后，则可以在普通用例中使用它，以`${DORIS_HOME}/regression-test/suites/demo/test_plugin.groovy`为例:
|
|
```groovy
|
|
suite("test_plugin", "demo") {
|
|
// register testPlugin function in ${DORIS_HOME}/regression-test/plugins/plugin_example.groovy
|
|
def result = testPlugin("message from suite")
|
|
assertEquals("OK", result)
|
|
}
|
|
```
|
|
|
|
## CI/CD的支持
|
|
### TeamCity
|
|
TeamCity可以通过stdout识别Service Message。当使用`--teamcity`参数启动回归测试框架时，回归测试框架就会在stdout打印TeamCity Service Message，TeamCity将会自动读取stdout中的事件日志，并在当前流水线中展示`Tests`，其中会展示测试的test及其日志。
|
|
因此只需要配置下面一行启动回归测试框架的命令即可。其中`-Dteamcity.enableStdErr=false`可以让错误日志也打印到stdout中，方便按时间顺序分析日志。
|
|
```shell
|
|
JAVA_OPTS="-Dteamcity.enableStdErr=${enableStdErr}" ./run-regression-test.sh --teamcity --run
|
|
``` |