# 如何跑 SYSBENCH 测试

SYSBENCH 是一个跨平台且支持多线程的模块化基准测试工具，可以执行CPU/内存/线程/IO/数据库等方面的性能测试，用于评估系统在运行高负载的数据库时相关核心参数的性能表现。使用SYSBENCH 是为了绕过复杂的数据库基准设置，甚至在没有安装数据库的前提下，快速了解数据库系统的性能。
SYSBENCH 可以做多种性能测试。本文主要介绍对数据库性能（OLTP）的测试。

## SYSBENCH 安装

### 编译安装
代码下载：`https://github.com/akopytov/sysbench/releases/tag/1.0.20` 。
解压缩后，编译安装。

```bash
cd sysbench-1.0.20
# yum -y install automake libtool
# yum -y install mysql.x86_64 mysql-devel.x86_64 --allowerasing 

# ./autogen.sh
# ./configure --prefix=/usr/sysbench/ --with-mysql-includes=/usr/include/mysql/ --with-mysql-libs=/usr/lib64/mysql/ --with-mysql

# make
# make install
# cp /usr/sysbench/bin/sysbench /usr/local/bin/
 
# cd /usr/sysbench/
# bin/sysbench --help
```

查看帮助命令，验证 SYSBENCH 是否安装成功：

```bash
[root@obce-0000 ~]# cd /usr/sysbench/
[root@obce-0000 sysbench]# bin/sysbench --help
Usage:
  sysbench [options]... [testname] [command]

Commands implemented by most tests: prepare run cleanup help

General options:
  --threads=N                     number of threads to use [1]
  --events=N                      limit for total number of events [0]
  --time=N                        limit for total execution time in seconds [10]
  --forced-shutdown=STRING        number of seconds to wait after the --time limit before forcing shutdown, or 'off' to disable [off]
  --thread-stack-size=SIZE        size of stack per thread [64K]
  --rate=N                        average transactions rate. 0 for unlimited rate [0]
  --report-interval=N             periodically report intermediate statistics with a specified interval in seconds. 0 disables intermediate reports [0]
  --report-checkpoints=[LIST,...] dump full statistics and reset all counters at specified points in time. The argument is a list of comma-separated values representing the amount of time in seconds elapsed from start of test when report checkpoint(s) must be performed. Report checkpoints are off by default. []
  --debug[=on|off]                print more debugging info [off]
  --validate[=on|off]             perform validation checks where possible [off]
  --help[=on|off]                 print help and exit [off]
  --version[=on|off]              print version and exit [off]
  --config-file=FILENAME          File containing command line options
  --tx-rate=N                     deprecated alias for --rate [0]
  --max-requests=N                deprecated alias for --events [0]
  --max-time=N                    deprecated alias for --time [0]
  --num-threads=N                 deprecated alias for --threads [1]

Pseudo-Random Numbers Generator options:
  --rand-type=STRING random numbers distribution {uniform,gaussian,special,pareto} [special]
  --rand-spec-iter=N number of iterations used for numbers generation [12]
  --rand-spec-pct=N  percentage of values to be treated as 'special' (for special distribution) [1]
  --rand-spec-res=N  percentage of 'special' values to use (for special distribution) [75]
  --rand-seed=N      seed for random number generator. When 0, the current time is used as a RNG seed. [0]
  --rand-pareto-h=N  parameter h for pareto distribution [0.2]

Log options:
  --verbosity=N verbosity level {5 - debug, 0 - only critical messages} [3]

  --percentile=N       percentile to calculate in latency statistics (1-100). Use the special value of 0 to disable percentile calculations [95]
  --histogram[=on|off] print latency histogram in report [off]

General database options:

  --db-driver=STRING  specifies database driver to use ('help' to get list of available drivers) [mysql]
  --db-ps-mode=STRING prepared statements usage mode {auto, disable} [auto]
  --db-debug[=on|off] print database-specific debug information [off]


Compiled-in database drivers:
  mysql - MySQL driver

mysql options:
  --mysql-host=[LIST,...]          MySQL server host [localhost]
  --mysql-port=[LIST,...]          MySQL server port [3306]
  --mysql-socket=[LIST,...]        MySQL socket
  --mysql-user=STRING              MySQL user [sbtest]
  --mysql-password=STRING          MySQL password []
  --mysql-db=STRING                MySQL database name [sbtest]
  --mysql-ssl[=on|off]             use SSL connections, if available in the client library [off]
  --mysql-ssl-cipher=STRING        use specific cipher for SSL connections []
  --mysql-compression[=on|off]     use compression, if available in the client library [off]
  --mysql-debug[=on|off]           trace all client library calls [off]
  --mysql-ignore-errors=[LIST,...] list of errors to ignore, or "all" [1213,1020,1205]
  --mysql-dry-run[=on|off]         Dry run, pretend that all MySQL client API calls are successful without executing them [off]

Compiled-in tests:
  fileio - File I/O test
  cpu - CPU performance test
  memory - Memory functions speed test
  threads - Threads subsystem performance test
  mutex - Mutex performance test

See 'sysbench <testname> help' for a list of options for each test.

```



### 常见安装问题

可能遇到的报错：

+ `automake 1.10.x (aclocal) wasn’t found, exiting`
解决办法：操作系统没有安装automake，运行命令：yum install automake.noarch，即可安装。

+ `libtoolize 1.4+ wasn’t found, exiting`
解决办法：操作系统没有安装libtool，运行命令：yum install libtool，即可安装。

+ `drv_mysql.c:35:19: fatal error: mysql.h: No such file or directory`
解决办法：操作系统没有安装mysql的开发lib库，运行命令：yum install mysql-devel.x86_64，即可安装。

+ `error while loading shared libraries: libmysqlclient_r.so.16`
这错误通常是直接下载编译好的文件碰到的。
解决办法：将下载文件中的`libmysqlclient_r.so.16.0.0` 复制到目录 `/usr/lib64/mysql` 中，并做一个软链接
```bash
sudo cp libmysqlclient_r.so.16.0.0 /usr/lib64/mysql
sudo ln -s /usr/lib64/mysql/libmysqlclient_r.so.16.0.0 /usr/lib64/mysql/libmysqlclient_r.so.16
sudo ldconfig
```


## 测试准备

### 参数设置

+ OS 参数

```bash
sudo sh -c 'for x in /sys/class/net/eth0/queues/rx-*; do echo ff>$x/rps_cpus; done'
sudo sh -c "echo 32768 > /proc/sys/net/core/rps_sock_flow_entries"
sudo sh -c "echo 4096 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt"

```

说明：ffffffff表示使用32个核。请根据实际配置修改，例如ECS为8核，则输入ff。

+ OceanBase 参数

SYS 租户参数

```sql
alter system set enable_auto_leader_switch=false;
alter system set enable_one_phase_commit=false;
alter system set enable_monotonic_weak_read = true;
alter system set weak_read_version_refresh_interval='5s';
alter system _ob_minor_merge_schedule_interval='5s';

alter system set memory_limit_percentage = 90;    --  OB占系统总内存的比例，提高OB可用的内存量。
alter system set memstore_limit_percentage = 55;  --  memstore占租户的内存比，尽量增大memstore的空间（但是可能对读操作有负面影响）。
alter system set freeze_trigger_percentage = 70;  --  启动major/minor freeze的时机，让转储（minor freeze）尽早启动，memstore内存尽早释放。
alter system set minor_freeze_times = 50;        --  minor freeze的次数，尽量不在测试期间触发major freeze。
alter system set minor_warm_up_duration_time = 0;  --  加快minor freeze

alter system set merge_thread_count = 32;  --  增大合并的线程数。
alter system set minor_merge_concurrency = 8;  --  增大转储的线程数，期望提高转储的速度。
alter system set _mini_merge_concurrency = 4;  --  增大mini_merge的线程数，期望提高mini_merge的速度（默认值为3）。调大为8以后，发现会导致压测中CPU使用率有时飙升至90%，对性能有影响。

```

PROXY 参数 

```sql
alter proxyconfig set proxy_mem_limited='4G'; --防止oom，可根据实际环境动态调整
alter proxyconfig set enable_compression_protocol=false; --关闭压缩，降低cpu%
alter proxyconfig set work_thread_num=32; -- 调整工作线程数，寻找最优性能
alter proxyconfig set enable_compression_protocol=false;
alter proxyconfig set enable_metadb_used=false;
alter proxyconfig set enable_standby=false;
alter proxyconfig set enable_strict_stat_time=false;
alter proxyconfig set use_local_dbconfig=true;
```

租户参数

```sql
数据库下租户设置，防止事务超时
set global ob_timestamp_service='GTS' ;
set global autocommit=ON;
set global ob_query_timeout=36000000000;
set global ob_trx_timeout=36000000000;
set global max_allowed_packet=67108864;
set global ob_sql_work_area_percentage=100;
set global parallel_max_servers=800;
set global parallel_servers_target=800;
```

### 数据准备

+ 准备数据库账户

```sql
create database sysbenchdb;
grant all privileges on sysbenchdb.* to u_sysbench identified by '123456';

```

+ （可选）修改建表语句

建表语句在 `share/sysbench/oltp_common.lua` 里。
```bash
cd /usr/sysbench/
vim share/sysbench/oltp_common.lua +150
150 function create_table(drv, con, table_num)
```

如果想建分区表，就修改这里的脚本。分区表需要选择一个分区键，并且主键要包含分区键。

+ 初始化表和数据

SYSBENCH 参数说明：
    - `--tables` ：指定表的数量
    - `--table_size` ：指定表的数据量
    - `--threads`：指定并发数
    - `--mysql-ignore-errors`：指定忽略的错误号，忽略后就继续跑。否则，报错就中断。
    - `--time` : 指定运行时间。
    - `--report-interval`：报告间隔。

示例：

```bash
sysbench --test=./oltp_read_only.lua --mysql-host=172.20.249.54 --mysql-port=2883 --mysql-db=sysbenchdb --mysql-user="obce-3zones:obmysql:u_sysbench"  --mysql-password=123456 --tables=12 --table_size=10000000 --threads=12 --time=300 --report-interval=3 --db-driver=mysql --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016,1062 prepare

输出：

WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Initializing worker threads...

Creating table 'sbtest3'...
Creating table 'sbtest1'...
Creating table 'sbtest6'...
Creating table 'sbtest7'...
Creating table 'sbtest5'...
Creating table 'sbtest2'...
Creating table 'sbtest8'...
Creating table 'sbtest4'...
Inserting 1000000 records into 'sbtest3'
Inserting 1000000 records into 'sbtest1'
Inserting 1000000 records into 'sbtest6'
Inserting 1000000 records into 'sbtest5'
Inserting 1000000 records into 'sbtest7'
Inserting 1000000 records into 'sbtest8'
Inserting 1000000 records into 'sbtest4'
Inserting 1000000 records into 'sbtest2'
Creating a secondary index on 'sbtest6'...
Creating a secondary index on 'sbtest8'...
Creating a secondary index on 'sbtest3'...
Creating a secondary index on 'sbtest5'...
Creating a secondary index on 'sbtest4'...
Creating a secondary index on 'sbtest1'...
Creating a secondary index on 'sbtest7'...
Creating a secondary index on 'sbtest2'...
Creating table 'sbtest9'...
Inserting 1000000 records into 'sbtest9'
Creating a secondary index on 'sbtest9'...
[root@obce-0000 sysbench]#
```

常见初始化问题：

+ `FATAL: mysql_drv_query() returned error 4030 (Over tenant memory limits) for query 'INSERT INTO sbtest5` 

这就是典型的增量内存消耗速度高于转储速度。如果租户内存很小的话，这个写入的并发不能太高。
上面初始化配置供参考：

```sql
+--------------------+------------------+---------+---------+------------+------------+---------+-------+--------------------+-----------+-------------+
| resource_pool_name | unit_config_name | max_cpu | min_cpu | max_mem_gb | min_mem_gb | unit_id | zone  | observer           | tenant_id | tenant_name |
+--------------------+------------------+---------+---------+------------+------------+---------+-------+--------------------+-----------+-------------+
| sys_pool           | sys_unit_config  |       5 |       5 |          2 |          2 |       1 | zone1 | 172.20.249.52:2882 |         1 | sys         |
| sys_pool           | sys_unit_config  |       5 |       5 |          2 |          2 |       2 | zone2 | 172.20.249.49:2882 |         1 | sys         |
| sys_pool           | sys_unit_config  |       5 |       5 |          2 |          2 |       3 | zone3 | 172.20.249.51:2882 |         1 | sys         |
| my_pool_zone1      | unit1            |       9 |       9 |         19 |         19 |    1001 | zone1 | 172.20.249.52:2882 |      1002 | obmysql     |
| my_pool_zone2      | unit1            |       9 |       9 |         19 |         19 |    1002 | zone2 | 172.20.249.49:2882 |      1002 | obmysql     |
| my_pool_zone3      | unit1            |       9 |       9 |         19 |         19 |    1003 | zone3 | 172.20.249.51:2882 |      1002 | obmysql     |
+--------------------+------------------+---------+---------+------------+------------+---------+-------+--------------------+-----------+-------------+
6 rows in set (0.042 sec)

alter system set merge_thread_count = 32;  --  增大合并的线程数。
alter system set minor_merge_concurrency = 8;  --  增大转储的线程数，期望提高转储的速度。
alter system set _mini_merge_concurrency = 4;  --  增大mini_merge的线程数，期望提高mini_merge的速度（默认值为3）。调大为8以后，发现会导致压测中CPU使用率有时飙升至90%，对性能有影响。

alter system set memory_limit = '24G';    --  OB占系统总内存的比例，提高OB可用的内存量。
alter system set memstore_limit_percentage = 55;  --  memstore占租户的内存比，尽量增大memstore的空间（但是可能对读操作有负面影响）。
alter system set freeze_trigger_percentage = 70;  --  启动major/minor freeze的时机，让转储（minor freeze）尽早启动，memstore内存尽早释放。
alter system set minor_freeze_times = 50;        --  minor freeze的次数，尽量不在测试期间触发major freeze。
alter system set minor_warm_up_duration_time = 0;  --  加快minor freeze

```

## 场景测试

不同场景使用不同的 LUA 脚本文件。

```bash
[root@obce-0000 sysbench]# pwd
/usr/sysbench/share/sysbench
[root@obce-0000 sysbench]# ls *.lua
bulk_insert.lua  oltp_delete.lua  oltp_point_select.lua  oltp_read_write.lua    oltp_update_non_index.lua  select_random_points.lua
oltp_common.lua  oltp_insert.lua  oltp_read_only.lua     oltp_update_index.lua  oltp_write_only.lua        select_random_ranges.lua
```

复制初始化脚本命令，替换场景文件名，`prepare` 改为 `run` 。

SYSBENCH 的场景测试，建议同一个场景设置不同的并发都跑一遍，然后用 EXCEL 记录相应的性能值，连成曲线，观察线性扩展关系。

+ 只读测试

只读测试的 SQL 都是读。OceanBase 的读默认是强一致性读，只读取数据分区的主副本。如果要观察 OceanBase 的读写分离能力，可以修改这个测试脚本，在读 SQL 里增加弱一致性读 Hint（`/*+ read_consistency(weak) */` )  即可随机读取备副本。
稍微不同的是，OceanBase 的弱一致性读备副本还受参数 `max_stale_time_for_weak_consistency` 限制。如果备副本延迟时间超出这个参数定义（默认 5s），则备副本不提供读服务。

```bash
sysbench --test=./oltp_read_only.lua --mysql-host=172.20.249.52 --mysql-port=2883 --mysql-db=sysbenchdb --mysql-user="obce-3zones:obmysql:u_sysbench"  --mysql-password=123456 --tables=9 --table_size=1000000 --threads=8 --time=300 --report-interval=60 --db-driver=mysql --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016,1062 run

输出：
WARNING: the --test option is deprecated. You can pass a script name or path on the command line without any options.
sysbench 1.0.20 (using bundled LuaJIT 2.1.0-beta2)

Running the test with following options:
Number of threads: 8
Report intermediate results every 60 second(s)
Initializing random number generator from current time


Initializing worker threads...

Threads started!

[ 60s ] thds: 8 tps: 114.73 qps: 1607.23 (r/w/o: 1607.23/0.00/0.00) lat (ms,95%): 179.94 err/s: 0.00 reconn/s: 0.00
[ 120s ] thds: 8 tps: 79.95 qps: 1119.26 (r/w/o: 1119.26/0.00/0.00) lat (ms,95%): 262.64 err/s: 0.00 reconn/s: 0.00
[ 180s ] thds: 8 tps: 97.37 qps: 1362.75 (r/w/o: 1362.75/0.00/0.00) lat (ms,95%): 215.44 err/s: 0.00 reconn/s: 0.00
[ 240s ] thds: 8 tps: 109.87 qps: 1538.27 (r/w/o: 1538.27/0.00/0.00) lat (ms,95%): 193.38 err/s: 0.00 reconn/s: 0.00
[ 300s ] thds: 8 tps: 193.03 qps: 2702.82 (r/w/o: 2702.82/0.00/0.00) lat (ms,95%): 95.81 err/s: 0.00 reconn/s: 0.00
SQL statistics:
    queries performed:
        read:                            499870
        write:                           0
        other:                           0
        total:                           499870
    transactions:                        35705  (118.99 per sec.)
    queries:                             499870 (1665.81 per sec.)
    ignored errors:                      0      (0.00 per sec.)
    reconnects:                          0      (0.00 per sec.)

General statistics:
    total time:                          300.0755s
    total number of events:              35705

Latency (ms):
         min:                                   11.09
         avg:                                   67.22
         max:                                 1347.51
         95th percentile:                      179.94
         sum:                              2400071.63

Threads fairness:
    events (avg/stddev):           4463.1250/52.63
    execution time (avg/stddev):   300.0090/0.03
```

+ 纯写场景

纯写场景就是 `insert`、`update` 和 `delete` SQL 都有。如果表是分区表，大概率会有跨节点的分布式事务。OceanBase 的分布式事务是强一致性模型，所以在并发不高节点规模不大时，相比传统数据库，这个测试性能会相对差一些。随着数据量的增长，并发的提升以及机器规模的增长，OceanBase 的多活和线性扩展优势会逐步体现出来。

```bash
sysbench --test=./oltp_write_only.lua --mysql-host=172.20.249.52 --mysql-port=2883 --mysql-db=sysbenchdb --mysql-user="obce-3zones:obmysql:u_sysbench"  --mysql-password=123456 --tables=9 --table_size=1000000 --threads=8 --time=300 --report-interval=60 --db-driver=mysql --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016 run

```

+ 读写混合场景

```bash
sysbench --test=./oltp_read_write.lua --mysql-host=172.20.249.52 --mysql-port=2883 --mysql-db=sysbenchdb --mysql-user="obce-3zones:obmysql:u_sysbench"  --mysql-password=123456 --tables=9 --table_size=1000000 --threads=8 --time=300 --report-interval=60 --db-driver=mysql --db-ps-mode=disable --skip-trx=on --mysql-ignore-errors=6002,6004,4012,2013,4016,1062 run
```

## 性能调优经验

### 转储&合并

对sysbench来说（1000000*30 行表），sysbench一般来说数据量较小，导完数据后不会发生转储，此时做合并有消极影响。
影响在于：合并转储将 `memtable` 刷到硬盘，多了一个 `major sstable`，从硬盘操作数据肯定比在 `memtable`要慢（`memtable scan` 另说），导致性能下降。

|               | 合并转储前（qps/rt）   | 转储后（qps/rt）     | 合并后（qps/rt）     |    |
|---------------|-----------------|-----------------|-----------------|----|
| point_select  | 367240.41/7.70  | 332106.87/8.28  | 342172.34/8.43  |    |
| write_only    | 137125.60/81.48 | 138654.65/84.47 | 135978.23/84.47 |    |
| 3千万行单表range查询 | 18.601s         | 10.853s         | 10.709s         |    |

### 调整 PRIMARY_ZONE

|                        | 集中式部署（zone1）    | random部署         |
|------------------------|-----------------|------------------|
| sysbench(point_select) | 261505.51/11.87 | 373923.98/7.56   |
| sysbench(read_write)   | 81818.40/297.92 | 125315.97/427.07 |
