OceanBase 数据库 TPC-C 测试 
===========================================



什么是 TPC-C 
------------------------------

TPC-C 是一个对 OLTP（联机交易处理）系统进行测试的规范，使用一个商品销售模型对 OLTP 系统进行测试。详细信息，参考 [TPC-C 中文简介](https://github.com/domino-succ/tpcc-hbase/wiki/%E4%B8%AD%E6%96%87-TPC-C%E7%AE%80%E4%BB%8B)。

环境准备 
-------------------------

* JDK：建议使用 1.8u131 及以上版本。

  

* JDBC：建议使用 mysql-connector-java-5.1.47 版本，其他版本可能存在语法兼容性问题。

  

* Ant：建议使用 apache-ant-1.10 及以上版本。

  

* Benchmark SQL：建议使用 Benchmark SQL 5.0。

  

* OBClient：详细信息，参考 [OBClient 文档](https://github.com/oceanbase/obclient/blob/master/README.md)。

  

* OceanBase 数据库：建议单独部署 ODP。详细信息，参考 [使用 OBD 部署 OceanBase 数据库](../2.quickstart/4.deploy-the-oceanbase-database-by-using-obd.md)。

  




测试方案 
-------------------------

* 使用 OBD 部署 OceanBase 数据库集群。ODP 和 TPC-C 单独部署在一台机器上, 作为客户端的压力机器。

  

* OceanBase 集群规模为 1：1：1。部署成功后，新建执行 TPC-C 测试的租户及用户（sys租户是管理集群的内置系统租户，请勿直接使用 sys 租户进行测试）。将租户的 `primary_zone` 设置为 `RANDOM`。`RANDOM` 表示新建表分区的 Leader 随机到这 3 台机器。租户详情，参考 [创建用户租户。](../6.administrator-guide/3.basic-database-management/5.tenants/2.create-a-user-tenant.md)

  




**测试规格** 

```shell
warehouses=1000
loadWorkers=40
terminals=600
runMins=5
newOrderWeight=45
paymentWeight=43
orderStatusWeight=4
deliveryWeight=4
stockLevelWeight=4
```



安装 Ant 
---------------------------

按照以下步骤安装 Ant：

1. 下载 Ant。

   ```unknow
   wget "http://archive.apache.org/dist/ant/binaries/apache-ant-1.10.6-bin.zip"
   ```

   

2. 设置环境变量。

   ```unknow
   export ANT_HOME=xx/benchmarksql/apache-ant-1.10.6/
   ```

   

3. 检查是否安装成功。

   ```unknow
   [OceanBase008064.gtjsqa xx/benchmarksql]$ant -version
   ```

   

   如果返回以下信息，则安装成功。

   ```unknow
   Apache Ant(TM) version 1.10.6 compiled on May 2 2019
   ```

   




安装 Benchmark SQL 
-------------------------------------

按照以下步骤安装 Benchmark SQL：

1. 下载 Benchmark SQL。

   详细信息参考 [Benchmark SQL 下载地址](https://sourceforge.net/projects/benchmarksql/files/latest/download/)。
   

2. 解压 Benchmark SQL。

   ```unknow
   unzip ./benchmarksql-5.0.zip
   ```

   

3. 编译 Benchmark SQL。

   进入 Benchmark SQL 解压后的目录，使用 Ant 编译 Benchmark SQL：

   ```unknow
   [wieck@localhost ~] $ cd benchmarksql
   [wieck@localhost benchmarksql] $ ant
   ```

   




适配 Benchmark SQL5 
--------------------------------------

由于 Benchmark SQL5 不支持 OceanBase 数据库的 TPC-C 测试，本节将详细介绍如何通过修改BenchMarkSQL5 部分源码支持 OceanBase 数据库。

1. 修改 `benchmarksql-5.0/src/client/jTPCC.java` 文件，增加 OceanBase 数据库相关内容。

   ```unknow
   if (iDB.equals("firebird"))
               dbType = DB_FIREBIRD;
           else if (iDB.equals("oracle"))
               dbType = DB_ORACLE;
           else if (iDB.equals("postgres"))
               dbType = DB_POSTGRES;
           else if (iDB.equals("oceanbase"))
               dbType = DB_OCEANBASE;
           else
           {
               log.error("unknown database type '" + iDB + "'");
               return;
           }
   ```

   

   修改 `benchmarksql-5.0/src/client/jTPCCConfig.java` 文件，增加 OceanBase 数据库类型。

   ```unknow
   public final static int         DB_UNKNOWN = 0,
   DB_FIREBIRD = 1,
   DB_ORACLE = 2,
   DB_POSTGRES = 3,
   DB_OCEANBASE = 4;
   ```

   
   

2. 修改 benchmarksql-5.0/src/client/jTPCCConnection.java 文件，在 SQL 子查询增加"AS L"别名。

   ```unknow
   default:
                   stmtStockLevelSelectLow = dbConn.prepareStatement(
                       "SELECT count(*) AS low_stock FROM (" +
                       "    SELECT s_w_id, s_i_id, s_quantity " +
                       "        FROM bmsql_stock " +
                       "        WHERE s_w_id = ? AND s_quantity < ? AND s_i_id IN (" +
                       "            SELECT ol_i_id " +
                       "                FROM bmsql_district " +
                       "                JOIN bmsql_order_line ON ol_w_id = d_w_id " +
                       "                 AND ol_d_id = d_id " +
                       "                 AND ol_o_id >= d_next_o_id - 20 " +
                       "                 AND ol_o_id < d_next_o_id " +
                       "                WHERE d_w_id = ? AND d_id = ? " +
                       "        ) " +
                       "    )AS L");
                   break;
   ```

   

3. 重新编译修改后的源码。

   ```unknow
   [oceanbase@testdrier test]# cd benchmarksql-5.0
   [oceanbase@testdrier benchmarksql-5.0]# ant
   ```

   

4. 在 `benchmarksql-5.0/run` 目录下，创建文件 `prop.oceanbase`。

   ```unknow
   db=oceanbase
   driver=com.mysql.jdbc.Driver
   conn=jdbc:mysql://$host_ip:$port/$db_name?rewriteBatchedStatements=true&allowMultiQueries=true&useLocalSessionState=true&useUnicode=true&characterEncoding=utf-8&socketTimeout=30000000
   //请填写完整user信息
   user=$user@$tenant 
   password=*****
   warehouses=1000
   loadWorkers=40
   terminals=600
   database=test
   //To run specified transactions per terminal- runMins must equal zero
   runTxnsPerTerminal=0
   //To run for specified minutes- runTxnsPerTerminal must equal zero
   runMins=5
   //Number of total transactions per minute
   limitTxnsPerMin=0
   //Set to true to run in 4.x compatible mode. Set to false to use the
   //entire configured database evenly.
   terminalWarehouseFixed=true
   //The following five values must add up to 100
   //The default percentages of 45, 43, 4, 4 & 4 match the TPC-C spec
   newOrderWeight=45
   paymentWeight=43
   orderStatusWeight=4
   deliveryWeight=4
   stockLevelWeight=4
   // Directory name to create for collecting detailed result data.
   // Comment this out to suppress.
   resultDirectory=my_result_%tY-%tm-%td_%tH%tM%tS
   osCollectorScript=./misc/os_collector_linux.py
   osCollectorInterval=1
   //osCollectorSSHAddr=user@dbhost
   //osCollectorDevices=net_eth0 blk_sda
   ```

   

   **prop.oceanbase中的参数说明：** 

   1. JDBC 连接串：conn=jdbc:mysql://x.x.x.x(ip):xx(port)/xxxx(dbname)?rewriteBatchedStatements=true\&allowMultiQueries=true\&useLocalSessionState=true\&useUnicode=true\&characterEncoding=utf-8\&socketTimeout=3000000

   2. rewriteBatchedStatements：

   

   3. 并发数量（terminals）：600，mysql 租户配置下并发需要结合具体配置动态调整。

   4. useLocalSessionState：是否使用autocommit，read_only和transaction isolation的内部值(jdbc端的本地值)，建议设置为true，如果设置为false，则需要发语句到远端请求，增加发送请求频次，影响性能。

   5. warehouses/loadWorkers这两项用于设置压测数据量，可以适当调整。

   6. numTerminals \<= 0 \|\| numTerminals \> 10\*numWarehouses，terminals的范围需要在这个区间内。
   

5. 修改文件：benchmarksql-5.0/run/funcs.sh，添加OceanBase数据库类型。

   ```unknow
   function setCP()
   {
       case "$(getProp db)" in
    firebird)
        cp="../lib/firebird/*:../lib/*"
        ;;
    oracle)
        cp="../lib/oracle/*"
        if [ ! -z "${ORACLE_HOME}" -a -d ${ORACLE_HOME}/lib ] ; then
     cp="${cp}:${ORACLE_HOME}/lib/*"
        fi
        cp="${cp}:../lib/*"
        ;;
    postgres)
        cp="../lib/postgres/*:../lib/*"
        ;;
    oceanbase)
        cp="../lib/oceanbase/*:../lib/*"
        ;;
       esac
       myCP=".:${cp}:../dist/*"
       export myCP
   }
   
   ...省略
   
   case "$(getProp db)" in
       firebird|oracle|postgres|oceanbase)
       ;;
       "") echo "ERROR: missing db= config option in ${PROPS}" >&2
       exit 1
       ;;
       *)  echo "ERROR: unsupported database type 'db=$(getProp db)' in ${PROPS}" >&2
       exit 1
       ;;
   esac
   ```

   

6. 添加mysql java connector驱动，推荐mysql-connector-java-5.1.47.jar。

   ```unknow
   [oceanbase@testdrier benchmarksql-5.0]# mkdir lib/oceanbase/
   [oceanbase@testdrier benchmarksql-5.0]# cp mysql-connector-java-5.1.47.jar benchmarksql-5.0/lib/oceanbase/
   ```

   

7. 修改benchmarksql-5.0/run/runDatabaseBuild.sh。

   ```unknow
   AFTER_LOAD="indexCreates foreignKeys extraHistID buildFinish"
   修改为：
   AFTER_LOAD="indexCreates buildFinish"
   ```

   

8. 删除benchmarksql-5.0/run/sql.common目录下sql文件的sequence。

   tableCreates.sql中去掉。

   ```unknow
   create sequence bmsql_hist_id_seq;
   ```

   

   tableDrops.sql中去掉。

   ```unknow
   drop sequence bmsql_hist_id_seq;
   ```

   
   

9. 改造BenchMarkSQL5中的sql。

   备份并重写benchmarksql-5.0/run/sql.common/tableCreates.sql。

   ```unknow
   create table bmsql_config (
     cfg_name    varchar(30) primary key,
     cfg_value   varchar(50)
   );
   
   create tablegroup if not exists tpcc_group binding true partition by hash partitions 128;
   
   create table bmsql_warehouse (
     w_id        integer   not null,
     w_ytd       decimal(12,2),
     w_tax       decimal(4,4),
     w_name      varchar(10),
     w_street_1  varchar(20),
     w_street_2  varchar(20),
     w_city      varchar(20),
     w_state     char(2),
     w_zip       char(9),
     primary key(w_id)
   )tablegroup='tpcc_group' partition by hash(w_id) partitions 128;
   
   create table bmsql_district (
     d_w_id       integer       not null,
     d_id         integer       not null,
     d_ytd        decimal(12,2),
     d_tax        decimal(4,4),
     d_next_o_id  integer,
     d_name       varchar(10),
     d_street_1   varchar(20),
     d_street_2   varchar(20),
     d_city       varchar(20),
     d_state      char(2),
     d_zip        char(9),
     PRIMARY KEY (d_w_id, d_id)
   )tablegroup='tpcc_group' partition by hash(d_w_id) partitions 128;
   
   create table bmsql_customer (
     c_w_id         integer        not null,
     c_d_id         integer        not null,
     c_id           integer        not null,
     c_discount     decimal(4,4),
     c_credit       char(2),
     c_last         varchar(16),
     c_first        varchar(16),
     c_credit_lim   decimal(12,2),
     c_balance      decimal(12,2),
     c_ytd_payment  decimal(12,2),
     c_payment_cnt  integer,
     c_delivery_cnt integer,
     c_street_1     varchar(20),
     c_street_2     varchar(20),
     c_city         varchar(20),
     c_state        char(2),
     c_zip          char(9),
     c_phone        char(16),
     c_since        timestamp,
     c_middle       char(2),
     c_data         varchar(500),
     PRIMARY KEY (c_w_id, c_d_id, c_id)
   )tablegroup='tpcc_group' partition by hash(c_w_id) partitions 128;
   
   
   create table bmsql_history (
     hist_id  integer AUTO_INCREMENT,
     h_c_id   integer,
     h_c_d_id integer,
     h_c_w_id integer,
     h_d_id   integer,
     h_w_id   integer,
     h_date   timestamp,
     h_amount decimal(6,2),
     h_data   varchar(24)
   )tablegroup='tpcc_group' partition by hash(h_w_id) partitions 128;
   
   create table bmsql_new_order (
     no_w_id  integer   not null ,
     no_d_id  integer   not null,
     no_o_id  integer   not null,
     PRIMARY KEY (no_w_id, no_d_id, no_o_id)
   )tablegroup='tpcc_group' partition by hash(no_w_id) partitions 128;
   
   create table bmsql_oorder (
     o_w_id       integer      not null,
     o_d_id       integer      not null,
     o_id         integer      not null,
     o_c_id       integer,
     o_carrier_id integer,
     o_ol_cnt     integer,
     o_all_local  integer,
     o_entry_d    timestamp,
     PRIMARY KEY (o_w_id, o_d_id, o_id)
   )tablegroup='tpcc_group' partition by hash(o_w_id) partitions 128;
   
   create table bmsql_order_line (
     ol_w_id         integer   not null,
     ol_d_id         integer   not null,
     ol_o_id         integer   not null,
     ol_number       integer   not null,
     ol_i_id         integer   not null,
     ol_delivery_d   timestamp,
     ol_amount       decimal(6,2),
     ol_supply_w_id  integer,
     ol_quantity     integer,
     ol_dist_info    char(24),
     PRIMARY KEY (ol_w_id, ol_d_id, ol_o_id, ol_number)
   )tablegroup='tpcc_group' partition by hash(ol_w_id) partitions 128;
   
   /* 当前 bmsql_item 表的 locality 针对三副本集群，具体参数根据实际部署修改 */
   create table bmsql_item (
     i_id     integer      not null,
     i_name   varchar(24),
     i_price  decimal(5,2),
     i_data   varchar(50),
     i_im_id  integer,
     PRIMARY KEY (i_id)
   ) locality='F,R{all_server}@zone1, F,R{all_server}@zone2, F,R{all_server}@zone3'  duplicate_scope='cluster';
   
   create table bmsql_stock (
     s_w_id       integer       not null,
     s_i_id       integer       not null,
     s_quantity   integer,
     s_ytd        integer,
     s_order_cnt  integer,
     s_remote_cnt integer,
     s_data       varchar(50),
     s_dist_01    char(24),
     s_dist_02    char(24),
     s_dist_03    char(24),
     s_dist_04    char(24),
     s_dist_05    char(24),
     s_dist_06    char(24),
     s_dist_07    char(24),
     s_dist_08    char(24),
     s_dist_09    char(24),
     s_dist_10    char(24),
     PRIMARY KEY (s_w_id, s_i_id)
   )tablegroup='tpcc_group' partition by hash(s_w_id) partitions 128;
   ```

   

   备份并重写benchmarksql-5.0/run/sql.common/tableDrops.sql。

   ```unknow
   drop table bmsql_config;
   
   drop table bmsql_new_order;
   
   drop table bmsql_order_line;
   
   drop table bmsql_oorder;
   
   drop table bmsql_history;
   
   drop table bmsql_customer;
   
   drop table bmsql_stock;
   
   drop table bmsql_item;
   
   drop table bmsql_district;
   
   drop table bmsql_warehouse;
   
   drop tablegroup tpcc_group;
   ```

   

   备份并重写 benchmarksql-5.0/run/sql.common/indexCreates.sql

   ```unknow
   create index bmsql_customer_idx1 on  bmsql_customer (c_w_id, c_d_id, c_last, c_first) local;
   create  index bmsql_oorder_idx1 on  bmsql_oorder (o_w_id, o_d_id, o_carrier_id, o_id) local;
   ```

   

   备份并重写 benchmarksql-5.0/run/sql.common/indexDrops.sql。

   ```unknow
   alter table bmsql_customer drop index bmsql_customer_idx1;
   alter table bmsql_oorder drop index bmsql_oorder_idx1;
   ```

   
   




环境调优 
-------------------------

1. ODP 调优，请在sys租户下执行。

   在系统租户下执行 `obclient -h$host_ip -P$host_port -uroot@sys -A` 命令。

   ```unknow
   （1）启动配置
   alter proxyconfig set enable_strict_kernel_release=false;
   alter proxyconfig set automatic_match_work_thread=false;
   （2）跑性能需要调整
   alter proxyconfig set proxy_mem_limited='4G'; --防止oom
   alter proxyconfig set enable_compression_protocol=false; --关闭压缩，降低cpu%
   alter proxyconfig set slow_proxy_process_time_threshold='500ms';
   alter proxyconfig set enable_ob_protocol_v2=false;
   alter proxyconfig set enable_qos=false;
   ```

   

   

2. OceanBase 数据库导数据阶段，请在sys租户下执行。

   在系统租户下执行 `obclient -h$host_ip -P$host_port -uroot@sys -A` 命令。

   ```unknow
   alter system set memory_chunk_cache_size ='16G';
   alter system set trx_try_wait_lock_timeout='0ms';
   alter system set large_query_threshold='1s';
   alter system set trace_log_slow_query_watermark='500ms';
   alter system set syslog_io_bandwidth_limit='30m';
   alter system set enable_async_syslog=true;
   alter system set merger_warm_up_duration_time='0';
   alter system set merger_switch_leader_duration_time='0';
   alter system set large_query_worker_percentage=10;
   alter system set builtin_db_data_verify_cycle = 0;
   alter system set enable_merge_by_turn = False;
   alter system set minor_merge_concurrency=30;
   alter system set memory_limit_percentage = 85;
   alter system set memstore_limit_percentage = 80;
   alter system set freeze_trigger_percentage = 30;
   alter system set enable_syslog_recycle='True';
   alter system set max_syslog_file_count=100;
   alter system set minor_freeze_times=500;
   alter system set minor_compact_trigger=5;
   alter system set max_kept_major_version_number=1;
   alter system set sys_bkgd_io_high_percentage = 90;
   alter system set sys_bkgd_io_low_percentage = 70;
   alter system set merge_thread_count  = 45;
   alter system set merge_stat_sampling_ratio = 1;
   alter system set writing_throttling_trigger_percentage=75 tenant=xxx;
   alter system set writing_throttling_maximum_duration='15m';
   set global ob_plan_cache_percentage=20;
   ```

   

3. 在observer部署之后，在进行测试TPCC的租户下执行。

   在测试用户下执行 `obclient -h$host_ip -P$host_port -u$user@$tenant -p$password -A` 命令。

   ```unknow
   alter system set _clog_aggregation_buffer_amount=8;
   alter system set _flush_clog_aggregation_buffer_timeout='1ms';
   ```

   

4. OceanBase 数据库压力测试阶段，请在sys租户下执行。

   在系统租户下执行 `obclient -h$host_ip -P$host_port -uroot@sys -A` 命令。

   ```unknow
   ##如果导入阶段开启了限速需要关闭
   alter system set writing_throttling_trigger_percentage=100 tenant=xxx;
   alter system set writing_throttling_maximum_duration='1h';
   alter system set memstore_limit_percentage = 80; 
   alter system set freeze_trigger_percentage = 30; 
   alter system set large_query_threshold = '200s';
   alter system set trx_try_wait_lock_timeout = '0ms';
   alter system set cpu_quota_concurrency = 4;
   alter system set minor_warm_up_duration_time = 0;
   alter system set minor_freeze_times=500;
   alter system set minor_compact_trigger=3;
   alter system set sys_bkgd_io_high_percentage = 90;
   alter system set sys_bkgd_io_low_percentage = 70;
   alter system set minor_merge_concurrency =20;
   alter system set builtin_db_data_verify_cycle = 0;
   alter system set trace_log_slow_query_watermark = '10s';
   alter system set gts_refresh_interval='500us'; 
   alter system set server_permanent_offline_time='36000s';
   alter system set weak_read_version_refresh_interval=0;
   alter system set  _ob_get_gts_ahead_interval = '5ms';
   ##为频繁空查的宏块建立bloomfilter并缓存，减少磁盘IO和CPU消耗，提升写入性能
   alter system set bf_cache_priority = 10;
   alter system set user_block_cache_priority=5;
   alter system set merge_stat_sampling_ratio = 0;
   ##close sql audit
   alter system set enable_sql_audit=false;
   ##调整日志级别及保存个数
   alter system set syslog_level='PERF';
   alter system set max_syslog_file_count=100;
   alter system set enable_syslog_recycle='True';
   alter system set ob_enable_batched_multi_statement=true tenant=all;
   alter system set _cache_wash_interval = '1m';
   alter system set plan_cache_evict_interval = '30s';
   alter system set enable_one_phase_commit=false;
   alter system set enable_monotonic_weak_read = false;
   ```

   

5. 设置租户。请在具体用户下执行。

   在测试用户下执行 `obclient -h$host_ip -P$host_port -u$user@$tenant -p$password -A` 命令。

   ```unknow
   数据库下租户设置，防止事务超时
   set global ob_query_timeout=36000000000;
   set global ob_trx_timeout=36000000000;
   set global max_allowed_packet=67108864;
   set global ob_sql_work_area_percentage=100;
   /*
   parallel_max_servers推荐设置为测试租户分配的resource unit cpu数的10倍
   如测试租户使用的unit配置为：create resource unit $unit_name max_cpu 26
   那么该值设置为260
   parallel_server_target推荐设置为parallel_max_servers * 机器数*0.8
   那么该值为260*3*0.8=624
   */
   set global parallel_max_servers=260;
   set global parallel_servers_target=624;
   ```

   






操作步骤 
-------------------------

以下命令均在 ../benchmarksql5/run 目录下执行。按照以下步骤进行 TPC-C 测试：

1. 运行以下命令，初始化环境：

   ```unknow
   ./runDatabaseDestroy.sh prop.oceanbase 
   ```

   

2. 运行以下命令，创建表并导入数据：

   ```unknow
    ./runDatabaseBuild.sh prop.oceanbase
   ```

   

3. 执行合并（需要使用sys租户登录）

   Major 合并将当前大版本的 SSTable 和 MemTable 与前一个大版本的全量静态数据进行合并，使存储层统计信息更准确，生成的执行计划更稳定。

   ```javascript
   MySQL [(none)]> use oceanbase
   Database changed
   MySQL [oceanbase]> alter system major freeze;
   Query OK, 0 rows affected
   ```

   

4. 查看合并是否完成

   ```unknow
   MySQL [oceanbase]> select name,value from oceanbase.__all_zone where name='frozen_version' or name='last_merged_version';
   +---------------------+-------+
   | name                | value |
   +---------------------+-------+
   | frozen_version      |     2 |
   | last_merged_version |     2 |
   | last_merged_version |     2 |
   | last_merged_version |     2 |
   | last_merged_version |     2 |
   +---------------------+-------+
   ```

   

   `frozen_version` 和 `last_merged_version` 的值相等即表示合并完成。
   

5. 运行以下命令，执行压力测试：

   ```unknow
   ./runBenchmark.sh prop.oceanbase
   ```

   




FAQ 
------------------------

* 如何停止 Benchmark SQL 工具。

  到期自动停止：依赖 prop.oceanbase 中 runMins 设置的值。

  强行停止：ps auxf \| grep java -- 删除Java进程删除 runBenchmark 进程。
  

* 找不到 main 类型。报错信息如下：

  ```unknow
  Could not find the main class: ExecJDBC.  Program will exit.
  ```

  

  可能原因有两个：要么升级 JDK，要么升级 JDBC 版本。运行以下命令安装 JDBC：

  ```unknow
  sudo yum install jdk
  ```

  

* 终端数量无效。报错信息如下：

  ```unknow
  Invalid number of terminals!
  ```

  

  这是 prop.oceanbase 中设置的 terminals 值不对，需填写 numTerminals \<= 0 \|\| numTerminals \> 10\*numWarehouses 范围内的 terminals 值。
  

* 事务超时。报错信息如下：

  ```unknow
  transaction timeout
  ```

  

  需增大超时时间，测试租户下执行set global ob_query_timeout=36000000000;set global ob_trx_timeout=36000000000。
  



