# 如何部署 OBAgent

## OBAgent 简介

OBAgent 是用 GO 语言开发的监控采集框架，通常部署在 OBServer 节点上。OBAgent 支持推、拉两种数据采集模式，可以满足不同的应用场景。OBAgent 默认支持的插件包括主机数据采集、OceanBase 数据库指标的采集、监控数据标签处理和 Prometheus 协议的 HTTP 服务。要使 OBAgent 支持其他数据源的采集，或者自定义数据的处理流程，您只需要开发对应的插件即可。

## 编辑 OBAgent 部署配置文件

OBAgent 部署配置文件可以跟 OceanBase 集群部署配置文件一起，也可以后期单独部署。附录 A.1 展示了同时部署 OceanBase 集群和 OBAgent。

下面示例是采用单独的配置文件部署 OBAgent 。OBAgent 的部署配置文件风格跟 OceanBase 集群部署配置文件一样。
首先是指定部署节点，包括节点名称和 IP 。节点名称保持唯一就行，可以是主机名（假设主机名是唯一的）。
然后指定全局配置。各个节点共同的配置都放在 `global` 节下。节点定制化的配置就不用放在这个下面。
然后指定各个节点定制化的配置。比如说每个节点的 `zone` 名称是不一样的。其他的根据实际情况填写。

```yaml
vim obagent-only.yaml

obagent:
  servers:
    - name: obce01
      # Please don't use hostname, only IP can be supported
      ip: 172.20.249.53
    - name: obce02
      ip: 172.20.249.55
    - name: obce03
      ip: 172.20.249.56
  global:
    # The working directory for obagent. obagent is started under this directory. This is a required field.
    home_path: /home/admin/obagent
    # The port that pulls and manages the metrics. The default port number is 8088.
    server_port: 8088
    # Debug port for pprof. The default port number is 8089.
    pprof_port: 8089
    sql_port: 2881
    rpc_port: 2882
    # Log level. The default value is INFO.
    log_level: INFO
    # Log path. The default value is log/monagent.log.
    log_path: log/monagent.log
    # Encryption method. OBD supports aes and plain. The default value is plain.
    crypto_method: plain
    # Path to store the crypto key. The default value is conf/.config_secret.key.
    # crypto_path: conf/.config_secret.key
    # Size for a single log file. Log size is measured in Megabytes. The default value is 30M.
    log_size: 30
    # Expiration time for logs. The default value is 7 days.
    log_expire_day: 7
    # The maximum number for log files. The default value is 10.
    log_file_count: 10
    # Whether to use local time for log files. The default value is true.
    # log_use_localtime: true
    # Whether to enable log compression. The default value is true.
    # log_compress: true
    # Username for HTTP authentication. The default value is admin.
    http_basic_auth_user: admin
    # Password for HTTP authentication. The default value is root.
    http_basic_auth_password: eIYf7NAZeT
    # Username for debug service. The default value is admin.
    pprof_basic_auth_user: admin
    # Password for debug service. The default value is root.
    pprof_basic_auth_password: eIYf7NAZeT

    # 以下配置必须与 OceanBase 数据库一致
    # Monitor username for OceanBase Database. The user must have read access to OceanBase Database as a system tenant. The default value is root.
    monitor_user: monitor
    # Monitor password for OceanBase Database. The default value is empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the root_password in oceanbase-ce.
    monitor_password: fLyaqjrp2R
    # Cluster name for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the appname in oceanbase-ce.
    cluster_name: obce-3zones
    # Cluster ID for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the cluster_id in oceanbase-ce.
    cluster_id: 1
  
  obce01:
    zone: zone1
  obce02:
    zone: zone2
  obce03:
    zone: zone3
```

注意：

+ 指定节点的连接端口用的是 `sql_port` 不是 `mysql_port` ，这点跟 OBSERVER 节点配置不一样。
+ 监控用户（`monitor_user`对应）和密码需要在 SYS 租户下创建。  `grant select on oceanbase.* to monitor identified by 'fLyaqjrp2R';` 。

## OBD 部署 OBAgent

第一次使用 `deploy` 命令，指定 OBAgent 的配置文件。

```bash
[admin@obce00 ~]$ obd cluster deploy obagent-only -c obagent-only.yaml
obagent-1.0.0 already installed.
+---------------------------------------------------------------------------+
|                                  Packages                                 |
+------------+---------+---------+------------------------------------------+
| Repository | Version | Release | Md5                                      |
+------------+---------+---------+------------------------------------------+
| obagent    | 1.0.0   | 2.el8   | 1d65fc3d2cd08b26d6142b6149eb6806260aa7db |
+------------+---------+---------+------------------------------------------+
Repository integrity check ok
Parameter check ok
Open ssh connection ok
Remote obagent-1.0.0-1d65fc3d2cd08b26d6142b6149eb6806260aa7db repository install ok
Remote obagent-1.0.0-1d65fc3d2cd08b26d6142b6149eb6806260aa7db repository lib check ok
Cluster status check ok
Initializes obagent work home ok
obagent-only deployed
[admin@obce00 ~]$

[admin@obce00 ~]$ obd cluster list
+------------------------------------------------------------------------+
|                              Cluster List                              |
+--------------+---------------------------------------+-----------------+
| Name         | Configuration Path                    | Status (Cached) |
+--------------+---------------------------------------+-----------------+
| obce-3zones  | /home/admin/.obd/cluster/obce-3zones  | running         |
| obagent-only | /home/admin/.obd/cluster/obagent-only | deployed        |
+--------------+---------------------------------------+-----------------+
```

上面 `deploy` 命令运行后，配置文件就被复制到 `~/.obd/cluster/obagent-only/config.yaml` 了。后续修改 `obagent-only.yaml` 文件就不会生效。此时可以采取 `edit-config` 编辑使用的配置文件，或者使用 `destroy` 命令清理部署，重新读取 `obagent-only.yaml` 开始部署。这个取决于改动的影响范围。

`deploy` 命令只是在各个节点上部署 OBAgent 软件（直接解压缩方式，不是 RPM 安装），目录如下：

```bash
[admin@obce01 ~]$ pwd
/home/admin
[admin@obce01 ~]$ tree obagent/
obagent/
├── bin
│   └── monagent -> /home/admin/.obd/repository/obagent/1.0.0/1d65fc3d2cd08b26d6142b6149eb6806260aa7db/bin/monagent
├── conf
│   ├── config_properties
│   │   ├── monagent_basic_auth.yaml
│   │   └── monagent_pipeline.yaml
│   ├── module_config
│   │   ├── monagent_basic_auth.yaml
│   │   ├── monagent_config.yaml
│   │   ├── monitor_node_host.yaml
│   │   └── monitor_ob.yaml
│   ├── monagent.yaml
│   └── prometheus_config
│       ├── prometheus.yaml
│       └── rules
│           ├── host_rules.yaml
│           └── ob_rules.yaml
├── lib
├── log
│   └── monagent.log
└── run

9 directories, 12 files
[admin@obce01 ~]$
```

## OBD 启动 OBAgent

启动命令是 `start` 。

```bash
[admin@obce00 ~]$ obd cluster start obagent-only
Get local repositories and plugins ok
Open ssh connection ok
Cluster param config check ok
Check before start obagent ok
obagent program health check ok
+---------------------------------------------------+
|                      obagent                      |
+---------------+-------------+------------+--------+
| ip            | server_port | pprof_port | status |
+---------------+-------------+------------+--------+
| 172.20.249.53 | 8088        | 8089       | active |
| 172.20.249.55 | 8088        | 8089       | active |
| 172.20.249.56 | 8088        | 8089       | active |
+---------------+-------------+------------+--------+
obagent-only running
[admin@obce00 ~]$
[admin@obce00 ~]$ obd cluster list
+------------------------------------------------------------------------+
|                              Cluster List                              |
+--------------+---------------------------------------+-----------------+
| Name         | Configuration Path                    | Status (Cached) |
+--------------+---------------------------------------+-----------------+
| obce-3zones  | /home/admin/.obd/cluster/obce-3zones  | running         |
| obagent-only | /home/admin/.obd/cluster/obagent-only | running         |
+--------------+---------------------------------------+-----------------+
[admin@obce00 ~]$
```

OBAgent 启动后有两个进程，其中进程 `moagent` 会监听指定端口。

```bash
[admin@obce01 ~]$ ps -ef|grep agent | grep -v grep
admin      90855       1  0 12:08 ?        00:00:00 /home/admin/obagent/bin/monagent -c conf/monagent.yaml
[admin@obce01 ~]$
[admin@obce01 ~]$ netstat -ntlp |grep 90855
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6       0      0 :::8088                 :::*                    LISTEN      90855/monagent
tcp6       0      0 :::8089                 :::*                    LISTEN      90855/monagent
[admin@obce01 ~]$

```

事后也可以通过 OBD 查看 OBAgent 部署情况。

```bash
[admin@obce00 ~]$ obd cluster display obagent-only
Get local repositories and plugins ok
Open ssh connection ok
Cluster status check ok
+---------------------------------------------------+
|                      obagent                      |
+---------------+-------------+------------+--------+
| ip            | server_port | pprof_port | status |
+---------------+-------------+------------+--------+
| 172.20.249.53 | 8088        | 8089       | active |
| 172.20.249.55 | 8088        | 8089       | active |
| 172.20.249.56 | 8088        | 8089       | active |
+---------------+-------------+------------+--------+
[admin@obce00 ~]$

```

## Prometheus 配置

OBAgent 启动后会在节点自动生成 `Prometheus` 配置文件, 位置在 OBAgent 安装目录下，如 `/home/admin/obagent/conf/prometheus_config/` 。这个配置文件可以给 `Prometheus` 产品直接使用。

示例如下：

```yaml
vim prometheus_config/prometheus.yaml

global:
  scrape_interval:     1s
  evaluation_interval: 10s

rule_files:
  - "rules/*rules.yaml"

scrape_configs:
  - job_name: prometheus
    metrics_path: /metrics
    scheme: http
    static_configs:
    - targets:
      - 'localhost:9090'
  - job_name: node
    basic_auth:
      username: admin
      password: eIYf7NAZeT
    metrics_path: /metrics/node/host
    scheme: http
    static_configs:
      - targets:
        - 172.20.249.53:8088
        - 172.20.249.55:8088
        - 172.20.249.56:8088
  - job_name: ob_basic
    basic_auth:
      username: admin
      password: eIYf7NAZeT
    metrics_path: /metrics/ob/basic
    scheme: http
    static_configs:
      - targets:
        - 172.20.249.53:8088
        - 172.20.249.55:8088
        - 172.20.249.56:8088
  - job_name: ob_extra
    basic_auth:
      username: admin
      password: eIYf7NAZeT
    metrics_path: /metrics/ob/extra
    scheme: http
    static_configs:
      - targets:
        - 172.20.249.53:8088
        - 172.20.249.55:8088
        - 172.20.249.56:8088
  - job_name: agent
    basic_auth:
      username: admin
      password: eIYf7NAZeT
    metrics_path: /metrics/stat
    scheme: http
    static_configs:
      - targets:
        - 172.20.249.53:8088
        - 172.20.249.55:8088
        - 172.20.249.56:8088
```

稍加说明如下：
| 配置项                 | 值                 | 说明     |
|---------------------|-------------------|--------|
| scrape_interval     | 1s                | 抓取间隔   |
| evaluation_interval | 10s               | 评估规则间隔 |
| rule_files          | rules/*rules.yaml | 报警规则   |
| scrape_configs      |                   | 抓取配置   |

下载解压缩 prometheus 后，启动方法

```bash
cd prometheus-2.30.3.linux-amd64 && ./prometheus ./prometheus.yaml
level=info ts=2021-11-19T05:41:57.789Z caller=main.go:400 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2021-11-19T05:41:57.789Z caller=main.go:438 msg="Starting Prometheus" version="(version=2.30.3, branch=HEAD, revision=f29caccc42557f6a8ec30ea9b3c8c089391bd5df)"
level=info ts=2021-11-19T05:41:57.789Z caller=main.go:443 build_context="(go=go1.17.1, user=root@5cff4265f0e3, date=20211005-16:10:52)"
level=info ts=2021-11-19T05:41:57.789Z caller=main.go:444 host_details="(Linux 3.10.0-1127.19.1.el7.x86_64 #1 SMP Tue Aug 25 17:23:54 UTC 2020 x86_64 obce00 (none))"
level=info ts=2021-11-19T05:41:57.789Z caller=main.go:445 fd_limits="(soft=65535, hard=65535)"
level=info ts=2021-11-19T05:41:57.789Z caller=main.go:446 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2021-11-19T05:41:57.791Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2021-11-19T05:41:57.792Z caller=main.go:822 msg="Starting TSDB ..."
level=info ts=2021-11-19T05:41:57.792Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=info ts=2021-11-19T05:41:57.795Z caller=head.go:479 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2021-11-19T05:41:57.795Z caller=head.go:513 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=7.579µs
level=info ts=2021-11-19T05:41:57.795Z caller=head.go:519 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2021-11-19T05:41:57.795Z caller=head.go:590 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0
level=info ts=2021-11-19T05:41:57.795Z caller=head.go:596 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=30.235µs wal_replay_duration=183.559µs total_replay_duration=238.771µs
level=info ts=2021-11-19T05:41:57.796Z caller=main.go:849 fs_type=EXT4_SUPER_MAGIC
level=info ts=2021-11-19T05:41:57.796Z caller=main.go:852 msg="TSDB started"
level=info ts=2021-11-19T05:41:57.796Z caller=main.go:979 msg="Loading configuration file" filename=prometheus.yaml
level=info ts=2021-11-19T05:41:57.802Z caller=main.go:1016 msg="Completed loading of configuration file" filename=prometheus.yaml totalDuration=6.363415ms db_storage=1.001µs remote_storage=4.404µs web_handler=3.04µs query_engine=886ns scrape=418.015µs scrape_sd=129.011µs notify=5.433µs notify_sd=4.872µs rules=5.284804ms
level=info ts=2021-11-19T05:41:57.802Z caller=main.go:794 msg="Server is ready to receive web requests."

```

启动后通过浏览器访问：[http://172.24.50.39:9090/graph](http://172.24.50.39:9090/graph) 。

具体 Prometheus 使用方法可以参考 [Prometheus 官方问答](https://prometheus.io/docs/introduction/overview/) 。

## OBAgent 重启方法

直接重启某个节点的 `OBAgent` 方法是：

```bash
kill -9 `pidof monagent`

cd /home/admin/obagent && nohup bin/monagent -c conf/monagent.yaml &
```

如果是集中重启，那就使用 OBD 命令：

```bash
obd cluster restart obagent_only
```

如果 `OBAgent` 是跟 `OceanBase` 一起部署的，那只能重启组件 `obagent` 。

```bash
obd cluster restart obce-3zones-obagent -c obagent

[admin@obce00 ~]$ obd cluster restart obce-3zones-obagent -c obagent
Get local repositories and plugins ok
Open ssh connection ok
Stop obagent ok
succeed
Get local repositories and plugins ok
Open ssh connection ok
Cluster param config check ok
Check before start obagent ok
obagent program health check ok
+--------------------------------------------------+
|                     obagent                      |
+--------------+-------------+------------+--------+
| ip           | server_port | pprof_port | status |
+--------------+-------------+------------+--------+
| 172.24.50.37 | 8088        | 8089       | active |
| 172.24.50.40 | 8088        | 8089       | active |
| 172.24.50.38 | 8088        | 8089       | active |
+--------------+-------------+------------+--------+
succeed

```

## Grafana 使用

首先请从 Grafana 官网下载最新版本，并安装启动。下载地址：[https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1](https://grafana.com/grafana/download?pg=get&plcmt=selfmanaged-box1-cta1) 。

然后在 Grafana 里新增 Datasource，填入 Prometheus 地址。

第三，从 Grafana 官网下载 OceanBase 提交的 主机性能模板和OceanBase 性能模板文件，文件是 json 格式。

+ [主机性能模板](https://grafana.com/grafana/dashboards/15216)
+ [OceanBase 性能模板](https://grafana.com/grafana/dashboards/15215)

下载到本机后，在Grafana 里 Import 这两个 json 文件。

## 附录

### A.1 OceanBase 和 OBAgent 一起的配置文件

```yaml
# Only need to configure when remote login is required
user:
   username: admin
#   password: your password if need
   key_file: /home/admin/.ssh/id_rsa.pub
   port: your ssh port, default 22
#   timeout: ssh connection timeout (second), default 30
oceanbase-ce:
  servers:
    - name: obce01
      # Please don't use hostname, only IP can be supported
      ip: 172.24.50.37
    - name: obce02
      ip: 172.24.50.40
    - name: obce03
      ip: 172.24.50.38
  global:
    # Please set devname as the network adaptor's name whose ip is  in the setting of severs.
    # if set severs as "127.0.0.1", please set devname as "lo"
    # if current ip is 192.168.1.10, and the ip's network adaptor's name is "eth0", please use "eth0"
    devname: eth0
    cluster_id: 2
    # please set memory limit to a suitable value which is matching resource.
    memory_limit: 8G # The maximum running memory for an observer
    system_memory: 3G # The reserved system memory. system_memory is reserved for general tenants. The default value is 30G.
    stack_size: 512K
    cpu_count: 16
    cache_wash_threshold: 1G
    __min_full_resource_pool_memory: 268435456
    workers_per_cpu_quota: 10
    schema_history_expire_time: 1d
    # The value of net_thread_count had better be same as cpu's core number.
    net_thread_count: 4
    major_freeze_duty_time: Disable
    minor_freeze_times: 10
    enable_separate_sys_clog: 0
    enable_merge_by_turn: FALSE
    #datafile_disk_percentage: 20 # The percentage of the data_dir space to the total disk space. This value takes effect only when datafile_size is 0. The default value is 90.
    datafile_size: 50G
    syslog_level: WARN # System log level. The default value is INFO.
    enable_syslog_wf: false # Print system logs whose levels are higher than WARNING to a separate log file. The default value is true.
    enable_syslog_recycle: true # Enable auto system log recycling or not. The default value is false.
    max_syslog_file_count: 10 # The maximum number of reserved log files before enabling auto recycling. The default value is 0.
    # observer cluster name, consistent with obproxy's cluster_name
    appname: obce-3zones
    root_password: 0EI5N08d # root user password, can be empty
    proxyro_password: uY7Yf8zx # proxyro user pasword, consistent with obproxy's observer_sys_password, can be empty
  obce01:
    mysql_port: 3881 # External port for OceanBase Database. The default value is 3881.
    rpc_port: 3882 # Internal port for OceanBase Database. The default value is 3882.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase-ce
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data/obce
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo/obce
    zone: zone1
  obce02:
    mysql_port: 3881 # External port for OceanBase Database. The default value is 3881.
    rpc_port: 3882 # Internal port for OceanBase Database. The default value is 3882.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase-ce
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data/obce
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo/obce
    zone: zone2
  obce03:
    mysql_port: 3881 # External port for OceanBase Database. The default value is 3881.
    rpc_port: 3882 # Internal port for OceanBase Database. The default value is 3882.
    #  The working directory for OceanBase Database. OceanBase Database is started under this directory. This is a required field.
    home_path: /home/admin/oceanbase-ce
    # The directory for data storage. The default value is $home_path/store.
    data_dir: /data/obce
    # The directory for clog, ilog, and slog. The default value is the same as the data_dir value.
    redo_dir: /redo/obce
    zone: zone3
obproxy:
  servers:
    - 172.24.50.39
    - 172.24.50.37
  # Set ependent components for the component.
  # When the associated configurations are not done, OBD will automatically get the these configurations from the dependent components.
  depends:
    - oceanbase-ce
  global:
    listen_port: 3883 # External port. The default value is 3883.
    prometheus_listen_port: 3884 # The Prometheus port. The default value is 3884.
    home_path: /home/admin/obproxy
    # oceanbase root server list
    # format: ip:mysql_port;ip:mysql_port
    rs_list: 172.24.50.37:3881;172.24.50.40:3881;172.24.50.38:3881
    enable_cluster_checkout: false
    # observer cluster name, consistent with oceanbase-ce's appname
    cluster_name: obce-3zones
    obproxy_sys_password: 0MdTv1tm # obproxy sys user password, can be empty
    observer_sys_password: uY7Yf8zx # proxyro user pasword, consistent with oceanbase-ce's proxyro_password, can be empty
obagent:
  servers:
    - name: obce01
      # Please don't use hostname, only IP can be supported
      ip: 172.24.50.37
    - name: obce02
      ip: 172.24.50.40
    - name: obce03
      ip: 172.24.50.38
  depends:
    - oceanbase-ce
  global:
    # The working directory for obagent. obagent is started under this directory. This is a required field.
    home_path: /home/admin/obagent
    # The port that pulls and manages the metrics. The default port number is 8088.
    server_port: 8088
    # Debug port for pprof. The default port number is 8089.
    pprof_port: 8089
    sql_port: 3881
    rpc_port: 3882
    # Log level. The default value is INFO.
    log_level: INFO
    # Log path. The default value is log/monagent.log.
    log_path: log/monagent.log
    # Encryption method. OBD supports aes and plain. The default value is plain.
    crypto_method: plain
    # Path to store the crypto key. The default value is conf/.config_secret.key.
    # crypto_path: conf/.config_secret.key
    # Size for a single log file. Log size is measured in Megabytes. The default value is 30M.
    log_size: 30
    # Expiration time for logs. The default value is 7 days.
    log_expire_day: 7
    # The maximum number for log files. The default value is 10.
    log_file_count: 10
    # Whether to use local time for log files. The default value is true.
    # log_use_localtime: true
    # Whether to enable log compression. The default value is true.
    # log_compress: true
    # Username for HTTP authentication. The default value is admin.
    http_basic_auth_user: admin
    # Password for HTTP authentication. The default value is root.
    http_basic_auth_password: eIYf7NAZeT
    # Username for debug service. The default value is admin.
    pprof_basic_auth_user: admin
    # Password for debug service. The default value is root.
    pprof_basic_auth_password: eIYf7NAZeT

    # 以下配置必须与 OceanBase 数据库一致
    # Monitor username for OceanBase Database. The user must have read access to OceanBase Database as a system tenant. The default value is root.
    monitor_user: monitor
    # Monitor password for OceanBase Database. The default value is empty. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the root_password in oceanbase-ce.
    monitor_password: fLyaqjrp2R
    # Cluster name for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the appname in oceanbase-ce.
    # cluster_name: obce-3zones
    # Cluster ID for OceanBase Database. When a depends exists, OBD gets this value from the oceanbase-ce of the depends. The value is the same as the cluster_id in oceanbase-ce.
    # cluster_id: 2
```

### A.2 `obagent` 输出的性能数据

示例数据。

```bash
[admin@obce00 ~]$ curl --user admin:eIYf7NAZeT -L 'http://172.24.50.40:8088/metrics/ob/basic'
# HELP ob_active_session_num monitor collected metric
# TYPE ob_active_session_num untyped
ob_active_session_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 6
# HELP ob_cache_size_bytes monitor collected metric
# TYPE ob_cache_size_bytes untyped
ob_cache_size_bytes{app="OB",cache_name="location_cache",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 4.193216e+06
ob_cache_size_bytes{app="OB",cache_name="user_tab_col_stat_cache",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 6.290304e+06
ob_cache_size_bytes{app="OB",cache_name="user_table_stat_cache",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 6.290304e+06
# HELP ob_partition_num monitor collected metric
# TYPE ob_partition_num untyped
ob_partition_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",role="1",svr_ip="172.24.50.40",tenant_name="sys"} 1198
# HELP ob_plan_cache_access_total monitor collected metric
# TYPE ob_plan_cache_access_total untyped
ob_plan_cache_access_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 22984
# HELP ob_plan_cache_hit_total monitor collected metric
# TYPE ob_plan_cache_hit_total untyped
ob_plan_cache_hit_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 8645
# HELP ob_plan_cache_memory_bytes monitor collected metric
# TYPE ob_plan_cache_memory_bytes untyped
ob_plan_cache_memory_bytes{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 1.3404239e+07
# HELP ob_server_num monitor collected metric
# TYPE ob_server_num untyped
ob_server_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",server_ips="172.24.50.37,172.24.50.38,172.24.50.40",status="active",svr_ip="172.24.50.40"} 3
# HELP ob_sysstat monitor collected metric
# TYPE ob_sysstat untyped
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 47136
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 3.0078186e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 46863
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4.291008e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} -2.050408e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="10006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4096
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.59442e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.8080384e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4.0265315e+08
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="130004",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.0530635e+08
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 1.610612736e+09
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 3.81681664e+08
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 500
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="140006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 1
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="20001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4122
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="20002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 104938
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 15330
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="30006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 7.566136e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 463
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 1.863916e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40004",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40006",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40007",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40008",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40009",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40010",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 108
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40011",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8339
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="40012",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 329
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50008",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="50009",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60000",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60001",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60002",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60003",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 4
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60004",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 43325
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60005",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 8.388608e+06
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60019",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60020",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60021",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60022",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60023",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="60024",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="80040",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 11980
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="80041",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 2.0937394e+07
ob_sysstat{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",stat_id="80057",svr_ip="172.24.50.40",tenant_id="1",tenant_name="sys"} 0
# HELP ob_table_num monitor collected metric
# TYPE ob_table_num untyped
ob_table_num{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 1013
# HELP ob_waitevent_wait_seconds_total monitor collected metric
# TYPE ob_waitevent_wait_seconds_total untyped
ob_waitevent_wait_seconds_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 49578.6026
# HELP ob_waitevent_wait_total monitor collected metric
# TYPE ob_waitevent_wait_total untyped
ob_waitevent_wait_total{app="OB",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",tenant_name="sys"} 787473
# HELP ob_zone_current_timestamp monitor collected metric
# TYPE ob_zone_current_timestamp untyped
ob_zone_current_timestamp{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="cluster",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="config_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="frozen_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="gc_schema_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="global_broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_error",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="lease_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="privilege_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="proposal_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="snapshot_gc_ts",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_format_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="time_zone_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="try_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="warm_up_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303541e+15
ob_zone_current_timestamp{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303541e+15
# HELP ob_zone_stat monitor collected metric
# TYPE ob_zone_stat untyped
ob_zone_stat{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1
ob_zone_stat{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1
ob_zone_stat{app="OB",name="all_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1
ob_zone_stat{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1
ob_zone_stat{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1
ob_zone_stat{app="OB",name="broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1
ob_zone_stat{app="OB",name="cluster",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="config_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.63730315985165e+15
ob_zone_stat{app="OB",name="frozen_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="gc_schema_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="global_broadcast_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="idc",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="is_merge_error",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="is_merge_timeout",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="is_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303150866399e+15
ob_zone_stat{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303150867716e+15
ob_zone_stat{app="OB",name="last_merged_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303150868863e+15
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1
ob_zone_stat{app="OB",name="last_merged_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1
ob_zone_stat{app="OB",name="lease_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1.63730315985492e+15
ob_zone_stat{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 1.637303150866399e+15
ob_zone_stat{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 1.637303150867716e+15
ob_zone_stat{app="OB",name="merge_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 1.637303150868863e+15
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="merge_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="privilege_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="proposal_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="recovery_status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="region",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="snapshot_gc_ts",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 2
ob_zone_stat{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 2
ob_zone_stat{app="OB",name="status",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 2
ob_zone_stat{app="OB",name="storage_format_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 4
ob_zone_stat{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="storage_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="suspend_merging",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
ob_zone_stat{app="OB",name="time_zone_info_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="try_frozen_version",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 1
ob_zone_stat{app="OB",name="warm_up_start_time",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone=""} 0
ob_zone_stat{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone1"} 0
ob_zone_stat{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone2"} 0
ob_zone_stat{app="OB",name="zone_type",ob_cluster_id="2",ob_cluster_name="obce-3zones",obzone="zone1",svr_ip="172.24.50.40",zone="zone3"} 0
```
