update 4.11.md v1 (#710)
This commit is contained in:
askdb
GitHub
parent
c979869b2f
commit
2a36347832
@ -1,3 +1,275 @@
|
||||
# 如何使用 LOGPROXY 将 OB 数据实时同步到 MySQL
|
||||
|
||||
<待补充>
|
||||
## 什么是 CDC
|
||||
CDC 全称是 Change Data Capture，即变更数据捕获。
|
||||
|
||||
## 为什么需要 CDC 功能
|
||||
CDC 能够帮助您识别从上次提取之后发生变化的数据。您可用 CDC 提供的数据做很多事情，比如：做历史库、做近实时缓存、提供给消息队列（MQ），用户消费 MQ 做分析和审计等。
|
||||
|
||||
|
||||
## OceanBase 里 CDC 是如何实现的
|
||||
|
||||
OblogProxy 是 OceanBase CE 的增量日志代理服务。基于 liboblog, 以服务的形式，提供实时增量链路接入和管理能力，方便应用接入 OceanBase 增量日志；能够解决网络隔离的情况下，订阅增量日志的需求；并提供多种链路接入方式。
|
||||
Canal 是一个开源的 MySQL 数据库 Binlog 的增量订阅和消费组件，基于 MySQL 数据库的增量日志解析，可以用于数据同步。
|
||||
|
||||
|
||||
数据链路：
|
||||
```
|
||||
ob_cluster -> oblogreader ->oblogmsg -> canal_server -> canal_client -> MySQL
|
||||
```
|
||||
|
||||
|
||||
组件介绍：
|
||||
+ liboblog 是 OceanBase CDC 的基本组件，liboblog 衍生出 oblogproxy，liboblog 依赖 oblogmsg。
|
||||
|
||||
+ liboblog : C++ 动态库，将从 OceanBase 集群拉到的增量日志按事务提交顺序向外透出，透出的方式是按 logmessage 协议创建相关对象给使用方使用。
|
||||
|
||||
+ Oblogreader : 对 liboblog 的 C++ 封装，在 liboblog 的基础上，对创建出的 logmessage 对象适配多种序列化输出.
|
||||
|
||||
+ logproxy 增量日志代理，并不对增量数据进行转发，而只是对新建连接请求创建对应的oblogreader子进程，并由该oblogreader 子进程通过该连接直接向客户端发送按 logmessage 序列化后的增量日志记录。logproxy 用于对同机器上所有 ObLogreader 子进程的生命周期进行管理。
|
||||
|
||||
+ logclient ：对字节流按 logmessage 协议反序列化后生成 logmessage 对象给使用方调用,比如和 canal 对接。
|
||||
|
||||
|
||||
## 安装 oblogproxy
|
||||
|
||||
+ 下载 oblogproxy rpm 包
|
||||
https://github.com/oceanbase/oblogproxy/releases/tag/v1.0.0
|
||||
|
||||
+ 安装 oblogproxy
|
||||
yum install oblogproxy-1.0.0-1.el7.x86_64.rpm
|
||||
|
||||
+ 确认 oblogproxy 依赖,重点是确认有 liboblog
|
||||
```shell
|
||||
[root@172.30.199.49 ~]$cd /usr/local/oblogproxy/
|
||||
[root@172.30.199.49 oblogproxy]$ldd ./bin/logproxy
|
||||
linux-vdso.so.1 => (0x00007fffe68fe000)
|
||||
liboblog.so.1 => /lib/liboblog.so.1 (0x00002ae0113f1000)
|
||||
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002ae046bf8000)
|
||||
libdl.so.2 => /lib64/libdl.so.2 (0x00002ae046e14000)
|
||||
librt.so.1 => /lib64/librt.so.1 (0x00002ae047018000)
|
||||
libm.so.6 => /lib64/libm.so.6 (0x00002ae047220000)
|
||||
libc.so.6 => /lib64/libc.so.6 (0x00002ae047522000)
|
||||
/lib64/ld-linux-x86-64.so.2 (0x00002ae0111cd000)
|
||||
libaio.so.1 => /lib64/libaio.so.1 (0x00002ae0478f0000)
|
||||
[root@172.30.199.49 oblogproxy]$
|
||||
```
|
||||
|
||||
## 启动 oblogproxy
|
||||
### 修改 oblogproxy 参数
|
||||
|
||||
+ 生成加密用户和对应的密码
|
||||
```
|
||||
[root@172.30.199.49 bin]$pwd
|
||||
/usr/local/oblogproxy/bin
|
||||
[root@172.30.199.49 bin]$./logproxy -x root@sys
|
||||
EA87898018FD1EDDC2AA11CE1556E917
|
||||
[root@172.30.199.49 bin]$
|
||||
[root@172.30.199.49 bin]$./logproxy -x Root@2021
|
||||
8852DB95B3D54B1A8F866836A9D8FD52
|
||||
[root@172.30.199.49 bin]$
|
||||
```
|
||||
|
||||
+ 修改对应的 conf.json 参数
|
||||
```
|
||||
[root@172.30.199.49 conf]$pwd
|
||||
/usr/local/oblogproxy/conf
|
||||
[root@172.30.199.49 conf]$ls -l
|
||||
总用量 4
|
||||
-rw-r--r-- 1 root root 1081 10月 25 18:00 conf.json
|
||||
[root@172.30.199.49 conf]$
|
||||
修改conf.json的内容
|
||||
"ob_sys_username": "EA87898018FD1EDDC2AA11CE1556E917",
|
||||
"ob_sys_password": "8852DB95B3D54B1A8F866836A9D8FD52",
|
||||
```
|
||||
|
||||
+ 启动 oblogproxy
|
||||
```
|
||||
[root@172.30.199.49 oblogproxy]$./run.sh start
|
||||
work path : /usr/local/oblogproxy
|
||||
is_running : (8252)/usr/local/oblogproxy logproxy is running !
|
||||
logproxy started!
|
||||
[root@172.30.199.49 oblogproxy]$
|
||||
[root@172.30.199.49 oblogproxy]$
|
||||
```
|
||||
|
||||
+ 启停 obproxy 和查看状态
|
||||
启动oblogproxy
|
||||
```
|
||||
[root@172.30.199.49 oblogproxy]$pwd
|
||||
/usr/local/oblogproxy
|
||||
[root@172.30.199.49 oblogproxy]$./run.sh start
|
||||
```
|
||||
|
||||
+ 关闭 oblogproxy
|
||||
```
|
||||
[root@172.30.199.49 oblogproxy]$pwd
|
||||
/usr/local/oblogproxy
|
||||
[root@172.30.199.49 oblogproxy]$./run.sh stop
|
||||
```
|
||||
|
||||
+ 查看 oblogproxy 状态
|
||||
```
|
||||
[root@172.30.199.49 oblogproxy]$pwd
|
||||
/usr/local/oblogproxy
|
||||
[root@172.30.199.49 oblogproxy]$./run.sh status
|
||||
```
|
||||
|
||||
+ 说明：
|
||||
oblogproxy 启动成功后进程确认
|
||||
```
|
||||
[root@172.30.199.49 ~]$ps -ef | grep logproxy | grep -v grep
|
||||
root 26379 26373 0 10:45 pts/1 00:00:00 ./bin/logproxy -f ./conf/conf.json
|
||||
[root@172.30.199.49 ~]$
|
||||
```
|
||||
|
||||
+ 当有一个 client 连接成功后会 fork 一个子进程
|
||||
```
|
||||
[root@172.30.199.49 ~]$ps -ef | grep oblogreader | grep -v grep
|
||||
root 26386 26379 2 10:45 pts/1 00:00:17 oblogreader -f ./conf/conf.json
|
||||
[root@172.30.199.49 ~]$
|
||||
```
|
||||
|
||||
## canal server
|
||||
### canal server
|
||||
|
||||
修改 /opt/canal_ob/canal.deployer-for-ob-rc1/conf/canal.properties
|
||||
```yaml
|
||||
canal.zkServers = 172.30.199.47:12181,172.30.199.48:12181,172.30.199.49:12181
|
||||
canal.serverMode = tcp
|
||||
canal.destinations = obtest2
|
||||
canal.instance.global.spring.xml = classpath:spring/ob-default-instance.xml
|
||||
```
|
||||
|
||||
### canal instance
|
||||
|
||||
```
|
||||
[root@172.30.199.49 conf]$ll
|
||||
总用量 28
|
||||
-rwxr-xr-x 1 root root 319 10月 21 16:15 canal_local.properties
|
||||
-rwxr-xr-x 1 root root 6577 10月 26 16:21 canal.properties
|
||||
-rwxr-xr-x 1 root root 3592 10月 21 16:15 logback.xml
|
||||
drwxrwxrwx 2 root root 4096 10月 21 16:15 metrics
|
||||
drwxr-xr-x 2 root root 4096 12月 7 13:29 obtest2
|
||||
drwxrwxrwx 3 root root 4096 11月 2 10:51 spring
|
||||
[root@172.30.199.49 conf]$pwd
|
||||
/opt/canal_ob/canal.deployer-for-ob-rc1/conf
|
||||
[root@172.30.199.49 conf]$cat obtest2/instance.properties
|
||||
```
|
||||
|
||||
```yaml
|
||||
# oceanbase集群参数
|
||||
canal.instance.oceanbase.rsList=172.30.199.49:2882:2881;172.30.199.47:2882:2881
|
||||
canal.instance.oceanbase.username=root@test_tenant_1#obcluster
|
||||
canal.instance.oceanbase.password=Root@2021
|
||||
canal.instance.oceanbase.startTimestamp=0
|
||||
|
||||
# oceanbase logproxy参数
|
||||
canal.instance.oceanbase.logproxy.address=127.0.0.1:2983
|
||||
canal.instance.oceanbase.logproxy.sslEnabled=false
|
||||
canal.instance.oceanbase.logproxy.serverCert=../conf/${canal.instance.destination:}/ca.crt
|
||||
canal.instance.oceanbase.logproxy.clientCert=../conf/${canal.instance.destination:}/client.crt
|
||||
canal.instance.oceanbase.logproxy.clientKey=../conf/${canal.instance.destination:}/client.key
|
||||
|
||||
# 是否要在库名中去掉租户前缀。logproxy输出的日志中库名默认为 [tenant].[db]
|
||||
canal.instance.parser.excludeTenantInDbName=true
|
||||
canal.instance.oceanbase.tenant=test_tenant_1
|
||||
|
||||
# 日志过滤。格式为[tenant].[database].[table]，支持正则
|
||||
canal.instance.filter.regex=test_tenant_1.db1.*
|
||||
```
|
||||
|
||||
+ 启动 canal server
|
||||
```
|
||||
[root@172.30.199.49 ~]$cd /opt/canal_ob
|
||||
[root@172.30.199.49 canal_ob]$cd canal.deployer-for-ob-rc1
|
||||
[root@172.30.199.49 canal.deployer-for-ob-rc1]$cd bin
|
||||
[root@172.30.199.49 bin]$ls -lrt
|
||||
总用量 20
|
||||
-rwxr-xr-x 1 root root 1244 10月 21 16:15 startup.bat
|
||||
-rwxr-xr-x 1 root root 1356 10月 21 16:15 stop.sh
|
||||
-rwxr-xr-x 1 root root 3167 10月 21 16:15 startup.sh
|
||||
-rwxr-xr-x 1 root root 226 10月 21 16:15 restart.sh
|
||||
-rw-r--r-- 1 root root 6 12月 7 10:42 canal.pid
|
||||
[root@172.30.199.49 bin]$./startup.sh
|
||||
```
|
||||
|
||||
|
||||
## canal client
|
||||
|
||||
```
|
||||
[root@172.30.199.49 conf]$pwd
|
||||
/opt/canal_ob/canal_adapter_for_ob/conf
|
||||
[root@172.30.199.49 conf]$ls -lrt
|
||||
总用量 44
|
||||
-rwxr-xr-x 1 root root 2172 10月 21 16:15 logback.xml
|
||||
drwxrwxrwx 2 root root 4096 10月 21 16:15 tablestore
|
||||
drwxr-xr-x 2 root root 4096 10月 21 16:15 hbase
|
||||
drwxr-xr-x 2 root root 4096 10月 21 16:15 kudu
|
||||
drwxrwxrwx 2 root root 4096 10月 21 16:15 META-INF
|
||||
drwxr-xr-x 2 root root 4096 10月 21 16:15 es6
|
||||
-rwxr-xr-x 1 root root 170 10月 21 16:15 bootstrap.yml
|
||||
-rwxr-xr-x 1 root root 552 10月 21 16:15 mytest_user.yml
|
||||
drwxr-xr-x 2 root root 4096 10月 21 16:15 es7
|
||||
-rwxr-xr-x 1 root root 3240 10月 26 16:26 application.yml
|
||||
drwxrwxrwx 2 root root 4096 12月 7 10:27 rdb
|
||||
```
|
||||
|
||||
[root@172.30.199.49 conf]$cat application.yml
|
||||
|
||||
+ 修改的部分如下：
|
||||
```yaml
|
||||
canalAdapters:
|
||||
- instance: obtest2 # canal instance Name or mq topic name
|
||||
groups:
|
||||
- groupId: g1
|
||||
outerAdapters:
|
||||
- name: logger
|
||||
- name: rdb
|
||||
key: mysql1
|
||||
properties:
|
||||
jdbc.driverClassName: com.mysql.jdbc.Driver
|
||||
jdbc.url: jdbc:mysql://172.30.199.49:3367/mysql?useUnicode=true&useSSL=false
|
||||
jdbc.username: root
|
||||
jdbc.password: Root@2021
|
||||
```
|
||||
|
||||
|
||||
+ 修改适配器库/表映射，这里已库映射为例：
|
||||
|
||||
```
|
||||
[root@172.30.199.49 conf]$pwd
|
||||
/opt/canal_ob/canal_adapter_for_ob/conf
|
||||
[root@172.30.199.49 conf]$ls -l rdb
|
||||
总用量 4
|
||||
-rwxr-xr-x 1 root root 186 12月 6 21:11 mytest_user.yml
|
||||
[root@172.30.199.49 conf]$
|
||||
```
|
||||
|
||||
```yaml
|
||||
## Mirror schema synchronize config
|
||||
dataSourceKey: defaultDS
|
||||
destination: obtest2
|
||||
groupId: g1
|
||||
outerAdapterKey: mysql1
|
||||
concurrent: true
|
||||
dbMapping:
|
||||
mirrorDb: true
|
||||
database: db1
|
||||
```
|
||||
|
||||
+ 启动 canal client
|
||||
|
||||
```
|
||||
[root@172.30.199.49 bin]$pwd
|
||||
/opt/canal_ob/canal_adapter_for_ob/bin
|
||||
[root@172.30.199.49 bin]$ls -lrt
|
||||
总用量 16
|
||||
-rwxr-xr-x 1 root root 2289 10月 21 16:15 startup.sh
|
||||
-rwxr-xr-x 1 root root 793 10月 21 16:15 startup.bat
|
||||
-rwxr-xr-x 1 root root 205 10月 21 16:15 restart.sh
|
||||
-rwxr-xr-x 1 root root 1370 10月 21 16:15 stop.sh
|
||||
[root@172.30.199.49 bin]$./startup.sh
|
||||
```
|
||||
|
||||
模拟在oceanbase源端写入数据，在mysql目标端查看是否有数据同步过来。
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user