Canal详细入门实战(使用总结)

wylc123 1年前 ⋅ 203 阅读

Canal介绍

Canal简介

canal [kə'næl]，译意为水道/管道/沟渠，主要用途是基于 MySQL 数据库增量日志解析，提供增量数据订阅和消费

早期阿里巴巴因为杭州和美国双机房部署，存在跨机房同步的业务需求，实现方式主要是基于业务 trigger 获取增量变更。从 2010 年开始，业务逐步尝试数据库日志解析获取增量变更进行同步，由此衍生出了大量的数据库增量订阅和消费业务。

基于日志增量订阅和消费的业务包括

  • 数据库镜像
  • 数据库实时备份
  • 索引构建和实时维护(拆分异构索引、倒排索引等)
  • 业务 cache 刷新
  • 带业务逻辑的增量数据处理

当前的 canal 支持源端 MySQL 版本包括 5.1.x , 5.5.x , 5.6.x , 5.7.x , 8.0.x

MySQL工作原理

  • MySQL master 将数据变更写入二进制日志( binary log, 其中记录叫做二进制日志事件binary log events，可以通过 show binlog events 进行查看)
  • MySQL slave 将 master 的 binary log events 拷贝到它的中继日志(relay log)
  • MySQL slave 重放 relay log 中事件，将数据变更反映它自己的数据

Canal工作原理

  • canal 模拟 MySQL slave 的交互协议，伪装自己为 MySQL slave ，向 MySQL master 发送dump 协议
  • MySQL master 收到 dump 请求，开始推送 binary log 给 slave (即 canal )
  • canal 解析 binary log 对象(原始为 byte 流)



Canal使用准备

MySQL准备

1、对于自建 MySQL , 需要先开启 Binlog 写入功能，配置 binlog-format 为 ROW 模式，my.cnf 中配置如下

[mysqld] log-bin=mysql-bin # 开启binlog binlog-format=ROW # 选择ROW模式 server_id=1 # 配置MySQL replaction需要定义，不要和canal的slaveId重复

2、授权 canal 链接 MySQL 账号具有作为 MySQL slave 的权限, 如果已有账户可直接 grant

-- 使用命令登录：mysql -u root -p -- 创建mysql用户 用户名：canal 密码：canal CREATE USER canal IDENTIFIED BY 'canal'; GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'canal'@'%'; -- GRANT ALL PRIVILEGES ON *.* TO 'canal'@'%' ; FLUSH PRIVILEGES;

3、重启数据库，查看配置是否生效

mysql> show variables like 'binlog_format'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | binlog_format | ROW | +---------------+-------+
mysql> show variables like 'log_bin'; +---------------+-------+ | Variable_name | Value | +---------------+-------+ | log_bin | ON | +---------------+-------+

Canal使用准备

Canal下载，这里使用1.1.5版本为例

github链接：https://github.com/alibaba/canal/releases

需要下载(其他可根据自行需要进行下载)：

  • canal.deployer-1.1.5.tar.gz
  • canal.adapter-1.1.5.tar.gz

下载后解压

1.1.5解压后目录结构如下

bin ：canal启动、重启、停止文件 conf ：canal配置文件 lib ：canal运行所需的jar包，注意数据库驱动包版本，可手动更换 logs ：canal运行日志 plugin ：一些扩展包，如消息队列

创建数据库和表

踩坑记录：数据库名不要使用下划线，一开始用canal_test_01的方式命名，发现扫描不到数据库

-- 源数据库：canal01 CREATE DATABASE `canal01` CHARACTER SET 'utf8mb4'; -- 创建user01表 CREATE TABLE `user01` ( `id` int(64) NOT NULL AUTO_INCREMENT, `username` varchar(64) DEFAULT NULL, `password` varchar(64) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4; -- 目标数据库：canal02 CREATE DATABASE `canal02` CHARACTER SET 'utf8mb4'; -- 创建user02表 CREATE TABLE `user02` ( `id` int(64) NOT NULL AUTO_INCREMENT, `username` varchar(64) DEFAULT NULL, `password` varchar(64) DEFAULT NULL, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;



Canal-deployer使用

1、conf\canal.properties配置文件(这里只介绍我用到的配置)

# canal-deployer的ip地址，不配置默认是本机 canal.ip = # canal-deployer的端口号 canal.port = 11111 # 模式 canal.serverMode = tcp # 加载多个配置文件，需在canal.deployer-1.1.5\conf下创建对应的文件夹，并加上instance.properties配置文件 canal.destinations = example # 配置多个使用英文逗号隔开 # canal.destinations = example,example2

2、conf\example\instance.properties配置文件(这里只介绍我用到的配置)

# 不能与my.ini下的server_id=1相同 canal.instance.mysql.slaveId=1234 # mysql地址和端口 canal.instance.master.address=127.0.0.1:3306 # 数据库账号密码，可以使用root账号或刚创建的canal账号 canal.instance.dbUsername=root canal.instance.dbPassword=root canal.instance.connectionCharset=UTF-8 # 配置监听，支持正则表达式 canal.instance.filter.regex=canal01\\..* # 配置不监听，支持正则表达式 canal.instance.filter.black.regex=mysql\\.slave_.*

3、检查lib目录下的jar包，一定要有对应的数据库驱动包，且版本要对得上

4、在bin目录中启动canal-deployer，启动成功如下输出

# 运行后输出 Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=128m; support was removed in 8.0 Listening for transport dt_socket at address: 9099 # 在logs\canal目录下，会记录canal.log # 有如下记录表示canal-deployer运行成功 2021-04-28 09:26:57.194 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## set default uncaught exception handler 2021-04-28 09:26:57.244 [main] INFO com.alibaba.otter.canal.deployer.CanalLauncher - ## load canal configurations 2021-04-28 09:26:57.257 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## start the canal server. 2021-04-28 09:26:57.399 [main] INFO com.alibaba.otter.canal.deployer.CanalController - ## start the canal server[192.168.3.51(192.168.3.51):11111] 2021-04-28 09:26:59.440 [main] INFO com.alibaba.otter.canal.deployer.CanalStarter - ## the canal server is running now ...... # 在logs\example目录下，会记录example.log # 有如下记录表示canal-deployer与数据库连接成功了 2021-04-28 09:30:37.159 [main] INFO c.a.otter.canal.instance.spring.CanalInstanceWithSpring - start CannalInstance for 1-example 2021-04-28 09:30:37.175 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table filter : ^fooddb\..*$ 2021-04-28 09:30:37.175 [main] WARN c.a.o.canal.parse.inbound.mysql.dbsync.LogEventConvert - --> init table black filter : ^mysql\.slave_.*$ 2021-04-28 09:30:37.270 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position 2021-04-28 09:30:37.288 [main] INFO c.a.otter.canal.instance.core.AbstractCanalInstance - start successful.... 2021-04-28 09:30:37.305 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position just last position {"identity":{"slaveId":-1,"sourceAddress":{"address":"ieonline.microsoft.com","port":3306}},"postion":{"gtid":"","included":false,"journalName":"binlog.000068","position":14636,"serverId":1,"timestamp":1619408047000}} 2021-04-28 09:30:37.744 [destination = example , address = /127.0.0.1:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=binlog.000068,position=14636,serverId=1,gtid=,timestamp=1619408047000] cost : 466ms , the next step is binlog dump
 
 

5、创建springboot工程监听canal-deployer(可跳过)

导入maven坐标(pom.xml)

<!--Canal--> <dependency> <groupId>com.alibaba.otter</groupId> <artifactId>canal.client</artifactId> <version>1.1.4</version> </dependency>

创建CanalClient类，并交给spring管理

import java.net.InetSocketAddress; import java.util.List; import com.alibaba.otter.canal.client.CanalConnectors; import com.alibaba.otter.canal.client.CanalConnector; import com.alibaba.otter.canal.protocol.Message; import com.alibaba.otter.canal.protocol.CanalEntry.Column; import com.alibaba.otter.canal.protocol.CanalEntry.Entry; import com.alibaba.otter.canal.protocol.CanalEntry.EntryType; import com.alibaba.otter.canal.protocol.CanalEntry.EventType; import com.alibaba.otter.canal.protocol.CanalEntry.RowChange; import com.alibaba.otter.canal.protocol.CanalEntry.RowData; import org.springframework.beans.factory.InitializingBean; import org.springframework.stereotype.Component; @Component public class CanalClient implements InitializingBean { private final static int BATCH_SIZE = 1000; @Override public void afterPropertiesSet() throws Exception { // 创建链接 CanalConnector connector = CanalConnectors.newSingleConnector(new InetSocketAddress("127.0.0.1", 11111), "example", "", ""); try { //打开连接 connector.connect(); //订阅数据库表,全部表 connector.subscribe(".*\\..*"); //回滚到未进行ack的地方，下次fetch的时候，可以从最后一个没有ack的地方开始拿 connector.rollback(); while (true) { // 获取指定数量的数据 Message message = connector.getWithoutAck(BATCH_SIZE); //获取批量ID long batchId = message.getId(); //获取批量的数量 int size = message.getEntries().size(); //如果没有数据 if (batchId == -1 || size == 0) { try { //线程休眠2秒 Thread.sleep(2000); } catch (InterruptedException e) { e.printStackTrace(); } } else { //如果有数据,处理数据 printEntry(message.getEntries()); } //进行 batch id 的确认。确认之后，小于等于此 batchId 的 Message 都会被确认。 connector.ack(batchId); } } catch (Exception e) { e.printStackTrace(); } finally { connector.disconnect(); } } /** * 打印canal server解析binlog获得的实体类信息 */ private static void printEntry(List<Entry> entrys) { for (Entry entry : entrys) { if (entry.getEntryType() == EntryType.TRANSACTIONBEGIN || entry.getEntryType() == EntryType.TRANSACTIONEND) { //开启/关闭事务的实体类型，跳过 continue; } //RowChange对象，包含了一行数据变化的所有特征 //比如isDdl 是否是ddl变更操作 sql 具体的ddl sql beforeColumns afterColumns 变更前后的数据字段等等 RowChange rowChage; try { rowChage = RowChange.parseFrom(entry.getStoreValue()); } catch (Exception e) { throw new RuntimeException("ERROR ## parser of eromanga-event has an error , data:" + entry.toString(), e); } //获取操作类型：insert/update/delete类型 EventType eventType = rowChage.getEventType(); //打印Header信息 System.out.println(String.format("================》; binlog[%s:%s] , name[%s,%s] , eventType : %s", entry.getHeader().getLogfileName(), entry.getHeader().getLogfileOffset(), entry.getHeader().getSchemaName(), entry.getHeader().getTableName(), eventType)); //判断是否是DDL语句 if (rowChage.getIsDdl()) { System.out.println("================》;isDdl: true,sql:" + rowChage.getSql()); } //获取RowChange对象里的每一行数据，打印出来 for (RowData rowData : rowChage.getRowDatasList()) { //如果是删除语句 if (eventType == EventType.DELETE) { printColumn(rowData.getBeforeColumnsList()); //如果是新增语句 } else if (eventType == EventType.INSERT) { printColumn(rowData.getAfterColumnsList()); //如果是更新的语句 } else { //变更前的数据 System.out.println("------->; before"); printColumn(rowData.getBeforeColumnsList()); //变更后的数据 System.out.println("------->; after"); printColumn(rowData.getAfterColumnsList()); } } } } private static void printColumn(List<Column> columns) { for (Column column : columns) { System.out.println(column.getName() + " : " + column.getValue() + " update=" + column.getUpdated()); } } }
 
 

启动springboot项目，在user01表中添加数据，控制台如下输出，监听成功

================》; binlog[binlog.000070:5154] , name[canal01,user01] , eventType : INSERT id : 1 update=true username : a update=true password : a update=true



Canal-adapter使用

1、conf\application.yml配置文件(这里只介绍我用到的配置)

server: port: 8081 # canal-adapter运行端口号 canal.conf: mode: tcp # canal client的模式: tcp kafka rocketMQ rabbitMQ consumerProperties: # canal tcp consumer canal.tcp.server.host: 127.0.0.1:11111 canal.tcp.zookeeper.hosts: canal.tcp.batch.size: 500 canal.tcp.username: canal.tcp.password: srcDataSources: # 源数据库 canal01: # 自定义名称 # 这里使用的是mysql8.0，需要加上serverTimezone=UTC url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true username: root password: root # 配置多个源数据库 # canal02: # url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true # username: root # password: root canalAdapters: # 适配器列表 - instance: example # canal 实例名或者 MQ topic 名 groups: # 分组列表 - groupId: # 分组id, 如果是MQ模式将用到该值 outerAdapters: # 分组内适配器列表 # - name: logger # 日志打印适配器 - name: rdb # 指定为rdb类型同步 key: mysql1 # 指定adapter的唯一key, 与表映射配置中outerAdapterKey对应 # 目标数据库配置 properties: jdbc.driverClassName: com.mysql.jdbc.Driver jdbc.url: jdbc:mysql://127.0.0.1:3306/canal02?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true jdbc.username: root jdbc.password: root threads: 5 # 并行执行的线程数,可以不修改，默认为1 # 配置多个目标数据库 # - instance: example2 # canal instance Name or mq topic name # groups: # - groupId: g2 # outerAdapters: # - name: rdb # key: mysql2 # properties: # jdbc.driverClassName: com.mysql.jdbc.Driver # jdbc.url: jdbc:mysql://127.0.0.1:3306/canal01?characterEncoding=utf8&useSSL=false&serverTimezone=UTC&rewriteBatchedStatements=true # jdbc.username: root # jdbc.password: root
 
 

2、配置RDB表映射文件(conf/rdb/mytest_user.yml文件)

dataSourceKey: canal01 # 源数据源的key, 对应上面配置的srcDataSources中的值 destination: example # cannal的instance或者MQ的topic groupId: g1 # 对应MQ模式下的groupId, 只会同步对应groupId的数据 outerAdapterKey: mysql1 # adapter key, 对应上面配置outAdapters中的key concurrent: true # 是否按主键hash并行同步, 并行同步的表必须保证主键不会更改及主键不能为其他同步表的外键!! dbMapping: database: canal01 # 源数据源的database/shcema table: user01 # 源数据源表名 targetTable: user02 # 直接写表名 targetPk: # 主键映射 id: id # 如果是复合主键可以换行映射多个 mapAll: true # 是否整表映射, 要求源表和目标表字段名一模一样 (如果targetColumns也配置了映射,则以targetColumns配置为准) # targetColumns: # 字段映射, 格式: 目标表字段: 源表字段, 如果字段名一样源表字段名可不填 # id: # name: # role_id: # c_time: # test1: etlCondition: "where c_time>={}" commitBatch: 3000 # 批量提交的大小
 
 

3、检查lib目录下的jar包，一定要有对应的数据库驱动包，且版本要对得上

4、在bin目录中启动canal-adapter，启动成功如下输出

# 运行后输出 2021-04-28 11:12:38.281 [main] INFO o.s.b.w.s.c.AnnotationConfigServletWebServe rApplicationContext - Refreshing org.springframework.boot.web.servlet.context.An notationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.Annotation ConfigApplicationContext@1a677343 2021-04-28 11:12:39.012 [main] INFO org.springframework.cloud.context.scope.Gen ericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039 2021-04-28 11:12:39.108 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$B eanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.Configur ationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.aut oconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGL IB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (f or example: not eligible for auto-proxying) 2021-04-28 11:12:39.610 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServe r - Tomcat initialized with port(s): 8081 (http) 2021-04-28 11:12:39.632 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"] 2021-04-28 11:12:39.646 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat] 2021-04-28 11:12:39.647 [main] INFO org.apache.catalina.core.StandardEngine - S tarting Servlet Engine: Apache Tomcat/8.5.29 2021-04-28 11:12:39.660 [localhost-startStop-1] INFO org.apache.catalina.core.A prLifecycleListener - The APR based Apache Tomcat Native library which allows op timal performance in production environments was not found on the java.library.p ath: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\sys tem32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Wind ows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java \jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\ bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Com mon;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\User s\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\Je tBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4 \bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C :\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.] 2021-04-28 11:12:39.830 [localhost-startStop-1] INFO o.a.catalina.core.Containe rBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationCont ext 2021-04-28 11:12:39.831 [localhost-startStop-1] INFO org.springframework.web.co ntext.ContextLoader - Root WebApplicationContext: initialization completed in 15 50 ms 2021-04-28 11:12:39.990 [localhost-startStop-1] INFO o.s.boot.web.servlet.Servl etRegistrationBean - Servlet dispatcherServlet mapped to [/] 2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte rRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*] 2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte rRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*] 2021-04-28 11:12:40.001 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte rRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*] 2021-04-28 11:12:40.002 [localhost-startStop-1] INFO o.s.boot.web.servlet.Filte rRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*] Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SP I and manual loading of the driver class is generally unnecessary. 2021-04-28 11:12:40.597 [main] INFO com.alibaba.druid.pool.DruidDataSource - {d ataSource-1} inited 2021-04-28 11:12:40.849 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap ping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springf ramework.web.servlet.resource.ResourceHttpRequestHandler] 2021-04-28 11:12:41.143 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.serv let.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 GMT+08:00 2021]; parent: org.springframework.context.a nnotation.AnnotationConfigApplicationContext@1a677343 2021-04-28 11:12:41.245 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto publi c java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adap ter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.S tring) 2021-04-28 11:12:41.247 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java .util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.la uncher.rest.CommonRest.count(java.lang.String,java.lang.String) 2021-04-28 11:12:41.248 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" on to public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otte r.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String) 2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapt er.launcher.rest.CommonRest.etl(java.lang.String) 2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.cana l.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.la ng.String,java.lang.String) 2021-04-28 11:12:41.250 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.a libaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adap ter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.Str ing) 2021-04-28 11:12:41.251 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.L ist<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.a dapter.launcher.rest.CommonRest.destinations() 2021-04-28 11:12:41.256 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/error]}" onto public org.springframework.http.Respons eEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.b oot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.htt p.HttpServletRequest) 2021-04-28 11:12:41.258 [main] INFO o.s.w.s.m.method.annotation.RequestMappingH andlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springf ramework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.ser vlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse) 2021-04-28 11:12:41.298 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap ping - Mapped URL path [/webjars/**] onto handler of type [class org.springframe work.web.servlet.resource.ResourceHttpRequestHandler] 2021-04-28 11:12:41.299 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMap ping - Mapped URL path [/**] onto handler of type [class org.springframework.web .servlet.resource.ResourceHttpRequestHandler] 2021-04-28 11:12:41.604 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx porter - Registering beans for JMX exposure on startup 2021-04-28 11:12:41.618 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx porter - Bean with name 'refreshScope' has been autodetected for JMX exposure 2021-04-28 11:12:41.620 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx porter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure 2021-04-28 11:12:41.622 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx porter - Bean with name 'environmentManager' has been autodetected for JMX expos ure 2021-04-28 11:12:41.625 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx porter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager, type=EnvironmentManager] 2021-04-28 11:12:41.644 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx porter - Located managed bean 'refreshScope': registering with JMX server as MBe an [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=Refre shScope] 2021-04-28 11:12:41.659 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanEx porter - Located managed bean 'configurationPropertiesRebinder': registering wit h JMX server as MBean [org.springframework.cloud.context.properties:name=configu rationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder] 2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd apterService - ## syncSwitch refreshed. 2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd apterService - ## start the canal client adapters. 2021-04-28 11:12:41.678 [main] INFO c.a.otter.canal.client.adapter.support.Exte nsi"hljs-string">D:\Darren\source\FoodMS\FoodMS-Docs\Tools \Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin 2021-04-28 11:12:41.777 [main] INFO c.a.otter.canal.client.adapter.rdb.config.C onfigLoader - ## Start loading rdb mapping config ... 2021-04-28 11:12:41.837 [main] INFO c.a.otter.canal.client.adapter.rdb.config.C onfigLoader - ## Rdb mapping config loaded 2021-04-28 11:12:42.224 [main] INFO com.alibaba.druid.pool.DruidDataSource - {d ataSource-2} inited 2021-04-28 11:12:42.234 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd apterLoader - Load canal adapter: rdb succeed 2021-04-28 11:12:42.246 [main] INFO c.alibaba.otter.canal.connector.core.spi.Ex tensi"hljs-string">D:\Darren\source\FoodMS\FoodMS-Docs\Too ls\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin 2021-04-28 11:12:42.275 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd apterLoader - Start adapter for canal-client mq topic: example-g1 succeed 2021-04-28 11:12:42.276 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader .AdapterProcessor - =============> Start to connect destination: example <====== ======= 2021-04-28 11:12:42.276 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAd apterService - ## the canal client adapters are running now ...... 2021-04-28 11:12:42.286 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"] 2021-04-28 11:12:42.296 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read 2021-04-28 11:12:42.316 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServe r - Tomcat started on port(s): 8081 (http) with context path '' 2021-04-28 11:12:42.320 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapt erApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running fo r 6.186) 2021-04-28 11:12:42.354 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader .AdapterProcessor - =============> Subscribe destination: example succeed <===== ======== # logs\adapter目录下，会记录adapter.log # 有如下记录表示canal-adapter成功运行了 2021-04-28 11:12:37.612 [main] INFO o.s.c.annotation.AnnotationConfigApplicationContext - Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343: startup date [Wed Apr 28 11:12:37 CST 2021]; root of context hierarchy 2021-04-28 11:12:38.027 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'configurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2021-04-28 11:12:38.254 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - No active profile set, falling back to default profiles: default 2021-04-28 11:12:38.281 [main] INFO o.s.b.w.s.c.AnnotationConfigServletWebServerApplicationContext - Refreshing org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 CST 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343 2021-04-28 11:12:39.012 [main] INFO org.springframework.cloud.context.scope.GenericScope - BeanFactory id=ba9c0aec-0105-3f1f-b89e-e85c68567039 2021-04-28 11:12:39.108 [main] INFO o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker - Bean 'org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration' of type [org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$60a96eee] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 2021-04-28 11:12:39.610 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat initialized with port(s): 8081 (http) 2021-04-28 11:12:39.632 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Initializing ProtocolHandler ["http-nio-8081"] 2021-04-28 11:12:39.646 [main] INFO org.apache.catalina.core.StandardService - Starting service [Tomcat] 2021-04-28 11:12:39.647 [main] INFO org.apache.catalina.core.StandardEngine - Starting Servlet Engine: Apache Tomcat/8.5.29 2021-04-28 11:12:39.660 [localhost-startStop-1] INFO org.apache.catalina.core.AprLifecycleListener - The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [C:\ProgramData\Oracle\Java\javapath;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\ProgramData\Oracle\Java\javapath;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Java\jdk1.8.0_131\bin;C:\Java\jdk1.8.0_131\jre\bin;D:\Tools\JAVA\apache-maven-3.6.3\bin;C:\Program Files\Git\cmd;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\go\bin;C:\Program Files\nodejs\;C:\Program Files (x86)\Yarn\bin\;;C:\Users\Administrator\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files\JetBrains\WebStorm 2019.3.1\bin;;C:\Program Files\JetBrains\IntelliJ IDEA 2019.3.4\bin;;C:\Users\Administrator\go\bin;C:\Users\Administrator\AppData\Roaming\npm;C:\Users\Administrator\AppData\Local\Yarn\bin;D:\Tools\zookeeper-3.4.13\bin;;.] 2021-04-28 11:12:39.830 [localhost-startStop-1] INFO o.a.catalina.core.ContainerBase.[Tomcat].[localhost].[/] - Initializing Spring embedded WebApplicationContext 2021-04-28 11:12:39.831 [localhost-startStop-1] INFO org.springframework.web.context.ContextLoader - Root WebApplicationContext: initialization completed in 1550 ms 2021-04-28 11:12:39.990 [localhost-startStop-1] INFO o.s.boot.web.servlet.ServletRegistrationBean - Servlet dispatcherServlet mapped to [/] 2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'characterEncodingFilter' to: [/*] 2021-04-28 11:12:40.000 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'hiddenHttpMethodFilter' to: [/*] 2021-04-28 11:12:40.001 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'httpPutFormContentFilter' to: [/*] 2021-04-28 11:12:40.002 [localhost-startStop-1] INFO o.s.boot.web.servlet.FilterRegistrationBean - Mapping filter: 'requestContextFilter' to: [/*] 2021-04-28 11:12:40.597 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited 2021-04-28 11:12:40.849 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**/favicon.ico] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler] 2021-04-28 11:12:41.143 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerAdapter - Looking for @ControllerAdvice: org.springframework.boot.web.servlet.context.AnnotationConfigServletWebServerApplicationContext@2ef14fe: startup date [Wed Apr 28 11:12:38 GMT+08:00 2021]; parent: org.springframework.context.annotation.AnnotationConfigApplicationContext@1a677343 2021-04-28 11:12:41.245 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{key}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String,java.lang.String) 2021-04-28 11:12:41.247 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/count/{type}/{task}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.Object> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.count(java.lang.String,java.lang.String) 2021-04-28 11:12:41.248 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}/{status}],methods=[PUT]}" onto public com.alibaba.otter.canal.client.adapter.support.Result com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String) 2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/syncSwitch/{destination}],methods=[GET]}" onto public java.util.Map<java.lang.String, java.lang.String> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String) 2021-04-28 11:12:41.249 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{key}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String,java.lang.String) 2021-04-28 11:12:41.250 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/etl/{type}/{task}],methods=[POST]}" onto public com.alibaba.otter.canal.client.adapter.support.EtlResult com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.etl(java.lang.String,java.lang.String,java.lang.String) 2021-04-28 11:12:41.251 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/destinations],methods=[GET]}" onto public java.util.List<java.util.Map<java.lang.String, java.lang.String>> com.alibaba.otter.canal.adapter.launcher.rest.CommonRest.destinations() 2021-04-28 11:12:41.256 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error]}" onto public org.springframework.http.ResponseEntity<java.util.Map<java.lang.String, java.lang.Object>> org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.error(javax.servlet.http.HttpServletRequest) 2021-04-28 11:12:41.258 [main] INFO o.s.w.s.m.method.annotation.RequestMappingHandlerMapping - Mapped "{[/error],produces=[text/html]}" onto public org.springframework.web.servlet.ModelAndView org.springframework.boot.autoconfigure.web.servlet.error.BasicErrorController.errorHtml(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) 2021-04-28 11:12:41.298 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/webjars/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler] 2021-04-28 11:12:41.299 [main] INFO o.s.web.servlet.handler.SimpleUrlHandlerMapping - Mapped URL path [/**] onto handler of type [class org.springframework.web.servlet.resource.ResourceHttpRequestHandler] 2021-04-28 11:12:41.604 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Registering beans for JMX exposure on startup 2021-04-28 11:12:41.618 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'refreshScope' has been autodetected for JMX exposure 2021-04-28 11:12:41.620 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'configurationPropertiesRebinder' has been autodetected for JMX exposure 2021-04-28 11:12:41.622 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Bean with name 'environmentManager' has been autodetected for JMX exposure 2021-04-28 11:12:41.625 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'environmentManager': registering with JMX server as MBean [org.springframework.cloud.context.environment:name=environmentManager,type=EnvironmentManager] 2021-04-28 11:12:41.644 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'refreshScope': registering with JMX server as MBean [org.springframework.cloud.context.scope.refresh:name=refreshScope,type=RefreshScope] 2021-04-28 11:12:41.659 [main] INFO o.s.jmx.export.annotation.AnnotationMBeanExporter - Located managed bean 'configurationPropertiesRebinder': registering with JMX server as MBean [org.springframework.cloud.context.properties:name=configurationPropertiesRebinder,context=2ef14fe,type=ConfigurationPropertiesRebinder] 2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## syncSwitch refreshed. 2021-04-28 11:12:41.675 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## start the canal client adapters. 2021-04-28 11:12:41.678 [main] INFO c.a.otter.canal.client.adapter.support.Extensi"hljs-string">D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin 2021-04-28 11:12:41.777 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Start loading rdb mapping config ... 2021-04-28 11:12:41.837 [main] INFO c.a.otter.canal.client.adapter.rdb.config.ConfigLoader - ## Rdb mapping config loaded 2021-04-28 11:12:42.224 [main] INFO com.alibaba.druid.pool.DruidDataSource - {dataSource-2} inited 2021-04-28 11:12:42.234 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: rdb succeed 2021-04-28 11:12:42.246 [main] INFO c.alibaba.otter.canal.connector.core.spi.Extensi"hljs-string">D:\Darren\source\FoodMS\FoodMS-Docs\Tools\Canal\canal.adapter-1.1.5\canal.adapter-1.1.5\plugin 2021-04-28 11:12:42.275 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: example-g1 succeed 2021-04-28 11:12:42.276 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: example <============= 2021-04-28 11:12:42.276 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ...... 2021-04-28 11:12:42.286 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"] 2021-04-28 11:12:42.296 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read 2021-04-28 11:12:42.316 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path '' 2021-04-28 11:12:42.320 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 5.287 seconds (JVM running for 6.186) 2021-04-28 11:12:42.354 [Thread-4] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: example succeed <=============
 
 

5、向user01表中添加数据，user02表也有对应数据，同步成功

2021-04-28 11:27:59.619 [pool-2-thread-1] DEBUG c.a.o.canal.client.adapter.rdb.s ervice.RdbSyncService - DML: {"data":{"id":1,"username":"a","password":"a"},"dat abase":"canal01","destination":"example","old":null,"table":"user01","type":"INS ERT"}

如需user02同步到user01：

​ （1）添加对应的canal-deployer中添加对应的example

​ （2）canal-adapter的application.yml配置文件中加入对应的源数据库和目标数据库(具体见上面配置)

​ （3）新建RDB表映射文件，在rdb目录下，canal-adapter会默认扫描rdb目录下所有的yml文件

同步多张表：

​ （1）在两个数据库中增加表（这里以person为例）

​ （2）配置RDB表映射文件，将mytest_user.yml复制一份，重命名为mytest_person.yml

​ （3）修改mytest_person.yml中的table(源表)、targetTable(目标表)

6、如果需要使用Java工程，可自行在github的canal中下载canal-adapter的工程，配置与上面完全一样


注意：本文归作者所有，未经作者允许，不得转载

#canal #数据同步 #字段映射
阅读全部
相关文章推荐
  • 1

全部评论: 0 条

    我有话说: