搭建 dataX前置条件JDK(1.8以上推荐1.8)Python(2或3都可以)Apache Maven 3.x (Compile DataX)以上配置好系统环境变量下载 datax 编译好的包https://datax-opensource.oss-cn-hangzhou.aliyuncs.com/202309/datax.tar.gz进入目录使用 powershell 打开执行解压命令tar -xzf datax.tar.gz得到配置datax根目录core文件不要改这是公共加载配置配置 windows 命令行支持 utf-8不配置这个命令执行同步命令会出现中文乱码先把日志“乱码”变成可读中文最简单两步即可临时改当前窗口编码打开cmd先执行chcp 65001再运行python datax.py C:\Users\longz\Downloads\1.json日志就会以 UTF-8 输出中文正常显示。永久生效可选在 Windows 10/11 上可以把系统全局控制台编码改成 UTF-8设置 → 时间和语言 → 区域 →相关设置→管理语言设置→更改系统区域设置→ 勾选“Beta: 使用 Unicode UTF-8 提供全球语言支持”→ 重启。之后所有 cmd/PowerShell 窗口默认就是 UTF-8不会再出现方块或乱码。或者直接在控制面板找把如下的复选框勾上重启准备测试数据新建两个库相同的表名userCREATE TABLE IF NOT EXISTS user ( id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, username VARCHAR(50) NOT NULL, password VARCHAR(100) NOT NULL, email VARCHAR(100) NOT NULL, created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ) ENGINEInnoDB DEFAULT CHARSETutf8mb4;在 test1 库新增测试数据。我这里用存储过程直接调用即可生成测试数据。DELIMITER$$CREATEPROCEDUREGenerateUserTestData(INnum_recordsINT)BEGINDECLAREiINTDEFAULT1;DECLAREv_usernameVARCHAR(50);DECLAREv_passwordVARCHAR(100);DECLAREv_emailVARCHAR(100);WHILEinum_recordsDO-- 生成随机用户名如 User_12345SETv_usernameCONCAT(User_,FLOOR(10000RAND()*90000));-- 生成随机密码简化示例实际应加密SETv_passwordCONCAT(Pass_,FLOOR(100000RAND()*900000));-- 生成随机邮箱如 user12345example.comSETv_emailCONCAT(user,FLOOR(10000RAND()*90000),example.com);INSERTINTOuser(username,password,email)VALUES(v_username,v_password,v_email);-- 每1000条提交一次避免事务过大IFi%10000THENCOMMIT;ENDIF;SETii1;ENDWHILE;END$$DELIMITER;生成完成后目标使用 datax 同步 test1 库的数据到 test2 库新建 datax 的同步脚本{job:{content:[{reader:{name:mysqlreader,parameter:{username:root,password:root,connection:[{jdbcUrl:[jdbc:mysql://localhost:3306/test1?useUnicodetruecharacterEncodingutf8serverTimezoneAsia/ShanghaiuseSSLfalseconnectTimeout60000socketTimeout300000],table:[user]}],column:[*],splitPk:id}},writer:{name:mysqlwriter,parameter:{username:root,password:root,connection:[{jdbcUrl:jdbc:mysql://localhost:3306/test2?useUnicodetruecharacterEncodingutf8serverTimezoneAsia/ShanghaiuseSSLfalserewriteBatchedStatementstruesocketTimeout300000,table:[user]}],column:[*],writeMode:insert}}}],setting:{speed:{channel:5}}}}执行同步命令python datax.py D:\Information_Technology\worksapce_tool\datax-new\datax\job\test-user.json执行完成D:\Information_Technology\worksapce_tool\datax-new\datax\binpython datax.py D:\Information_Technology\worksapce_tool\datax-new\datax\job\test-user.json DataX(DATAX-OPENSOURCE-3.0), From Alibaba!Copyright(C)2010-2017, Alibaba Group. All Rights Reserved.2025-08-0814:00:36.524[main]INFO MessageSource - JVM TimeZone: GMT08:00, Locale: zh_CN2025-08-0814:00:36.526[main]INFO MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[idGMT08:00,offset28800000,dstSavings0,useDaylightfalse,transitions0,lastRulenull]2025-08-0814:00:36.536[main]INFO VMInfo - VMInfo# operatingSystem class sun.management.OperatingSystemImpl2025-08-0814:00:36.541[main]INFO Engine - the machine infoosInfo: Windows10amd6410.0jvmInfo: Oracle Corporation1.825.341-b10 cpu num:16totalPhysicalMemory:-0.00GfreePhysicalMemory:-0.00GmaxFileDescriptorCount:-1currentOpenFileDescriptorCount:-1GC Names[PS MarkSweep, PS Scavenge]MEMORY_NAME|allocation_size|init_size PS Eden Space|256.00MB|256.00MB Code Cache|240.00MB|2.44MB Compressed Class Space|1,024.00MB|0.00MB PS Survivor Space|42.50MB|42.50MB PS Old Gen|683.00MB|683.00MB Metaspace|-0.00MB|0.00MB2025-08-0814:00:36.552[main]INFO Engine -{content:[{reader:{name:mysqlreader,parameter:{username:root,password:****,connection:[{jdbcUrl:[jdbc:mysql://localhost:3306/test1],table:[user]}],column:[*],splitPk:}},writer:{name:mysqlwriter,parameter:{username:root,password:****,connection:[{jdbcUrl:jdbc:mysql://localhost:3306/test2,table:[user]}],column:[*],writeMode:insert}}}],setting:{speed:{channel:1}}}2025-08-0814:00:36.565[main]INFO PerfTrace - PerfTracetraceIdjob_-1,isEnablefalse2025-08-0814:00:36.566[main]INFO JobContainer - DataX jobContainer starts job.2025-08-0814:00:36.566[main]INFO JobContainer - Set jobId0Fri Aug 0814:00:36 GMT08:002025WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property issettofalse.You need either to explicitly disable SSL by settinguseSSLfalse, orsetuseSSLtrue and provide truststoreforserver certificate verification.2025-08-0814:00:42.909[job-0]INFO OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:mysql://localhost:3306/test1?yearIsDateTypefalsezeroDateTimeBehaviorconvertToNulltinyInt1isBitfalserewriteBatchedStatementstrue.2025-08-0814:00:42.910[job-0]WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列当您的表字段个数、类型有变动时可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改. Fri Aug 0814:00:43 GMT08:002025WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property issettofalse.You need either to explicitly disable SSL by settinguseSSLfalse, orsetuseSSLtrue and provide truststoreforserver certificate verification.2025-08-0814:00:43.091[job-0]INFO OriginalConfPretreatmentUtil - table:[user]all columns:[id,username,password,email,created_at].2025-08-0814:00:43.091[job-0]WARN OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*当您的表字段个数、类型有变动时可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.2025-08-0814:00:43.092[job-0]INFO OriginalConfPretreatmentUtil - Write data[insert INTO %s(id,username,password,email,created_at)VALUES(?,?,?,?,?)],whichjdbcUrl like:[jdbc:mysql://localhost:3306/test2?yearIsDateTypefalsezeroDateTimeBehaviorconvertToNullrewriteBatchedStatementstruetinyInt1isBitfalse]2025-08-0814:00:43.093[job-0]INFO JobContainer - jobContainer starts todoprepare...2025-08-0814:00:43.093[job-0]INFO JobContainer - DataX Reader.Job[mysqlreader]doprepare work.2025-08-0814:00:43.093[job-0]INFO JobContainer - DataX Writer.Job[mysqlwriter]doprepare work.2025-08-0814:00:43.094[job-0]INFO JobContainer - jobContainer starts todosplit...2025-08-0814:00:43.094[job-0]INFO JobContainer - JobsetChannel-Number to1channels.2025-08-0814:00:43.097[job-0]INFO JobContainer - DataX Reader.Job[mysqlreader]splits to[1]tasks.2025-08-0814:00:43.097[job-0]INFO JobContainer - DataX Writer.Job[mysqlwriter]splits to[1]tasks.2025-08-0814:00:43.115[job-0]INFO JobContainer - jobContainer starts todoschedule...2025-08-0814:00:43.117[job-0]INFO JobContainer - Scheduler starts[1]taskGroups.2025-08-0814:00:43.119[job-0]INFO JobContainer - Running by standalone Mode.2025-08-0814:00:43.124[taskGroup-0]INFO TaskGroupContainer -taskGroupId[0]start[1]channelsfor[1]tasks.2025-08-0814:00:43.126[taskGroup-0]INFO Channel - Channelsetbyte_speed_limit to -1, No bps activated.2025-08-0814:00:43.127[taskGroup-0]INFO Channel - Channelsetrecord_speed_limit to -1, No tps activated.2025-08-0814:00:43.136[taskGroup-0]INFO TaskGroupContainer - taskGroup[0]taskId[0]attemptCount[1]is started2025-08-0814:00:43.140[0-0-0-reader]INFO CommonRdbmsReader$Task- Begin toreadrecord by Sql:[select * from user]jdbcUrl:[jdbc:mysql://localhost:3306/test1?yearIsDateTypefalsezeroDateTimeBehaviorconvertToNulltinyInt1isBitfalserewriteBatchedStatementstrue]. Fri Aug 0814:00:43 GMT08:002025WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property issettofalse.You need either to explicitly disable SSL by settinguseSSLfalse, orsetuseSSLtrue and provide truststoreforserver certificate verification. Fri Aug 0814:00:43 GMT08:002025WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property issettofalse.You need either to explicitly disable SSL by settinguseSSLfalse, orsetuseSSLtrue and provide truststoreforserver certificate verification. Fri Aug 0814:00:43 GMT08:002025WARN: Establishing SSL connection without servers identity verification is not recommended. According to MySQL 5.5.45, 5.6.26 and 5.7.6 requirements SSL connection must be established by default if explicit option isnt set. For compliance with existing applications not using SSL the verifyServerCertificate property issettofalse.You need either to explicitly disable SSL by settinguseSSLfalse, orsetuseSSLtrue and provide truststoreforserver certificate verification.2025-08-0814:00:43.179[0-0-0-reader]INFO CommonRdbmsReader$Task- Finishedreadrecord by Sql:[select * from user]jdbcUrl:[jdbc:mysql://localhost:3306/test1?yearIsDateTypefalsezeroDateTimeBehaviorconvertToNulltinyInt1isBitfalserewriteBatchedStatementstrue].2025-08-0814:00:43.456[taskGroup-0]INFO TaskGroupContainer - taskGroup[0]taskId[0]is successed, used[321]ms2025-08-0814:00:43.456[taskGroup-0]INFO TaskGroupContainer - taskGroup[0]completed its tasks.2025-08-0814:00:53.142[job-0]INFO StandAloneJobContainerCommunicator - Total100records,5192bytes|Speed 519B/s,10records/s|Error0records,0bytes|All Task WaitWriterTime0.000s|All Task WaitReaderTime0.000s|Percentage100.00%2025-08-0814:00:53.142[job-0]INFO AbstractScheduler - Scheduler accomplished all tasks.2025-08-0814:00:53.142[job-0]INFO JobContainer - DataX Writer.Job[mysqlwriter]dopost work.2025-08-0814:00:53.143[job-0]INFO JobContainer - DataX Reader.Job[mysqlreader]dopost work.2025-08-0814:00:53.143[job-0]INFO JobContainer - DataX jobId[0]completed successfully.2025-08-0814:00:53.144[job-0]INFO HookInvoker - No hook invoked, because basedirnot exists or is a file: D:\Information_Technology\worksapce_tool\datax-new\datax\hook2025-08-0814:00:53.144[job-0]INFO JobContainer -[total cpu info]averageCpu|maxDeltaCpu|minDeltaCpu -1.00%|-1.00%|-1.00%[total gc info]NAME|totalGCCount|maxDeltaGCCount|minDeltaGCCount|totalGCTime|maxDeltaGCTime|minDeltaGCTime PS MarkSweep|1|1|1|0.016s|0.016s|0.016s PS Scavenge|1|1|1|0.008s|0.008s|0.008s2025-08-0814:00:53.144[job-0]INFO JobContainer - PerfTrace not enable!2025-08-0814:00:53.145[job-0]INFO StandAloneJobContainerCommunicator - Total100records,5192bytes|Speed 519B/s,10records/s|Error0records,0bytes|All Task WaitWriterTime0.000s|All Task WaitReaderTime0.000s|Percentage100.00%2025-08-0814:00:53.146[job-0]INFO JobContainer - 任务启动时刻:2025-08-0814:00:36 任务结束时刻:2025-08-0814:00:53 任务总计耗时:16s 任务平均流量:519B/s 记录写入速度:10rec/s 读出记录总数:100读写失败总数:0搭建 datax-web下载源码git clone https://github.com/WeiYe-Jing/datax-web.git创建数据库执行bin/db下面的datax_web.sql文件(注意老版本更新语句有指定库名)修改项目配置1.修改datax_admin下resources/application.yml文件#数据源 datasource: username: root password: root url: jdbc:mysql://localhost:3306/datax_web?serverTimezoneAsia/ShanghaiuseLegacyDatetimeCodefalseuseSSLfalsenullNamePatternMatchesAlltrueuseUnicodetruecharacterEncodingUTF-8 driver-class-name: com.mysql.jdbc.Driver修改数据源配置目前仅支持mysql# 配置mybatis-plus打印sql日志 logging: level: com.wugui.datax.admin.mapper: error path: ./data/applogs/admin修改日志路径path# datax-web email mail: host: smtp.qq.com port: 25 username: xxxqq.com password: xxx properties: mail: smtp: auth: true starttls: enable: true required: true socketFactory: class: javax.net.ssl.SSLSocketFactory修改邮件发送配置(不需要可以不修改)2.修改datax_executor下resources/application.yml文件# log config logging: config: classpath:logback.xml path: ./data/applogs/executor/jobhandler修改日志路径pathdatax: job: admin: ### datax-web admin address addresses: http://127.0.0.1:8080 executor: appname: datax-executor ip: port: 9999 ### job log path logpath: ./data/applogs/executor/jobhandler ### job log retention days logretentiondays: 30 executor: jsonpath: D:\\Information_Technology\\worksapce_tool\\tmp\\executor\\json\\ pypath: D:\\Information_Technology\\worksapce_tool\\datax-new\\datax\\bin\\datax.py修改datax.job配置admin.addresses datax_admin部署地址如调度中心集群部署存在多个地址则用逗号分隔执行器将会使用该地址进行执行器心跳注册和任务结果回调executor.appname 执行器AppName每个执行器机器集群的唯一标示,执行器心跳注册分组依据executor.ip 默认为空表示自动获取IP多网卡时可手动设置指定IP该IP不会绑定Host仅作为通讯实用地址信息用于 “执行器注册” 和 “调度中心请求并触发任务”executor.port 执行器Server端口号默认端口为9999单机部署多个执行器时注意要配置不同执行器端口executor.logpath 执行器运行日志文件存储磁盘路径,需要对该路径拥有读写权限;executor.logretentiondays 执行器日志文件保存天数,过期日志自动清理, 限制值大于等于3时生效; 否则, 如-1, 关闭自动清理功能executor.jsonpath datax json临时文件保存路径pypath DataX启动脚本地址例如xxx/datax/bin/datax.py这个路径是上面搭建 dataX 已经创建好的启动脚本如果系统配置DataX环境变量DATAX_HOMElogpath、jsonpath、pypath可不配log文件和临时json存放在环境变量路径下。启动项目本地idea开发环境1.运行datax_admin下 DataXAdminApplication2.运行datax_executor下 DataXExecutorApplicationadmin启动成功后日志会输出三个地址两个接口文档地址一个前端页面地址启动成功启动成功后打开页面默认管理员用户名admin 密码123456http://localhost:8080/index.html#/dashboard实战项目管理-添加项目配置数据源创建执行器构建任务生成同步 json点构建会自动生成同步 datax 脚本复制 json回到菜单创建任务创建任务按照如下填写并把 json 贴过来并把 byte 的值改为 0执行成功