DataX 是阿里巴巴集团内被广泛使用的离线数据同步工具/平台,实现包括 MySQL、Oracle、HDFS、Hive、OceanBase、HBase、OTS、ODPS 等各种异构数据源之间高效的数据同步功能。

dependabot[bot] 7ae7f19689 Bump org.json:json from 20160810 to 20231013 in /obhbasereader 3 ay önce
adbmysqlwriter 32a11b9bb1 add adbmysqlwriter plugin 1 yıl önce
adbpgwriter 5d4b55af89 update:补充adbpgwriter组件doc,column参数说明 1 yıl önce
adswriter f79f742d7c datax 202302 1 yıl önce
cassandrareader 6d03421f10 Merge pull request #1601 from wenshao/master 1 yıl önce
cassandrawriter 27964108aa update fastjson version 2 yıl önce
clickhousereader 74f12ff9a7 to #51999348 datax month 9 1 yıl önce
clickhousewriter 27964108aa update fastjson version 2 yıl önce
common b0224dc01d [BugFix] Fix Comfiguration类中checkPath方法未生效的问题 5 ay önce
core 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
databendwriter ae6e6630ef fix 1 yıl önce
datahubreader 27964108aa update fastjson version 2 yıl önce
datahubwriter 27964108aa update fastjson version 2 yıl önce
datax-example 3207b63c52 对example工程结构进行了重新规划,给出两个编写测试模块的案例。 1 yıl önce
dorisreader 41fc378f24 Add Dorisreader plugin 9 ay önce
doriswriter 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
drdsreader 2d2f3e22b1 Merge pull request #1183 from PeineLiang/update-doc 3 yıl önce
drdswriter a35bddcc8e https://github.com/alibaba/DataX/issues/390 3 yıl önce
elasticsearchwriter 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
ftpreader fa123778cb Merge pull request #1788 from chenhaoyus/master 1 yıl önce
ftpwriter 8bddafba08 update com.jcraft:jsch 0.1.51 to 0.1.54 1 yıl önce
gaussdbreader 0e79171132 华为GaussDB读写插件 2 yıl önce
gaussdbwriter 5ba5ed025d Update package.xml 1 yıl önce
gdbreader 05d1851d99 gdbreader: reader for Aliyun GDB 4 yıl önce
gdbwriter 27964108aa update fastjson version 2 yıl önce
hbase094xreader 27964108aa update fastjson version 2 yıl önce
hbase094xwriter 27964108aa update fastjson version 2 yıl önce
hbase11xreader 27964108aa update fastjson version 2 yıl önce
hbase11xsqlreader 8db76ddd5d Merge pull request #1606 from hadoop835/master 1 yıl önce
hbase11xsqlwriter 27964108aa update fastjson version 2 yıl önce
hbase11xwriter 27964108aa update fastjson version 2 yıl önce
hbase20xsqlreader eb77e7ad6c Merge pull request #1408 from cuishuang/master 2 yıl önce
hbase20xsqlwriter 874a256a03 DataX 2209, add plugin: datahubreader/datahubwriter/loghubreader/loghubwriter/starrocksreader, update plugin: odpsreader/oceanbasev10reader/oceanbasev10writer/elasticserachwriter/mysqlreader/mysqlwriter 2 yıl önce
hdfsreader 1c2d5d0bb1 升级hdfs hive,parquet-format依赖 8 ay önce
hdfswriter 1c2d5d0bb1 升级hdfs hive,parquet-format依赖 8 ay önce
hologresjdbcwriter 6e9e11a741 fix hologresjdbcwriter: use safer jdbc url replacement 9 ay önce
images 2151860a07 add new datax logo 2 yıl önce
kingbaseesreader 329e9c74e1 Add kingbasees plug-in. 4 yıl önce
kingbaseeswriter 329e9c74e1 Add kingbasees plug-in. 4 yıl önce
kuduwriter f79f742d7c datax 202302 1 yıl önce
loghubreader 27964108aa update fastjson version 2 yıl önce
loghubwriter 874a256a03 DataX 2209, add plugin: datahubreader/datahubwriter/loghubreader/loghubwriter/starrocksreader, update plugin: odpsreader/oceanbasev10reader/oceanbasev10writer/elasticserachwriter/mysqlreader/mysqlwriter 2 yıl önce
mongodbreader f79f742d7c datax 202302 1 yıl önce
mongodbwriter 27964108aa update fastjson version 2 yıl önce
mysqlreader 874a256a03 DataX 2209, add plugin: datahubreader/datahubwriter/loghubreader/loghubwriter/starrocksreader, update plugin: odpsreader/oceanbasev10reader/oceanbasev10writer/elasticserachwriter/mysqlreader/mysqlwriter 2 yıl önce
mysqlwriter 5c040c2989 Update mysqlwriter.md 1 yıl önce
neo4jwriter 74f12ff9a7 to #51999348 datax month 9 1 yıl önce
obhbasereader 7ae7f19689 Bump org.json:json from 20160810 to 20231013 in /obhbasereader 3 ay önce
obhbasewriter 3614c2633e fix obhbase reader and writer package error 3 ay önce
oceanbasev10reader 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
oceanbasev10writer 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
ocswriter 0af1933802 Bump jdk to 1.8 5 yıl önce
odpsreader f79f742d7c datax 202302 1 yıl önce
odpswriter f79f742d7c datax 202302 1 yıl önce
opentsdbreader 27964108aa update fastjson version 2 yıl önce
oraclereader c1f0d380b9 Update oracle driver GAV 2 yıl önce
oraclewriter c1f0d380b9 Update oracle driver GAV 2 yıl önce
oscarwriter 874a256a03 DataX 2209, add plugin: datahubreader/datahubwriter/loghubreader/loghubwriter/starrocksreader, update plugin: odpsreader/oceanbasev10reader/oceanbasev10writer/elasticserachwriter/mysqlreader/mysqlwriter 2 yıl önce
ossreader fd80754feb to #51999348 datax month 9 1 yıl önce
osswriter d187c4657e update oss parquet writer 1 yıl önce
otsreader b875e38446 update ots plugins 1 yıl önce
otsstreamreader b875e38446 update ots plugins 1 yıl önce
otswriter b875e38446 update ots plugins 1 yıl önce
plugin-rdbms-util 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
plugin-unstructured-storage-util 7097651cd8 fix an error 1 yıl önce
postgresqlreader ed117381d2 Update postgresqlreader.md 2 yıl önce
postgresqlwriter e9a579759e DataX 通道能力更新(MaxCompute、Hologres、Tdengine、OSS)、安全漏洞更新、通用打包更新等 2 yıl önce
rdbmsreader ebdfe63fd7 Update rdbmsreader.md 8 ay önce
rdbmswriter 5e747cad63 Update rdbmswriter.md 8 ay önce
rpm 7ec3fb68e5 v0.0 6 yıl önce
selectdbwriter 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
sqlserverreader 874a256a03 DataX 2209, add plugin: datahubreader/datahubwriter/loghubreader/loghubwriter/starrocksreader, update plugin: odpsreader/oceanbasev10reader/oceanbasev10writer/elasticserachwriter/mysqlreader/mysqlwriter 2 yıl önce
sqlserverwriter a944984d74 sqlserver readme add session config 1 yıl önce
starrocksreader 874a256a03 DataX 2209, add plugin: datahubreader/datahubwriter/loghubreader/loghubwriter/starrocksreader, update plugin: odpsreader/oceanbasev10reader/oceanbasev10writer/elasticserachwriter/mysqlreader/mysqlwriter 2 yıl önce
starrockswriter 47f6dff024 修复日志关键词错误 1 yıl önce
streamreader af631a0a00 不依赖于user.dir环境变量,所以插件加载工厂便没有意义了 1 yıl önce
streamwriter af631a0a00 不依赖于user.dir环境变量,所以插件加载工厂便没有意义了 1 yıl önce
sybasereader 612bf37b7c 添加sybase jconn依赖,并添加到打包依赖中。 8 ay önce
sybasewriter ec13e37fa0 fix: wrong class in sybasewriter 7 ay önce
tdenginereader bef624414a merge taosdata/master 2 yıl önce
tdenginewriter 622524593a Fix typo in tdenginewriter.md 1 yıl önce
transformer 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
tsdbreader 27964108aa update fastjson version 2 yıl önce
tsdbwriter 27964108aa update fastjson version 2 yıl önce
txtfilereader 5ee2ea58bc bug:修复TxtFileReader读CSV文件时,不读CSVConfig问题 1 yıl önce
txtfilewriter 66de85112d Update txtfilewriter.md 4 yıl önce
.gitignore 4d70b5ab86 Add OpenTSDB reader and TSDB writer 5 yıl önce
NOTICE e9a579759e DataX 通道能力更新(MaxCompute、Hologres、Tdengine、OSS)、安全漏洞更新、通用打包更新等 2 yıl önce
README.md 9d5a82bd7b Update README.md 1 yıl önce
datax-opensource-dingding.png 7ec3fb68e5 v0.0 6 yıl önce
dataxPluginDev.md 34bf78f75f Update dataxPluginDev.md 1 yıl önce
introduction.md 09374d2596 Add OceanBase supporting list. 3 yıl önce
license.txt e9a579759e DataX 通道能力更新(MaxCompute、Hologres、Tdengine、OSS)、安全漏洞更新、通用打包更新等 2 yıl önce
package.xml 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
pom.xml 0ec767730e add obhbase reader and writer plugin by cjyyz64 3 ay önce
userGuid.md 74f12ff9a7 to #51999348 datax month 9 1 yıl önce

README.md

Datax-logo

DataX

Leaderboard

DataX 是阿里云 DataWorks数据集成 的开源版本,在阿里巴巴集团内被广泛使用的离线数据同步工具/平台。DataX 实现了包括 MySQL、Oracle、OceanBase、SqlServer、Postgre、HDFS、Hive、ADS、HBase、TableStore(OTS)、MaxCompute(ODPS)、Hologres、DRDS, databend 等各种异构数据源之间高效的数据同步功能。

DataX 商业版本

阿里云DataWorks数据集成是DataX团队在阿里云上的商业化产品,致力于提供复杂网络环境下、丰富的异构数据源之间高速稳定的数据移动能力,以及繁杂业务背景下的数据同步解决方案。目前已经支持云上近3000家客户,单日同步数据超过3万亿条。DataWorks数据集成目前支持离线50+种数据源,可以进行整库迁移、批量上云、增量同步、分库分表等各类同步解决方案。2020年更新实时同步能力,支持10+种数据源的读写任意组合。提供MySQL,Oracle等多种数据源到阿里云MaxCompute,Hologres等大数据引擎的一键全增量同步解决方案。

商业版本参见: https://www.aliyun.com/product/bigdata/ide

Features

DataX本身作为数据同步框架,将不同数据源的同步抽象为从源头数据源读取数据的Reader插件,以及向目标端写入数据的Writer插件,理论上DataX框架可以支持任意数据源类型的数据同步工作。同时DataX插件体系作为一套生态系统, 每接入一套新数据源该新加入的数据源即可实现和现有的数据源互通。

DataX详细介绍

请参考:DataX-Introduction

Quick Start

Download DataX下载地址
请点击:Quick Start

Support Data Channels

DataX目前已经有了比较全面的插件体系,主流的RDBMS数据库、NOSQL、大数据计算系统都已经接入,目前支持数据如下图,详情请点击:DataX数据源参考指南

类型 数据源 Reader(读) Writer(写) 文档
RDBMS 关系型数据库 MySQL
Oracle
OceanBase
SQLServer
PostgreSQL
DRDS
Kingbase
通用RDBMS(支持所有关系型数据库)
阿里云数仓数据存储 ODPS
ADB
ADS
OSS
OCS
Hologres
AnalyticDB For PostgreSQL
阿里云中间件 datahub 读 、写
SLS 读 、写
图数据库 阿里云 GDB
Neo4j
NoSQL数据存储 OTS
Hbase0.94
Hbase1.1
Phoenix4.x
Phoenix5.x
MongoDB
Cassandra
数仓数据存储 StarRocks 读 、
ApacheDoris
ClickHouse
Databend
Hive
kudu
selectdb
无结构化数据存储 TxtFile
FTP
HDFS
Elasticsearch
时间序列数据库 OpenTSDB
TSDB
TDengine

阿里云DataWorks数据集成

目前DataX的已有能力已经全部融和进阿里云的数据集成,并且比DataX更加高效、安全,同时数据集成具备DataX不具备的其它高级特性和功能。可以理解为数据集成是DataX的全面升级的商业化用版本,为企业可以提供稳定、可靠、安全的数据传输服务。与DataX相比,数据集成主要有以下几大突出特点:

支持实时同步:

离线同步数据源种类大幅度扩充:

我要开发新的插件

请点击:DataX插件开发宝典

重要版本更新说明

DataX 后续计划月度迭代更新,也欢迎感兴趣的同学提交 Pull requests,月度更新内容如下。

项目成员

核心Contributions: 言柏 、枕水、秋奇、青砾、一斅、云时

感谢天烬、光戈、祁然、巴真、静行对DataX做出的贡献。

License

This software is free to use under the Apache License Apache license.

请及时提出issue给我们。请前往:DataxIssue

开源版DataX企业用户

Datax-logo

``` 长期招聘 联系邮箱:datax@alibabacloud.com 【JAVA开发职位】 职位名称:JAVA资深开发工程师/专家/高级专家 工作年限 : 2年以上 学历要求 : 本科(如果能力靠谱,这些都不是条件) 期望层级 : P6/P7/P8

岗位描述:

1. 负责阿里云大数据平台(数加)的开发设计。 
2. 负责面向政企客户的大数据相关产品开发;
3. 利用大规模机器学习算法挖掘数据之间的联系,探索数据挖掘技术在实际场景中的产品应用 ;
4. 一站式大数据开发平台
5. 大数据任务调度引擎
6. 任务执行引擎
7. 任务监控告警
8. 海量异构数据同步

岗位要求:

1. 拥有3年以上JAVA Web开发经验;
2. 熟悉Java的基础技术体系。包括JVM、类装载、线程、并发、IO资源管理、网络;
3. 熟练使用常用Java技术框架、对新技术框架有敏锐感知能力;深刻理解面向对象、设计原则、封装抽象;
4. 熟悉HTML/HTML5和JavaScript;熟悉SQL语言;
5. 执行力强,具有优秀的团队合作精神、敬业精神;
6. 深刻理解设计模式及应用场景者加分;
7. 具有较强的问题分析和处理能力、比较强的动手能力,对技术有强烈追求者优先考虑;
8. 对高并发、高稳定可用性、高性能、大数据处理有过实际项目及产品经验者优先考虑;
9. 有大数据产品、云产品、中间件技术解决方案者优先考虑。

````

用户咨询支持:

钉钉群目前暂时受到了一些管控策略影响,建议大家有问题优先在这里提交问题 Issue,DataX研发和社区会定期回答Issue中的问题,知识库丰富后也能帮助到后来的使用者。