DataX/tdenginewriter/doc/tdenginewriter.md

DataX TDengineWriter

简体中文 | English

1 Quick Introduction

The TDengineWriter plugin enables writing data to the target table of the TDengine database. At the bottom level, TDengineWriter connects TDengine through JDBC, executes insert statement /schemaless statement according to TDengine SQL syntax, and writes data to TDengine.

TDengineWriter can be used as a data migration tool for DBAs to import data from other databases into TDengine.

2 Implementation

TDengineWriter obtains the protocol data generated by Reader through DataX framework, connects to TDengine through JDBC Driver, executes insert statement /schemaless statement, and writes the data to TDengine.

In TDengine, table can be divided into super table, sub-table and ordinary table. Super table and sub-table include Column and Tag. The value of tag column of sub-table is fixed value. (details please refer to: data model)

The TDengineWriter can write data to super tables, sub-tables, and ordinary tables using the following methods based on the type of the table and whether the column parameter contains TBName:

1. Table is a super table and column specifies tbname: use the automatic insert statement to create the table and use tbname as the name of the sub-table.

2. Table is a super table and column does not contain tbname: use schemaless to write the table. TDengine will auto-create a tbname based on the super table name and tag value.

3. Table is a sub-table: Use insert statement to write, ignoreTagUnmatched parameter is true, ignore data in record whose tag value is inconsistent with that of table.

4. Table is a common table: use insert statement to write data.

3 Features Introduction

3.1 Sample

Configure a job to write to TDengine

Create a supertable on TDengine:

create database if not exists test;
create table test.weather (ts timestamp, temperature int, humidity double) tags(is_normal bool, device_id binary(100), address nchar(100));

Write data to TDengine using the following Job configuration:

{
  "job": {
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "type": "string",
                "value": "tb1"
              },
              {
                "type": "date",
                "value": "2022-02-20 12:00:01"
              },
              {
                "type": "long",
                "random": "0, 10"
              },
              {
                "type": "double",
                "random": "0, 10"
              },
              {
                "type": "bool",
                "random": "0, 50"
              },
              {
                "type": "bytes",
                "value": "abcABC123"
              },
              {
                "type": "string",
                "value": "北京朝阳望京"
              }
            ],
            "sliceRecordCount": 1
          }
        },
        "writer": {
          "name": "tdenginewriter",
          "parameter": {
            "username": "root",
            "password": "taosdata",
            "column": [
              "tbname",
              "ts",
              "temperature",
              "humidity",
              "is_normal",
              "device_id",
              "address"
            ],
            "connection": [
              {
                "table": [
                  "weather"
                ],
                "jdbcUrl": "jdbc:TAOS-RS://192.168.56.105:6041/test"
              }
            ],
            "batchSize": 100,
            "ignoreTagsUnmatched": true
          }
        }
      }
    ],
    "setting": {
      "speed": {
        "channel": 1
      }
    }
  }
}

3.2 Configuration

• jdbcUrl
  • Descrption: Data source JDBC connection information, TDengine JDBC information please refer to: Java connector
  • Required: yes
  • Default: none

• username

  • Descrption: username
  • Required: yes
  • Default: none

• password

  • Descrption: password of username
  • Required: yes
  • Default: none

• table

  • Descrption: A list of table names that should contain all of the columns in the column parameter (except tbname). Note that tbname in column is used as the TDengine sub-table name.
  • Required: yes
  • Default: none

• column

  • Descrption: A list of field names, the order of the fields should be the column in the record
  • Required: yes
  • Default: none

• batchSize

  • Descrption: Each batchSize record is written to a batch
  • Required: no
  • Default: 1

• ignoreTagsUnmatched

  • Descrption: When table is a sub-table in TDengine, table has a tag value. If the tag value of the data and the tag value of the table are not equal, the data is not written to the table.
  • Required: no
  • Default: false

3.3 Type Convert

Data types in datax that can be mapped to data types in TDengine

DataX Type	TDengine Type
INT	TINYINT, SMALLINT, INT
LONG	TIMESTAMP, TINYINT, SMALLINT, INT, BIGINT
DOUBLE	FLOAT, DOUBLE
STRING	TIMESTAMP, BINARY, NCHAR
BOOL	BOOL
DATE	TIMESTAMP
BYTES	BINARY

3.2 From MongoDB to TDengine

Here are some examples of data sources migrating to TDengine

Sample	Configuration
TDengine to TDengine	super table to super table with tbname
TDengine to TDengine	super table to super table without tbname
TDengine to TDengine	super table to sub-table
TDengine to TDengine	table to table
RDBMS to TDengine	table to super table with tbname
RDBMS to TDengine	table to super table without tbname
RDBMS to TDengine	table to sub-table
RDBMS to TDengine	table to table
OpenTSDB to TDengine	metric to table

4 Restriction

FAQ

Do columns in source table and columns in target table must be in the same order?

Yes, TDengineWriter parses the data from the Datax in the order of the fields in the column.