airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dong.yajun" <>
Subject HiveToMySqlTransfer with auto increment id in MySQL.
Date Tue, 31 Oct 2017 11:23:34 GMT
hi list,

we try to use HiveToMySqlTransfer to write the hive data into MySQL. but
there was an auto increment id in our table.

like this:

> create table users_simple (`id` int NOT NULL AUTO_INCREMENT,
`user_id` bigint NOT NULL, PRIMARY KEY(`id`));

in airflow DAG we use HiveToMySqlTransfer:

t = HiveToMySqlTransfer(





    SELECT login

    FROM staging.users_simple limit 100




when run the code above, we met the following exception:

> _mysql_exceptions.OperationalError: (1136, "Column count doesn't match
value count at row 1")

I believe the airflow translate the task as the SQL statement:

> insert into users_simple(id, user_id) values (login)

so the solution is how we pass the exactly columns to
  via HiveToMySqlTransfer ?

*Ric Dong*

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message