airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dong.yajun" <dongt...@gmail.com>
Subject HiveToMySqlTransfer with auto increment id in MySQL.
Date Tue, 31 Oct 2017 11:23:34 GMT
hi list,

we try to use HiveToMySqlTransfer to write the hive data into MySQL. but
there was an auto increment id in our table.

like this:

> create table users_simple (`id` int NOT NULL AUTO_INCREMENT,
`user_id` bigint NOT NULL, PRIMARY KEY(`id`));

in airflow DAG we use HiveToMySqlTransfer:

t = HiveToMySqlTransfer(

    mysql_conn_id='data_pipeline_mysql',

    task_id='hive_to_mysql_check',

    hiveserver2_conn_id='hiveserver2_default',

    sql="""

    SELECT login

    FROM staging.users_simple limit 100

    """,

    mysql_table='users_simple',

    dag=dag)

when run the code above, we met the following exception:

> _mysql_exceptions.OperationalError: (1136, "Column count doesn't match
value count at row 1")

I believe the airflow translate the task as the SQL statement:

> insert into users_simple(id, user_id) values (login)

so the solution is how we pass the exactly columns to dbapi_hook.py
https://github.com/apache/incubator-airflow/blob/eb2f589099b87743482c2eb16261b49e284dcd96/airflow/hooks/dbapi_hook.py#L187
  via HiveToMySqlTransfer ?


-- 
*Ric Dong*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message