airflow-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bernardo Najlis <b52....@gmail.com>
Subject AIRFLOW-2224: Add support for CSV files in mysql_to_gcs operator
Date Sat, 24 Mar 2018 17:46:22 GMT
Hey,

I submitted a PR associated with AIRFLOW-2224
<https://github.com/apache/incubator-airflow/pull/3139> last week that
updates the mysql_to_gcs contrib operator, adding support for CSV files
(currently only supports JSON).

One of the Travis CI tests failed and I'm having some challenges
understanding what the problem is: went through the test output log a
couple of times and it seems the issue is with a test completely unrelated
to my changes.

Is this something that anybody has seen before? Currently working to create
a test for my added functionality (which was missing from the original PR)
but I'm getting concerned that this unrelated test that failed before may
fail once again...

Here's a link to the failed test job
<https://travis-ci.org/apache/incubator-airflow/jobs/355158204> and a
summary of (what I think) the error is:


[2018-03-18 23:56:15,972] {models.py:4526} ERROR -
(_mysql_exceptions.IntegrityError) (1062, "Duplicate entry
'example_bash_operator-2018-03-18 23:56:15.000000' for key 'dag_id'") [SQL:
'INSERT INTO dag_run (dag_id, execution_date, start_date, end_date, state,
run_id, external_trigger, conf) VALUES (%s, %s, %s, %s, %s, %s, %s, %s)']
[parameters: ('example_bash_operator', datetime.datetime(2018, 3, 18, 23,
56, 15, tzinfo=datetime.timezone.utc), datetime.datetime(2018, 3, 18, 23,
56, 15, 971501, tzinfo=datetime.timezone.utc), None, 'running',
'my_run2018-03-18T23:56:15.605960', 1, None)]

Any kind of help is greatly appreciated, really interested in getting this
PR into master, and also adding the same functionality (CSV file support)
to multiple other operators.

Thanks!

Bernardo Najlis

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message