airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility to Airflow
Date Wed, 03 Oct 2018 10:44:19 GMT
simonvanderveldt edited a comment on issue #3504: [AIRFLOW-2310]: Add AWS Glue Job Compatibility
to Airflow
URL: https://github.com/apache/incubator-airflow/pull/3504#issuecomment-426541089
 
 
   @OElesin Have you been able to make any progress on this?
   There's one duplicate file in there `airflow/contrib/aws_glue_job_hook.py` vs `airflow/contrib/hook/aws_glue_job_hook.py`.
   
   [edit] From looking at the PR it seems it will both create a job (using boto3's `create_job()`)
as well as run the job (using boto3's `start_job_run()`). IMHO at the very least these should
be separate things so one can run jobs without creating them.
   To expand on this, we create the jobs using a normal development workflow/CI and we just
want to be able to run them from Airflow. IMHO it's not Airflow's responsibility to create
resources, it should only invoke them.
   
   FYI All the unrelated changes are because you merged `master` into your branch instead
of rebasing your branch on top of `master`

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message