airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikram Fugro (JIRA)" <>
Subject [jira] [Commented] (AIRFLOW-1936) EmrCreateJobFlowOperator is unable to launch the aws emr cluster
Date Mon, 18 Dec 2017 12:01:00 GMT


Vikram Fugro commented on AIRFLOW-1936:

My bad! These guys got started in another region (N.Virginia)!   I overlooked ""
in the logs.  (sorry! been running fever).  My ~/.aws/config has eu-west-1.

Closing the bug! 

BTW, what does emr_conn_id mean w.r.t  the connection. (admin->connection)? I am not sure
what values to put in there for a 
 custom emr_conn?  I mean what am I supposed to fill in for conn_type, Host, schema, login,
etc for emr_conn?

> EmrCreateJobFlowOperator is unable to launch the aws emr cluster
> ----------------------------------------------------------------
>                 Key: AIRFLOW-1936
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: aws, boto3, DAG, hooks, operators
>    Affects Versions: Airflow 1.8
>         Environment: ubuntu 16.04 , python 2.7, boto3
>            Reporter: Vikram Fugro
>              Labels: newbie
>         Attachments: dag_logs.txt,, my_emr_boto_script(works).py,
> The EmrCreateJobFlowOperator  operator is unable to create the emr cluster, although
it returns  success. The most strange thing that I see in the logs is that it also returns
the jobflow-Id but I see no cluster starting up in the aws emr console. If I try to launch
the emr with my boto3 script , it works. I even  hardcoded the arguments (taken from my boto3
script as is)  in self.get_conn().run_job_flow(), in contrib/hooks/ , but no luck.

> I am running airflow 1.8, localexecutor with metadb postgres. I have tried both ways
; setting up a schedule and also by  command 'airflow run  -f emr_job_flow_manual_steps_dag2
create_job_flow 2017-12-18' 
> I am attaching the logs of my dag,  my dag itself , my boto3 script and
only for testing purpose).

This message was sent by Atlassian JIRA

View raw message