airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trevor Edwards (JIRA)" <j...@apache.org>
Subject [jira] [Created] (AIRFLOW-1504) Log Cluster Name on Dataproc Operator When Executed
Date Fri, 11 Aug 2017 15:50:00 GMT
Trevor Edwards created AIRFLOW-1504:
---------------------------------------

             Summary: Log Cluster Name on Dataproc Operator When Executed
                 Key: AIRFLOW-1504
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1504
             Project: Apache Airflow
          Issue Type: Improvement
            Reporter: Trevor Edwards
            Priority: Minor
         Attachments: dataproc_cluster_logging.py

Since Jinja template substitution happens only just before execution of a task instance, it
is not possible to inspect the cluster name populated for a Dataproc operator. This makes
it hard to debug issues such as using ts_nodash as part of the clustername, as printing when
the DAG is read would give the output:

Cluster name: smoke-cluster-{{ ts_nodash }}

then the operator would execute and the user receives the error:

<HttpError 400 when requesting ... returned "clusterName must be a match of regex '(?:[a-z](?:[-a-z0-9]{0,49}[a-z0-9])?).'">

but the user has no idea what cluster name was provided to the API.

See attached example pipeline.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message