airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Trevor Edwards (JIRA)" <>
Subject [jira] [Assigned] (AIRFLOW-1504) Log Cluster Name on Dataproc Operator When Executed
Date Tue, 15 Aug 2017 16:33:00 GMT


Trevor Edwards reassigned AIRFLOW-1504:

    Assignee: Trevor Edwards

> Log Cluster Name on Dataproc Operator When Executed
> ---------------------------------------------------
>                 Key: AIRFLOW-1504
>                 URL:
>             Project: Apache Airflow
>          Issue Type: Improvement
>            Reporter: Trevor Edwards
>            Assignee: Trevor Edwards
>            Priority: Minor
>         Attachments:
> Since Jinja template substitution happens only just before execution of a task instance,
it is not possible to inspect the cluster name populated for a Dataproc operator. This makes
it hard to debug issues such as using ts_nodash as part of the clustername, as printing when
the DAG is read would give the output:
> Cluster name: smoke-cluster-{{ ts_nodash }}
> then the operator would execute and the user receives the error:
> <HttpError 400 when requesting ... returned "clusterName must be a match of regex
> but the user has no idea what cluster name was provided to the API.
> See attached example pipeline.

This message was sent by Atlassian JIRA

View raw message