airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miłosz Szymczak (JIRA) <j...@apache.org>
Subject [jira] [Work started] (AIRFLOW-2991) Log path to driver output after Dataproc job is completed
Date Fri, 31 Aug 2018 13:13:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Work on AIRFLOW-2991 started by Miłosz Szymczak.
------------------------------------------------
> Log path to driver output after Dataproc job is completed
> ---------------------------------------------------------
>
>                 Key: AIRFLOW-2991
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-2991
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.10.1
>            Reporter: Miłosz Szymczak
>            Assignee: Miłosz Szymczak
>            Priority: Minor
>              Labels: pull-request-available
>
> It's difficult to keep track of logs from Google Cloud Dataproc jobs invoked by Airflow.
Dataproc API provides an output property [driverOutputResourceUri|https://cloud.google.com/dataproc/docs/reference/rest/v1/projects.regions.jobs#Job] containing
path to stdout of job's driver. It's very convenient to be able to quickly check the output
log in case of errors without opening Google Cloud Console.
> The change involves printing INFO-level log with value of *driverOutputResourceUri*
after Dataproc job is finished.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message