airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kris Geusebroek (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-1219) Spark submit hook fix realtime logging.
Date Wed, 17 May 2017 13:44:04 GMT
Kris Geusebroek created AIRFLOW-1219:

             Summary: Spark submit hook fix realtime logging.
                 Key: AIRFLOW-1219
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Kris Geusebroek

Redirecting the logging to airflow doesn't work correctly.

Spark's logging goes to stderr and output of the spark program goes to stdout. In the loop
of the two sources stdout is used first.
It seems to be blocking until output is written at the end of the spark-submit execution.

Fix this by connecting stderr to stdout in the subprocess invocation and only pipe stdout
to the airflow logging

This message was sent by Atlassian JIRA

View raw message