airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Fokko Driesprong (JIRA)" <>
Subject [jira] [Created] (AIRFLOW-1224) Consolidate logging of subprocess
Date Thu, 18 May 2017 07:09:04 GMT
Fokko Driesprong created AIRFLOW-1224:

             Summary: Consolidate logging of subprocess
                 Key: AIRFLOW-1224
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Fokko Driesprong


Currently a lot of hooks are using a subprocess to execute a command. The obvious one is the
BashHook, but also the SqoopHook and the SparkSubmitHook.

Right now in every hook there is an implementation of how to consume stdout, stderr of the
subprocess. Currently this code is copied several times across the different hooks, and there
are several versions of it. It would be nice to create a method in the BaseHook that accepts
an iterator and a function as a parameter. The iterator will be consumed and the logging output
will be passed to Airflow. The second argument, the function is optional, and can be used
to process the output of the subprocess in case something needs to be extracted.

This message was sent by Atlassian JIRA

View raw message