airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nimrod Morag (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-3394) Cannot redirect output from subprocesses
Date Sun, 25 Nov 2018 08:29:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Nimrod Morag updated AIRFLOW-3394:
----------------------------------
    Description: 
When using PythonOperator, I'm using the subprocess module and redirecting the output to
sys.stdout in the parent process

 

since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:

{{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object has no attribute
'fileno' Traceback (most recent call last):}}

{{File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659, in _run_raw_task
result = task_copy.execute(context=context)_}}

{{_File "/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch = super(MyMultiBranchPythonOperator,
self).execute(context)_}}

{{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line
95, in execute return_value = self.execute_callable()_}}

{{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line
100, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs)_}}

{{_File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining run('sudo echo -e
"Host bitbucket.org\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config')_}}

{{_File "/vagrant/pysrc/Airflow/Utils.py", line 23, in run if not subprocess.run(args=cmd,
stdout=sys.stdout, stderr=sys.stderr, shell=True):_}}

{{_File "/usr/lib/python3.5/subprocess.py", line 693, in run with Popen(*popenargs, **kwargs)
as process:_}}

{{_File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite) = self._get_handles(stdin,
stdout, stderr)}}

{{File "/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite = stdout.fileno()}}

{{AttributeError: 'StreamLogWriter' object has no attribute 'fileno'}}

{{[2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}

 

{{Fixed locally by adding to the StreamLogWriter class:}}

{{def fileno(self):}}

{{    return 1}}

 

{{(1 is stdout's file descriptor)}}

 

  was:
When using PythonOperator, I'm using the subprocess module and redirecting the output to
sys.stdout in the parent process

 

since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:

{{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object has no attribute
'fileno' Traceback (most recent call last): File "/usr/local/lib/python3.5/dist-packages/airflow/models.py",
line 1659, in _run_raw_task result = task_copy.execute(context=context) File "/vagrant/pysrc/Airflow/Utils.py",
line 65, in execute branch = super(MyMultiBranchPythonOperator, self).execute(context) File
"/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py", line 95, in
execute return_value = self.execute_callable() File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 100, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs)
File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining run('sudo echo -e "Host
bitbucket.org\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config') File "/vagrant/pysrc/Airflow/Utils.py",
line 23, in run if not subprocess.run(args=cmd, stdout=sys.stdout, stderr=sys.stderr, shell=True):
File "/usr/lib/python3.5/subprocess.py", line 693, in run with Popen(*popenargs, **kwargs)
as process: File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite)
= self._get_handles(stdin, stdout, stderr) File "/usr/lib/python3.5/subprocess.py", line 1404,
in _get_handles c2pwrite = stdout.fileno() AttributeError: 'StreamLogWriter' object has no
attribute 'fileno' [2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}

 

{{Fixed locally by adding to the StreamLogWriter class:}}

{{def fileno(self):}}

{{    return 1}}

 

{{(1 is stdout's file descriptor)}}

 


> Cannot redirect output from subprocesses
> ----------------------------------------
>
>                 Key: AIRFLOW-3394
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3394
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: logging
>    Affects Versions: 1.10.1
>         Environment: ubuntu 16.04
> python 3.5.2
>            Reporter: Nimrod Morag
>            Priority: Major
>              Labels: beginner, easyfix, patch, usability
>   Original Estimate: 0.25h
>  Remaining Estimate: 0.25h
>
> When using PythonOperator, I'm using the subprocess module and redirecting the output
to sys.stdout in the parent process
>  
> since `StreamLogWriter` doesn't implement fileno() this causes the task to fail:
> {{[2018-11-25 08:00:56,064] \{models.py:1760} ERROR - 'StreamLogWriter' object has no
attribute 'fileno' Traceback (most recent call last):}}
> {{File "/usr/local/lib/python3.5/dist-packages/airflow/models.py", line 1659, in _run_raw_task
result = task_copy.execute(context=context)_}}
> {{_File "/vagrant/pysrc/Airflow/Utils.py", line 65, in execute branch = super(MyMultiBranchPythonOperator,
self).execute(context)_}}
> {{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 95, in execute return_value = self.execute_callable()_}}
> {{_File "/usr/local/lib/python3.5/dist-packages/airflow/operators/python_operator.py",
line 100, in execute_callable return self.python_callable(*self.op_args, **self.op_kwargs)_}}
> {{_File "/vagrant/pysrc/Airflow/Tasks.py", line 133, in triggerTraining run('sudo echo
-e "Host bitbucket.org\n\tStrictHostKeyChecking no\n" >> ~/.ssh/config')_}}
> {{_File "/vagrant/pysrc/Airflow/Utils.py", line 23, in run if not subprocess.run(args=cmd,
stdout=sys.stdout, stderr=sys.stderr, shell=True):_}}
> {{_File "/usr/lib/python3.5/subprocess.py", line 693, in run with Popen(*popenargs, **kwargs)
as process:_}}
> {{_File "/usr/lib/python3.5/subprocess.py", line 911, in __init__ errread, errwrite)
= self._get_handles(stdin, stdout, stderr)}}
> {{File "/usr/lib/python3.5/subprocess.py", line 1404, in _get_handles c2pwrite = stdout.fileno()}}
> {{AttributeError: 'StreamLogWriter' object has no attribute 'fileno'}}
> {{[2018-11-25 08:00:56,068] \{models.py:1791} INFO - Marking task as FAILED.}}
>  
> {{Fixed locally by adding to the StreamLogWriter class:}}
> {{def fileno(self):}}
> {{    return 1}}
>  
> {{(1 is stdout's file descriptor)}}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message