airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maximilian Roos (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AIRFLOW-1758) Print full traceback on errors
Date Thu, 26 Oct 2017 21:55:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-1758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Maximilian Roos updated AIRFLOW-1758:
-------------------------------------
    Description: 
Currently when there is a failure during a run, it's difficult to see what the cause was.
Could we at least print the python stack trace? 

As an example: 
{code:python}
[2017-10-26 21:43:38,155] {models.py:1563} ERROR - DataFlow failed with return code 1
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1461, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/dataflow_operator.py",
line 192, in execute
    self.py_file, self.py_options)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 155, in start_python_dataflow
    task_id, variables, dataflow, name, ["python"] + py_options)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 141, in _start_dataflow
    _Dataflow(cmd).wait_for_done()
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 122, in wait_for_done
    self._proc.returncode))
Exception: DataFlow failed with return code 1
{code}


I then need to jump into a repl and attempt simulate the command that airflow would have run,
which is both difficult and error prone. (Or is there a simpler way of doing this??)

I then get a better stack-trace:

{code:python}
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line 328, in run
    return self.runner.run(self)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 283, in run
    self.dataflow_client.create_job(self.job), self)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 168, in wrapper
    return fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 423, in create_job
    self.create_job_description(job)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 446, in create_job_description
    job.options, file_copy=self._gcs_file_copy)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 347, in stage_job_resources
    build_setup_args)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 439, in _build_setup_package
    os.chdir(os.path.dirname(setup_file))
OSError: [Errno 2] No such file or directory: ''
{code}

Somewhat related to: https://issues.apache.org/jira/browse/AIRFLOW-174

(I'm using the DataFlowPythonOperator at the moment, but I suspect the issue is wider)

  was:
Currently when there is a failure during a run, it's difficult to see what the cause was.
Could we at least print the python stack trace? 

As an example: 
```
[2017-10-26 21:43:38,155] {models.py:1563} ERROR - DataFlow failed with return code 1
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1461, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/dataflow_operator.py",
line 192, in execute
    self.py_file, self.py_options)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 155, in start_python_dataflow
    task_id, variables, dataflow, name, ["python"] + py_options)
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 141, in _start_dataflow
    _Dataflow(cmd).wait_for_done()
  File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 122, in wait_for_done
    self._proc.returncode))
Exception: DataFlow failed with return code 1
```

I then need to jump into a repl and attempt simulate the command that airflow would have run,
which is both difficult and error prone. (Or is there a simpler way of doing this??)

I then get a better stack-trace:

```
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line 328, in run
    return self.runner.run(self)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 283, in run
    self.dataflow_client.create_job(self.job), self)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 168, in wrapper
    return fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 423, in create_job
    self.create_job_description(job)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 446, in create_job_description
    job.options, file_copy=self._gcs_file_copy)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 347, in stage_job_resources
    build_setup_args)
  File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 439, in _build_setup_package
    os.chdir(os.path.dirname(setup_file))
OSError: [Errno 2] No such file or directory: ''
```

Somewhat related to: https://issues.apache.org/jira/browse/AIRFLOW-174

(I'm using the DataFlowPythonOperator at the moment, but I suspect the issue is wider)


> Print full traceback on errors
> ------------------------------
>
>                 Key: AIRFLOW-1758
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1758
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: cli
>            Reporter: Maximilian Roos
>            Priority: Minor
>
> Currently when there is a failure during a run, it's difficult to see what the cause
was. Could we at least print the python stack trace? 
> As an example: 
> {code:python}
> [2017-10-26 21:43:38,155] {models.py:1563} ERROR - DataFlow failed with return code 1
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/airflow/models.py", line 1461, in _run_raw_task
>     result = task_copy.execute(context=context)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/operators/dataflow_operator.py",
line 192, in execute
>     self.py_file, self.py_options)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 155, in start_python_dataflow
>     task_id, variables, dataflow, name, ["python"] + py_options)
>   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 141, in _start_dataflow
>     _Dataflow(cmd).wait_for_done()
>   File "/usr/local/lib/python2.7/dist-packages/airflow/contrib/hooks/gcp_dataflow_hook.py",
line 122, in wait_for_done
>     self._proc.returncode))
> Exception: DataFlow failed with return code 1
> {code}
> I then need to jump into a repl and attempt simulate the command that airflow would have
run, which is both difficult and error prone. (Or is there a simpler way of doing this??)
> I then get a better stack-trace:
> {code:python}
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/pipeline.py", line 328, in
run
>     return self.runner.run(self)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/dataflow_runner.py",
line 283, in run
>     self.dataflow_client.create_job(self.job), self)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 168,
in wrapper
>     return fun(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 423, in create_job
>     self.create_job_description(job)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/apiclient.py",
line 446, in create_job_description
>     job.options, file_copy=self._gcs_file_copy)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 347, in stage_job_resources
>     build_setup_args)
>   File "/usr/local/lib/python2.7/dist-packages/apache_beam/runners/dataflow/internal/dependency.py",
line 439, in _build_setup_package
>     os.chdir(os.path.dirname(setup_file))
> OSError: [Errno 2] No such file or directory: ''
> {code}
> Somewhat related to: https://issues.apache.org/jira/browse/AIRFLOW-174
> (I'm using the DataFlowPythonOperator at the moment, but I suspect the issue is wider)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message