airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kaxil Naik (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (AIRFLOW-3002) ValueError in dataflow operators when using GCS jar or py_file
Date Wed, 05 Sep 2018 08:47:00 GMT

     [ https://issues.apache.org/jira/browse/AIRFLOW-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kaxil Naik resolved AIRFLOW-3002.
---------------------------------
    Resolution: Fixed

> ValueError in dataflow operators when using GCS jar or py_file
> --------------------------------------------------------------
>
>                 Key: AIRFLOW-3002
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3002
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib, Dataflow
>    Affects Versions: 1.10.0, 2.0.0
>            Reporter: Jeffrey Payne
>            Assignee: Kaxil Naik
>            Priority: Major
>             Fix For: 1.10.1
>
>
> The {{GoogleCloudBucketHelper.google_cloud_to_local}} function now fails with a ValueError,
with:
> {noformat}
> ...
> file_size = self._gcs_hook.download(bucket_id, object_id, local_file)
> if os.stat(file_size).st_size > 0:
>     return local_file
> ...
> {noformat}
> The {{os.stat()}} function takes a _path_, but the {{file_size}} var passed in is actually
the downloaded bytes from {{GoogleCloudStorageHook.download()}}.
> The error is like:
> {noformat}
> [2018-09-04 14:46:49,840] {base_task_runner.py:107} INFO - Job 59: Subtask surge_export
  File "/opt/conda/envs/bairflow-gke/lib/python3.5/site-packages/airflow/contrib/operators/dataflow_operator.py",
line 372, in google_cloud_to_local
> [2018-09-04 14:46:49,841] {base_task_runner.py:107} INFO - Job 59: Subtask surge_export
    if os.stat(file_size).st_size > 0:
> [2018-09-04 14:46:49,841] {base_task_runner.py:107} INFO - Job 59: Subtask surge_export
ValueError: stat: embedded null character in path
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message