airflow-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AIRFLOW-1023) Upload file to S3 using S3 hook fails with "Connection reset by peer"
Date Sun, 02 Sep 2018 18:07:04 GMT

    [ https://issues.apache.org/jira/browse/AIRFLOW-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16601582#comment-16601582
] 

Apache Spark commented on AIRFLOW-1023:
---------------------------------------

User 'adrpar' has created a pull request for this issue:
https://github.com/apache/incubator-airflow/pull/2176

> Upload file to S3 using S3 hook fails with "Connection reset by peer"
> ---------------------------------------------------------------------
>
>                 Key: AIRFLOW-1023
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1023
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: hooks
>    Affects Versions: Airflow 1.7.1
>            Reporter: Adrian Partl
>            Assignee: Adrian Partl
>            Priority: Major
>
> Using the S3 hook to upload local files to an S3 bucket fails with 
> {noformat}
>   File "/usr/lib/python2.7/site-packages/airflow/hooks/S3_hook.py", line 364, in load_file
>     replace=replace)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1362, in set_contents_from_filename
>     encrypt_key=encrypt_key)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file
>     chunked_transfer=chunked_transfer, size=size)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file
>     chunked_transfer=chunked_transfer, size=size)
>   File "/usr/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal
>     query_args=query_args
>   File "/usr/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request
>     retry_handler=retry_handler
>   File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request
>     retry_handler=retry_handler)
>   File "/usr/lib/python2.7/site-packages/boto/connection.py", line 1030, in _mexe
>     raise ex
> error: [Errno 104] Connection reset by peer
> {noformat}
> This is a known issue with boto and only affects uploads to S3 buckets outside of the
standard US location (in my case {{eu-west-1}}).
> The issue is reported on boto side as:
> https://github.com/boto/boto/issues/2207
> A work around is mentioned by user {{anna-buttfield-sirca}} which basically reconnects
the boto S3 connection to the corresponding location.
> I will provide a PR implementing the work around, since a resolution of the issue on
the boto side seems unlikely.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message