beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmet Altay (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-391) Exceptions in gcsio upload thread causes pipeline to stall
Date Fri, 08 Jul 2016 20:48:11 GMT

    [ https://issues.apache.org/jira/browse/BEAM-391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368449#comment-15368449
] 

Ahmet Altay commented on BEAM-391:
----------------------------------

Another type of Exception that result in the same behavior:

Exception in thread Thread-10:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner self.run()
File "/usr/lib/python2.7/threading.py", line 763, in run self.__target(*self.__args, **self.__kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/utils/retry.py", line 160, in wrapper
return fun(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/io/gcsio.py", line 563, in _start_upload
self.client.objects.Insert(self.insert_request, upload=self.upload)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/clients/storage/storage_v1_client.py",
line 970, in Insertdownload=download)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/base_api.py", line 687, in _RunMethodhttp_request,
client=self.client)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/transfer.py", line 838, in InitializeUploadretries=self.num_retries)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/http_wrapper.py", line 351,
in MakeRequestmax_retry_wait, total_wait_sec))
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/http_wrapper.py", line 341,
in MakeRequestcheck_response_func=check_response_func)
File "/usr/local/lib/python2.7/dist-packages/apitools/base/py/http_wrapper.py", line 391,
in _MakeRequestNoRetry redirections=redirections, connection_type=connection_type)
File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 616, in new_request
self._refresh(request_orig)
File "/usr/local/lib/python2.7/dist-packages/apache_beam/internal/auth.py", line 90, in _refresh
token_data = json.loads(urllib2.urlopen(req).read())
File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen return opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 431, in open response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in _open '_open', req)
File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1227, in http_open return self.do_open(httplib.HTTPConnection,
req)
File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open raise URLError(err)

Error is coming from auth.py _refresh(). That may require retries based on the type of error.

> Exceptions in gcsio upload thread causes pipeline to stall
> ----------------------------------------------------------
>
>                 Key: BEAM-391
>                 URL: https://issues.apache.org/jira/browse/BEAM-391
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Ahmet Altay
>
> gcsio got stuck with invalid bucket name
> GcsBufferedWriter._start_upload (gcsio.py) raises an exception if the bucket does not
exist. This causes upload thread to silenty fail. It logs exception to the log but this does
not stop the pipeline or closes the receiving end of the multiprocessing.Pipe(). Later a call
in to write() blocks at self.conn.send_bytes(). Note that send may block if the buffer is
full.
> Upload thread should have a finally clause to close the socket connection. Or better
propagating the exception to its parent. This is true for other types of exceptions also.
> Another small issue in the GcsBufferedWriter.close(). It does not self self.close to
True.
> reproduction: python -m apache_beam.examples.wordcount --output gs://no-such-thing/
> Prints the exception but goes on forever. Ctrl + C breaks the main thread shows where
it got stuck.
> Similarly reproducible on the service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message