beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <>
Subject [jira] [Commented] (BEAM-391) Exceptions in gcsio upload thread causes pipeline to stall
Date Fri, 15 Jul 2016 01:33:20 GMT


ASF GitHub Bot commented on BEAM-391:

Github user aaltay closed the pull request at:

> Exceptions in gcsio upload thread causes pipeline to stall
> ----------------------------------------------------------
>                 Key: BEAM-391
>                 URL:
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-py
>            Reporter: Ahmet Altay
> gcsio got stuck with invalid bucket name
> GcsBufferedWriter._start_upload ( raises an exception if the bucket does not
exist. This causes upload thread to silenty fail. It logs exception to the log but this does
not stop the pipeline or closes the receiving end of the multiprocessing.Pipe(). Later a call
in to write() blocks at self.conn.send_bytes(). Note that send may block if the buffer is
> Upload thread should have a finally clause to close the socket connection. Or better
propagating the exception to its parent. This is true for other types of exceptions also.
> Another small issue in the GcsBufferedWriter.close(). It does not self self.close to
> reproduction: python -m apache_beam.examples.wordcount --output gs://no-such-thing/
> Prints the exception but goes on forever. Ctrl + C breaks the main thread shows where
it got stuck.
> Similarly reproducible on the service.

This message was sent by Atlassian JIRA

View raw message