beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ahmet Altay (JIRA)" <>
Subject [jira] [Created] (BEAM-391) gcsio got stuck with invalid bucket name
Date Wed, 29 Jun 2016 21:50:30 GMT
Ahmet Altay created BEAM-391:

             Summary: gcsio got stuck with invalid bucket name
                 Key: BEAM-391
             Project: Beam
          Issue Type: Bug
          Components: sdk-py
            Reporter: Ahmet Altay

gcsio got stuck with invalid bucket name

GcsBufferedWriter._start_upload ( raises an exception if the bucket does not exist.
This causes upload thread to silenty fail. It logs exception to the log but this does not
stop the pipeline or closes the receiving end of the multiprocessing.Pipe(). Later a call
in to write() blocks at self.conn.send_bytes(). Note that send may block if the buffer is

Upload thread should have a finally clause to close the socket connection. Or better propagating
the exception to its parent. This is true for other types of exceptions also.

Another small issue in the GcsBufferedWriter.close(). It does not self self.close to True.

reproduction: python -m apache_beam.examples.wordcount --output gs://no-such-thing/

Prints the exception but goes on forever. Ctrl + C breaks the main thread shows where it got

Similarly reproducible on the service.

This message was sent by Atlassian JIRA

View raw message