Mailing-List: contact commits-help@beam.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.apache.org
Date: Wed, 29 Mar 2017 19:05:41 +0000 (UTC)
From: "Chamikara Jayalath (JIRA)" <jira@apache.org>
To: commits@beam.apache.org
Message-ID: <JIRA.13013576.1476894661000.157663.1490814341700@Atlassian.JIRA>
In-Reply-To: <JIRA.13013576.1476894661000@Atlassian.JIRA>
References: <JIRA.13013576.1476894661000@Atlassian.JIRA> <JIRA.13013576.1476894661792@jira-lw-us.apache.org>
Subject: [jira] [Commented] (BEAM-778) Make fileio._CompressedFile seekable.
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 29 Mar 2017 19:05:46 -0000


    [ https://issues.apache.org/jira/browse/BEAM-778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15947718#comment-15947718 ] 

Chamikara Jayalath commented on BEAM-778:
-----------------------------------------

Currently this is not an issue since Beam FileBasedSoure and FileBasedSink are the only users of CompressedFile/File objects and they are used in a pretty straightforward way where each FileBasedSource/FileBasedSink object owns it's File/CompressedFile object and reading is done using a single thread. A secondary thread that performs dynamic work rebalancing might execute seek() operations for File objects but not for CompressedFile objects.

In the future we might have other places where we access CompressedFile objects using multiple thread but I think we should probably wait till such needs arise. Also it might be enough to declare CompressedFile objects to be not thread safe and expect users to address thread safety instead of embedding a lock in CompressedFile objects which would potentially add a performance penalty for all users.

WDYT ? 

> Make fileio._CompressedFile seekable.
> -------------------------------------
>
>                 Key: BEAM-778
>                 URL: https://issues.apache.org/jira/browse/BEAM-778
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-py
>            Reporter: Chamikara Jayalath
>            Assignee: Tibor Kiss
>             Fix For: Not applicable
>
>
> We have a TODO to make fileio._CompressedFile seekable.
> https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/fileio.py#L692
> Without this, compressed file objects produce for FileBasedSource implementations may not be able to use libraries that utilize methods seek() and tell().
> For example tarfile.open().


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)