beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-360) Add a framework for creating Python-SDK sources for new file types
Date Thu, 07 Jul 2016 04:49:10 GMT

    [ https://issues.apache.org/jira/browse/BEAM-360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365604#comment-15365604
] 

ASF GitHub Bot commented on BEAM-360:
-------------------------------------

GitHub user chamikaramj opened a pull request:

    https://github.com/apache/incubator-beam/pull/599

    [BEAM-360] Some updates related to dynamic work rebalancing of custom sources.

    Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing results
of custom sources.
    
    Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing
custom sources.
    
    Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long'
instead of a 'float'.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/chamikaramj/incubator-beam custom_sources_dwr

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/599.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #599
    
----
commit e51d4acf12133a79671c567c9ff709c941c54f8c
Author: Chamikara Jayalath <chamikara@apache.org>
Date:   2016-06-21T01:09:50Z

    Implements a framework for developing sources for new file types.
    
    Module 'filebasedsource' provides a framework for  creating sources for new file types.
This framework readily implements several features common to many sources based on files.
    
    Additionally, module 'avroio' contains a new source, 'AvroSource', that is implemented
using the framework described above. 'AvroSource' is a source for reading Avro files.
    
    Adds many unit tests for 'filebasedsource' and 'avroio' modules.

commit cacb613448b47592f8415570f7b64bc6de797f91
Author: Chamikara Jayalath <chamikara@apache.org>
Date:   2016-07-07T03:25:04Z

    Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result
of custom sources.
    
    Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing
custom sources.
    
    Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long'
instead of a 'float'.

commit 264b4afc17c255e568a490e02ce47e9fb4b1e17a
Author: Chamikara Jayalath <chamikara@apache.org>
Date:   2016-07-07T03:34:21Z

    Adds more comments.

commit 49e097f9c5c3d8c2bca48d3416b4934a4d86ed34
Author: Chamikara Jayalath <chamikara@google.com>
Date:   2016-07-07T04:41:06Z

    Some updates related to dynamic work rebalancing custom sources.
    
    Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result
of custom sources.
    
    Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing
custom sources.
    
    Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long'
instead of a 'float'.

commit c9696c9e17c9c7a6fc13d53d4da21ac9b325c73c
Author: Chamikara Jayalath <chamikara@google.com>
Date:   2016-07-07T04:41:20Z

    Some updates related to dynamic work rebalancing custom sources.
    
    Adds a class 'iobase.BoundedSourceSplit' to represent dynamic work rebalancing result
of custom sources.
    
    Updates Dataflow runner specific code (apiclient.py) to support dynamic work rebalancing
custom sources.
    
    Updates 'OffsetRangeTracker' so that the result of 'position_at_fraction()'' is a 'long'
instead of a 'float'.

----


> Add a framework for creating Python-SDK sources for new file types
> ------------------------------------------------------------------
>
>                 Key: BEAM-360
>                 URL: https://issues.apache.org/jira/browse/BEAM-360
>             Project: Beam
>          Issue Type: New Feature
>          Components: sdk-py
>            Reporter: Chamikara Jayalath
>            Assignee: Chamikara Jayalath
>
> We already have a framework for creating new sources for Beam Python SDK - https://github.com/apache/incubator-beam/blob/python-sdk/sdks/python/apache_beam/io/iobase.py#L326
> It would be great if we can add a framework on top of this that encapsulates logic common
to sources that are based on files. This framework can include following features that are
common to sources based on files.
> (1) glob expansion
> (2) support for new file-systems
> (3) dynamic work rebalancing based on byte offsets
> (4) support for reading compressed files.
> Java SDK has a similar framework and it's available at - https://github.com/apache/incubator-beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileBasedSource.java



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message