beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-1040) Hadoop InputFormat - IO Transform for reads
Date Mon, 27 Feb 2017 18:46:45 GMT

    [ https://issues.apache.org/jira/browse/BEAM-1040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15886299#comment-15886299
] 

ASF GitHub Bot commented on BEAM-1040:
--------------------------------------

GitHub user diptikul opened a pull request:

    https://github.com/apache/beam/pull/2115

    BEAM-1040 Add HadoopInputFormatIO 

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
    
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/diptikul/incubator-beam hif-io-clean

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/2115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #2115
    
----
commit 45cf382e14fceb32b2738b548c851472659f84d9
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-22T13:49:38Z

    First Hadoop InputFormat IO project code check in

commit 9c1a54d8d686ac396a1f500c9b69abe4d44e18a5
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-23T08:15:07Z

    Merge rtest from original
    
    Take latest from original. remote-tracking branch 'upstream/master'

commit 5f927add640a9888ca26662450ceb1ecfa92c08c
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2016-12-23T08:55:12Z

     erg end() method
    
     remote-tracking branch 'upstream/master'

commit 6c7ec6a37286ae55e16f267f126342694eb02b0e
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-23T09:43:48Z

    Merge remote-tracking branch 'upstream/master'
    taking latest-23/12

commit e49f22fed8fe0a8798fa1fe729c33df105e54cb5
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-23T10:20:45Z

    Updated Pom files

commit 9e4fdba32171b86b2f67f0c73780a06f6ca1b187
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-23T10:24:04Z

    Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit 6357bdf3c151c8c7575abad0f0e2f4bc4a424d7a
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2016-12-23T14:41:23Z

    Added code and tests to read from DummyInputFormat

commit a2163e823d7de3a024913acbb6a4035b8d8b6dd3
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2016-12-23T14:48:38Z

     Resolved conflictsMerge branch 'master' of https://github.com/diptikul/incubator-beam

commit fb1f2de7e479920ac53de25ba18dd673046c69da
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-23T14:50:06Z

    Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit 571fcb40838a61148d0d49669ec18df35779b784
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-26T05:01:37Z

    Added Hadoop Input Format module in io pom file and minor changes in pom file.

commit a785180fc3b95f66a9b30a03c184fb7d637777ad
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-26T05:30:25Z

    Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit b29276ae953e9fff7ed6eb0c241d21e34122c5d2
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-26T09:49:39Z

    Moved MyCassandraRow to test folder

commit 70f05266328d62ba3a9127d54bc9f3769effbde4
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-26T10:51:29Z

    Removed CassandraRowCoder and Custom coder unit test, added MyCassandraRow in test, used
Avro coder for MyCassandraRow

commit f129ead54b1b7047f4c5ff592bc4f2e7ecc7bc43
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-26T11:56:58Z

    Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit 276c3c217df6c062b8dd4b5deda0e3425c775513
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2016-12-27T06:02:14Z

    Added test cases for validate method, getFractionsConsumed method

commit 020576283b10c7630bb38df8836387404a57b76e
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-27T07:03:24Z

    Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit c01544ffb61d6c129a932eef6d957e57c698b858
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2016-12-28T10:46:54Z

    Added junits for splitIntoBundles(),computeSplits and added autovalue builder.

commit 6e6819eec404ec131ec1f2ef98d985a2d51e4c9e
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-28T10:48:28Z

    Merge branch 'master' of https://github.com/diptikul/incubator-beam

commit 8ebbd2b124b5f40e67be19ef9caba8f1f3b3c6bc
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-28T10:56:50Z

    Made tests compile with read()

commit e0b291958e804e7c328be87ce860e63ea9253b20
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-28T11:23:24Z

    New package for dummy input formats, and add reference of DisplayDataMatchers for unit
test

commit a1b092a67fd0456a835fb3c4115c0b14a3f2d094
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2016-12-28T11:28:23Z

    Added Apache License header to all classes

commit 049473e3350fe12a2b4c217cc1cfae1737872135
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-30T07:44:52Z

    Merge remote-tracking branch 'upstream/master'

commit 46b13e4a3851b262cb1a0f51bf791cacee2b6d8f
Author: Radhika S Kulkarni <radhika_kulkarni1@persistent.co.in>
Date:   2016-12-30T07:55:01Z

    Added TypeDescriptors

commit b04eb728120fe42442184eecae16ac987cfc4226
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2016-12-30T09:55:21Z

    Added junits for testing immutability of HadoopInputFormatIO and junit for checking key/value
classes set by user in configuration.

commit 59b0f272a5432b2738416850161f0e61e5739d6f
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2017-01-03T06:29:21Z

    InputFormat changes, Exception thrown using annotation , testPipeline used for creation
of pipeline

commit 8f2a57071cfe83a8586328b3dd4c30d5c1596948
Author: Dipti Kulkarni <dipti_dkulkarni@persistent.co.in>
Date:   2017-01-03T07:26:06Z

    Added few comments, TestPipeline Usage

commit 995bfa9b32ee8e5ded8225ef5fa1e15e3b4cb2df
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2017-01-03T14:00:09Z

    Added cassandra inegration test and modified junits as per test review comments.

commit 6d5fc47d9a2fb74d5eec634fba4d63a6f87789f7
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2017-01-04T13:06:13Z

    Added logging into HadoopInputFormatIO and modifications in Junits and cassandra integration
test.

commit a5db53f986e4a8e95713f9e7d8a1e1d0a22eab59
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2017-01-05T12:56:49Z

    Modification in code according to code review comments and google code formatting.

commit 8d6392b058b08cdc7bc9b21623e35d57e996a29a
Author: Sheetal Tarodekar <sheetal_tarodekar@persistent.co.in>
Date:   2017-01-06T12:02:26Z

    Modifications as per code review comments.

----


> Hadoop InputFormat - IO Transform for reads
> -------------------------------------------
>
>                 Key: BEAM-1040
>                 URL: https://issues.apache.org/jira/browse/BEAM-1040
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Stephen Sisk
>            Assignee: Dipti Kulkarni
>
> We should build a IO read transform that will read data from services supporting the
Hadoop InputFormat  interface [1]
> This will make it easy to connect to a variety of databases while still providing some
aspects of scalability since the InputFormat interface provides for parallel reading. 
> [1] https://hadoop.apache.org/docs/r2.7.3/api/org/apache/hadoop/mapred/InputFormat.html



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message