apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2013) HDFS output module for file copy
Date Tue, 19 Apr 2016 11:36:25 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247591#comment-15247591
] 

ASF GitHub Bot commented on APEXMALHAR-2013:
--------------------------------------------

GitHub user yogidevendra reopened a pull request:

    https://github.com/apache/incubator-apex-malhar/pull/216

    APEXMALHAR-2013 : HDFS output module for file copy

    1. Added operators required for file copy
    2. Added junit tests
    3. Added sample app for HDFS to HDFS file copy app

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yogidevendra/incubator-apex-malhar APEXMALHAR-2013-hdfs-file-copy-module-PR2

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-apex-malhar/pull/216.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #216
    
----
commit 622d8393ed4ab421e88d37bd054e8a0979c1b09e
Author: yogidevendra <devendra@datatorrent.com>
Date:   2016-03-04T07:12:17Z

    APEXMALHAR-2013 : HDFS output module for file copy
    1. Added operators BlockWriter, Synchronizer, FileStitcher,
       FileMerger, HDFSFileMerger
    2. Added junit tests
    3. Added sample app for HDFS to HDFS file copy app
    
    4. Incorporating review comments
    
    5. Setting block threshold same as number of readers
    
    6. Adding limit on mvn memory usage.

commit 0c5494f203af73bc1b76fc54246fc173fe10a468
Author: yogidevendra <devendra@datatorrent.com>
Date:   2016-04-19T11:23:58Z

    Adding max violations for benchmark

----


> HDFS output module for file copy
> --------------------------------
>
>                 Key: APEXMALHAR-2013
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2013
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: Yogi Devendra
>            Assignee: Yogi Devendra
>
> To write files to HDFS using block-by-block approach.
> Main use-case being to copy the files. Thus, original sequence of blocks has to be maintained.

> To achieve this goal, this module would use information emitted by  HDFS input module
(APEXMALHAR-2008) viz. FileMetaData, BlockMetaData, BlockData.
>   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message