apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Weise (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (APEXMALHAR-2369) S3 output module for tuple based output
Date Tue, 28 Mar 2017 05:26:41 GMT

     [ https://issues.apache.org/jira/browse/APEXMALHAR-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Weise updated APEXMALHAR-2369:
-------------------------------------
    Issue Type: New Feature  (was: Task)

> S3 output module for tuple based output
> ---------------------------------------
>
>                 Key: APEXMALHAR-2369
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2369
>             Project: Apache Apex Malhar
>          Issue Type: New Feature
>            Reporter: Yogi Devendra
>            Assignee: Yogi Devendra
>             Fix For: 3.7.0
>
>
> Currently, S3 output is available using S3OutputModule which is restricted for copying
files from FileSystem to S3. Use-cases where all the tuples/records to be written to S3 cannot
use this approach. Thus, we need to develop alternative module which would take care of writing
tuples on S3. 
> Design: 
> Sending separate requests to S3 for each tuple would be too expensive. This module can
choose to write tuples to HDFS. And then upload HDFS files to S3. This would lead to some
end-to-end latency. But, it should OK for the S3 output case.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message