apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Weise (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (APEXCORE-2) Enable HDFS write on streams as an attribute
Date Sun, 27 Dec 2015 23:57:49 GMT

     [ https://issues.apache.org/jira/browse/APEXCORE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Thomas Weise closed APEXCORE-2.
-------------------------------

> Enable HDFS write on streams as an attribute
> --------------------------------------------
>
>                 Key: APEXCORE-2
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-2
>             Project: Apache Apex Core
>          Issue Type: Task
>            Reporter: Gavin
>            Assignee: Chetan Narsude
>            Priority: Critical
>
> On a stream, enable spooling the streams to HDFS. This needs the following to be specified
> - Physical partitions: Since partitions are decided via input port, putting partitions
on stream (written to by output ports) requires special handling
> - Write policy (directory, partitionining to be consumed by consumers, hive?, hcatalog,
...)
> - Ability to ask for partitions to be derived from bandwidth. Assume one physical HDFS
write can only do 30 MB/sec. User should be able to say "derive the number of HDFS writers",
or "use N writers and grow them if throughput goes up"
> Amol



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message