hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5979) Streaming partitioner should allow command, not just Java class
Date Tue, 09 Jun 2009 08:20:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717602#action_12717602

Devaraj Das commented on HADOOP-5979:

bq. it's simpler (the partitioner command doesn't need to know how many partitions there are),
But still, the command needs to have an idea of how many partitions there are, isn't it? Or
maybe, you are saying that it's up to the command developer to assume a certain partition
count and implement the command... I agree that it's simple but am not sure whether all use
cases would be covered with this model..

bq. we could reuse more code that's already there (if we let the the partitioner command output
both a key and a value and pass that on to a wrapped partitioner, like in the code sample
I gave above, we even wouldn't need any additional reading/writing logic).
What did you mean by "we wouldn't need any additional reading/writing logic" ? There is at
least that much reading/writing as your code outlined, ist it?

> Streaming partitioner should allow command, not just Java class
> ---------------------------------------------------------------
>                 Key: HADOOP-5979
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5979
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: contrib/streaming
>            Reporter: Klaas Bosteels
> Since HADOOP-4842 got committed, Streaming allows both commands and Java classes to be
specified as mapper, reducer, and combiner, but the {{-partitioner}} option is still limited
to Java classes only. Allowing commands to be specified as partitioner as well would greatly
improve the flexibility of Streaming programs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message