hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3019) want input sampler & sorted partitioner
Date Fri, 14 Mar 2008 20:12:28 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12578915#action_12578915
] 

Doug Cutting commented on HADOOP-3019:
--------------------------------------

> Should this be a part of examples like sort?

I guess  it could live with the examples, but I was thinking that this would be more like
mapred/lib.  The sampler should be generic enough that folks won't have to modify it to find
it useful: it should work for different key/value types and for sequencefile and text data.

> want input sampler & sorted partitioner
> ---------------------------------------
>
>                 Key: HADOOP-3019
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3019
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Doug Cutting
>
> The input sampler should generate a small, random sample of the input, saved to a file.
> The partitioner should read the sample file and partition keys into relatively even-sized
key-ranges, where the partition numbers correspond to key order.
> Note that when the sampler is used for partitioning, the number of samples required is
proportional to the number of reduce partitions.  10x the intended reducer count should give
good results.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message