hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1284) clean up the protocol between stream mapper/reducer and the framework
Date Wed, 25 Apr 2007 18:39:15 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491725
] 

Hadoop QA commented on HADOOP-1284:
-----------------------------------

+1

http://issues.apache.org/jira/secure/attachment/12356261/patch-1284.txt applied and successfully
tested against trunk revision r532083.

Test results:   http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/78/testReport/
Console output: http://lucene.zones.apache.org:8080/hudson/job/Hadoop-Patch/78/console

> clean up the protocol between stream mapper/reducer and the framework
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-1284
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1284
>             Project: Hadoop
>          Issue Type: Improvement
>            Reporter: Runping Qi
>         Assigned To: Runping Qi
>         Attachments: patch-1284.txt
>
>
> Right now, the protocol between stream mapper/reducer and the framework is very inflexible.
> The mapper/reducer generates line oriented output. The framework picks up line by line,
and split 
> each line into a key/value pair. By default, the substring up to the first tab char is
the key, and the 
> substring after the first tab char is the value.
> However, in many cases, the application wants some control over how the pair is split.

> Here, I'd like to introduce the following configuration variables for that:
> 1. "streaming.output.field.separator": the value will be the tab key, by default. 
> But the user can specify a different one (e.g. ':', or ', ', etc.)
> A map output line can be considered as a list of fields separated by the separator.
> 2. "streaming.num.fields.for.mapout.key":  the number of the first fields will be used
the map output key  
> (and for sorting in the reduce side). 
> The default value is 1.
> The rest of the fields will be used as the value.  For example, I can specify the first
5 fields as my mapout key.
> 3. "streaming.num.fields.for.partitioning": Sometimes, I want to use fewer fields for
partitioning to 
> achieve "primary/secondary" composite 
> key effect as proposed in HADOOP485. The default value is 1. 
> For example, I can set "streaming.num.fields.for.partitioning" to 3 
> and "streaming.num.fields.for.mapout.key" to 5. 
> This effectively amounts to saying that fields 4 and 5 are my secondary key.
> With the above default values, it is compatible with the current behavior 
> while introducing a new desirable feature in a clean way.
> Thoughts?

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message