hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Runping Qi (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1247) Make Hadoop Abacus work with Hadoop Streaming
Date Tue, 17 Apr 2007 18:31:15 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Runping Qi updated HADOOP-1247:
-------------------------------

    Attachment: patch_1247.txt


The attached patch made the following code changes:

1. Added a new option for StreamJob: -reduce abacus
When this option is used, the Hadoop Abacus combiner and reducer will be used. The mapper
is expected to 
generate output that conforms to the Abacus protocol (i.e. each line will be something like
LongValueSum:ID\t1"

2. Changed build.xml under contrib/streaming to include the source code of Hadoop Abacus 
in compile.

3. Added unit test code in src/contrib/streaming/src/test/org/apache/hadoop/streaming/TestStreamAbacus.java
and src/contrib/streaming/src/test/org/apache/hadoop/streaming/StreamAbacus.java

All the unit tests passed.


> Make Hadoop Abacus work with Hadoop Streaming
> ---------------------------------------------
>
>                 Key: HADOOP-1247
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1247
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: contrib/streaming
>            Reporter: Runping Qi
>         Assigned To: Runping Qi
>         Attachments: patch_1247.txt
>
>
> It will be nice if Hadoop streaming can use Hadoop Abacus.
> This requires to make some changes on StreamJob and StreamMapRed classes so that the
streaming job uses the Abacus reducer/combiner classes, and streaming mapper generates intermediate
data conforming to Hadoop Abacus protocol.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message