hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Enis Soztutar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3702) add support for chaining Maps in a single Map and after a Reduce [M*/RM*]
Date Mon, 04 Aug 2008 14:42:44 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619529#action_12619529
] 

Enis Soztutar commented on HADOOP-3702:
---------------------------------------

# Having a distinct serializer for Configuration is not desired. Configuration is already
marshaled/unmarshaled to/from string form as XML. So having Configuration implement Writable
is the choice here. There is no ambiguity in the Configuration.write() method. We can keep
current write(OutputStream out) method and add : 
{code}
public void write(final DataOutput out) throws IOException {
    write(new OutputStream() {
      @Override
      public void write(int b) throws IOException {
        out.writeByte(b);
      }
    });
  }
{code}
The readFields() can be implemented by factoring common functionality of loadProperties()
method, instead of reading from a URL or file, the XML will be built using DataInput. 
# It would be better if ChainMapper, ChainReducer, ChainOutputCollector to use generics, similar
to Mapper, Reducer, and OutputCollector classes. 
# creating a new ChainOutputCollector() at each collect() call might be suboptimal. Could
this not be done once for each Map/Reduce task? 

> add support for chaining Maps in a single Map and after a Reduce [M*/RM*]
> -------------------------------------------------------------------------
>
>                 Key: HADOOP-3702
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3702
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>         Environment: all
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>            Priority: Minor
>         Attachments: patch3702.txt, patch3702.txt, patch3702.txt, patch3702.txt, patch3702.txt,
patch3702.txt, patch3702.txt, patch3702.txt, patch3702.txt, patch3702.txt
>
>
> On the same input, we usually need to run multiple Maps one after the other without no
Reduce. We also have to run multiple Maps after the Reduce.
> If all pre-Reduce Maps are chained together and run as a single Map a significant amount
of Disk I/O will be avoided. 
> Similarly all post-Reduce Maps can be chained together and run in the Reduce phase after
the Reduce.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message