hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1986) Add support for a general serialization mechanism for Map Reduce
Date Wed, 02 Apr 2008 22:28:24 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom White updated HADOOP-1986:
------------------------------

    Release Note: 
Programs that implement the raw Mapper or Reducer interfaces will need modification to compile
with this release. For example, 

class MyMapper implements Mapper {
  public void map(WritableComparable key, Writable val,
    OutputCollector out, Reporter reporter) throws IOException {
    // ...
  }
  // ...
}

will need to be changed to refer to the parameterized type. For example:

class MyMapper implements Mapper<WritableComparable, Writable, WritableComparable, Writable>
{
  public void map(WritableComparable key, Writable val,
    OutputCollector<WritableComparable, Writable> out, Reporter reporter) throws IOException
{
    // ...
  }
  // ...
}

Similarly implementations of the following raw interfaces will need modification: InputFormat,
OutputCollector, OutputFormat, Partitioner, RecordReader, RecordWriter 
    Hadoop Flags: [Incompatible change, Reviewed]  (was: [Reviewed, Incompatible change])

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.17.0
>
>         Attachments: hadoop-serializer-v2.tar.gz, SequenceFileWriterBenchmark.java, SerializableWritable.java,
serializer-v1.patch, serializer-v2.patch, serializer-v3.patch, serializer-v4.patch, serializer-v5.patch,
serializer-v6.patch, serializer-v7.patch
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs.
While it's possible to write Writable wrappers for other serialization frameworks (such as
Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types
directly, without explicit wrapping and unwrapping.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message