Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Message-ID: <21988005.1193250291121.JavaMail.jira@brutus>
Date: Wed, 24 Oct 2007 11:24:51 -0700 (PDT)
From: "Doug Cutting (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-1986) Add support for a general
 serialization mechanism for Map Reduce
In-Reply-To: <8648883.1191358670825.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-1986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12537377 ] 

Doug Cutting commented on HADOOP-1986:
--------------------------------------

> This might be difficult if you have a serializer that can handle lots of classes.

I don't yet see the difficulty.

{noformat}
public class RecordSerializerFactory implements SerializerFactory {
  public Serializer getSerializer(Class c) {
    return new RecordSerializer(c);
  }
}

public class RecordSerializer {
  private Class recordClass;
  public RecordSerializer(Class c) {
    this.recordClass = c;
  }
  public void serialize(Object o, OutputStream) { ... }
  public Object deserialize(Object reuse, InputStream) {
    private Object instance = this.recordClass.newInstance();
    ...
    return instance;
  }
}
{noformat}
So record serialization can use the same method implementation to serialize all instances.  That method can know the class of the instance to be serialized.  And that method need not reuse an instance, but can always create a new instance.  What's difficult about this?

> Add support for a general serialization mechanism for Map Reduce
> ----------------------------------------------------------------
>
>                 Key: HADOOP-1986
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1986
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: mapred
>            Reporter: Tom White
>            Assignee: Tom White
>             Fix For: 0.16.0
>
>         Attachments: SerializableWritable.java, serializer-v1.patch
>
>
> Currently Map Reduce programs have to use WritableComparable-Writable key-value pairs. While it's possible to write Writable wrappers for other serialization frameworks (such as Thrift), this is not very convenient: it would be nicer to be able to use arbitrary types directly, without explicit wrapping and unwrapping.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.