hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6660) add Writable wrapper for Avro data
Date Thu, 25 Mar 2010 20:15:27 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12849879#action_12849879
] 

Doug Cutting commented on HADOOP-6660:
--------------------------------------

A feature of this proposal is that it can easily be ported to 0.20-based releases.

We would also define a RawComparator and use WritableCompator#define() to register this comparator.

Two Writable classes would be implemented:
{code}
public class AvroValue implements Writable {
  private Object value;
  private Schema schema;
  private DatumWriter writer;
  private DatumReader reader;
  ...
}
// marker class for keys
public class AvroKey extends AvroValue implements Comparable {}
{code}

Instances would be constructed with a factory, e.g.:

{code}
public class AvroValueFactory {
  public static makeGeneric(Schema schema);
  public static makeSpecific(Schema schema);

  public AvroValue value(Object datum);
  public AvroKey key(Object datum);
}
{code}

A SerializationFactory would be defined for these classes.  MapReduce passes the serialization
factory a class name and a configuration.  One of four configuration properties would be used
to identify the schema, based on whether the class name is AvroKey or AvroValue and whether
the configuration specifies mapred.task.is.map or not, thus permitting jobs to specify distinct
Avro schemas for map and reduce output keys and values.


> add Writable wrapper for Avro data
> ----------------------------------
>
>                 Key: HADOOP-6660
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6660
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: io
>            Reporter: Doug Cutting
>             Fix For: 0.22.0
>
>
> To permit passing Avro data through MapReduce we can add a wrapper class that implements
Writable.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message