hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-4243) Serialization framework use SequenceFile/TFile/Other metadata to instantiate deserializer
Date Mon, 22 Sep 2008 18:30:44 GMT
Serialization framework use SequenceFile/TFile/Other metadata to instantiate deserializer

                 Key: HADOOP-4243
                 URL: https://issues.apache.org/jira/browse/HADOOP-4243
             Project: Hadoop Core
          Issue Type: Improvement
          Components: contrib/serialization
            Reporter: Pete Wyckoff

SequenceFile metadata is useful for storing additional information about the serialized data,
for example, for RecordIO, whether the data is CSV or Binary.  For thrift, the same thing
- Binary, JSON, ...

For Hive, this may be especially important, because it has a Dynamic generic serializer/deserializer
that takes its DDL at runtime (as opposed to RecordIO and Thrift which require pre-compilation
into a specific class whose name can be stored in the sequence file key or value class). 
 In this case, the class name is like Record.java in RecordIO - it doesn't tell you anything
without the DDL.

One way to address this could be adding the sequence file metadata to the getDeserializer
call in Serialization interface.  The api would then be something like getDeserializer(Class<?>,
Map<Text, Text> metadata) or Properties metadata.

But, I am open to proposals.

This also means that saying a class implements Writable is not enough to necessarily deserialize
it since it may do specific actions based on the metadata - e.g., RecordIO might determine
whether to use CSV rather than the default Binary deserialization.

There's the other issue of the getSerializer returning the metadata to be written to the Sequence/T

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message