Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1923573096.1221002384687.JavaMail.jira@brutus>
Date: Tue, 9 Sep 2008 16:19:44 -0700 (PDT)
From: "Pete Wyckoff (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4065) support for reading binary data
 from flat files
In-Reply-To: <165315971.1220482544298.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629654#action_12629654 ] 

Pete Wyckoff commented on HADOOP-4065:
--------------------------------------

Yes, good point.

I will change it to DeserializerTypedFile.

But, the SequenceFileRecordReader is re-usable for all these.  From the reader of a file that does its own deserializing of its types, it's all the same.

With this interface, the SequenceFileRecordReader can read SequenceFiles, DeserializerTypedFiles (thrift, proto buffers, record io whatever) and any other self describing typed files; sequencefile's being one example of these.

Otherwise, I don't see how not to be re-implementing the current SequenceFileRecordReader functionality for all these use cases??

-- pete


> support for reading binary data from flat files
> -----------------------------------------------
>
>                 Key: HADOOP-4065
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4065
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Joydeep Sen Sarma
>         Attachments: HADOOP-4065.0.txt, ThriftFlatFile.java
>
>
> like textinputformat - looking for a concrete implementation to read binary records from a flat file (that may be compressed).
> it's assumed that hadoop can't split such a file. so the inputformat can set splittable to false.
> tricky aspects are:
> - how to know what class the file contains (has to be in a configuration somewhere).
> - how to determine EOF (would be nice if hadoop can determine EOF and not have the deserializer throw an exception  (which is hard to distinguish from a exception due to corruptions?)). this is easy for non-compressed streams - for compressed streams - DecompressorStream has a useful looking getAvailable() call - except the class is marked package private.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.