Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <1859715739.1212536445197.JavaMail.jira@brutus>
Date: Tue, 3 Jun 2008 16:40:45 -0700 (PDT)
From: "Srikanth Kakani (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-3315) New binary file format
In-Reply-To: <774412089.1209158875801.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602114#action_12602114 ] 

Srikanth Kakani commented on HADOOP-3315:
-----------------------------------------

Owen,

There would be one complication in exposing the append(long keyLength, long valueLength) that we did not discuss earlier. Although it can be handled.

If it the key,value is at the beginning of a block we need to copy to a byte array in the key.serialize(outputstream). We can do this by having a keyValueOutputStream(keybytes,valuebytes, outputstream), that captures the first keybytes of data written into a buffer. This needs to be done to generate an index. But it starts getting ugly.

I would also suggest ObjectFile should be extending the TFile and it can do all this in a neater fashion without exposing the append(keyLength, valueLength).

Additionally to make any of this feasible (You mentioned this earlier, I just want to record it), serializers should also have getSerializedLength().

> New binary file format
> ----------------------
>
>                 Key: HADOOP-3315
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3315
>             Project: Hadoop Core
>          Issue Type: New Feature
>          Components: io
>            Reporter: Owen O'Malley
>            Assignee: Srikanth Kakani
>         Attachments: Tfile-1.pdf, TFile-2.pdf
>
>
> SequenceFile's block compression format is too complex and requires 4 codecs to compress or decompress. It would be good to have a file format that only needs 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.