hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-441) SequenceFile should support 'custom compressors'
Date Thu, 24 Aug 2006 06:47:57 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-441?page=comments#action_12430068 ] 
            
Arun C Murthy commented on HADOOP-441:
--------------------------------------

Doug,

  Conceptually this makes sense, I'm all for it.
  
  There is one irritant with the create methods for DeflaterOutputStream/InflaterInputStream
in CompressionCodec... it's got to do with the Deflater/Inflater fields for streams respectively.
Specifically the issue is we need to pass in our own Deflater/Inflater objects in order to
be able to 'reset' the internal compressor/decompressor of the respective streams; this ability
to 'reset' is otherwise absent (since we can't access those 'protected' fields) as is crucial
for SequenceFile's block compression (in the case where next(key,val) is followed by next(key)
and block boundary is hit). We probably need to define our own interface for compressor/decompressor
streams or atleast have the CompressionCodec work with an extended class of DeflaterOutputStream/InflaterInputStream
(which will have the 'reset' functionality we need).

 Thoughts?

> SequenceFile should support 'custom compressors'
> ------------------------------------------------
>
>                 Key: HADOOP-441
>                 URL: http://issues.apache.org/jira/browse/HADOOP-441
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Arun C Murthy
>         Assigned To: Arun C Murthy
>             Fix For: 0.6.0
>
>
> SequenceFiles should support 'custom compressors' which can be specified by the user
on creation of the file. 
> Readily available packages for gzip and zip (java.util.zip) are among obvious choices
to support. Of course there will be hooks so that other compressors can be added in future
as long as there is a way to construct (input/output) streams on top of the compressor/decompressor.
> The 'classname' of the 'custom compressor/decompressor' could be stored in the header
of the SequenceFile which can then be used by SequenceFile.Reader to figure out the appropriate
'decompressor'. Thus I propose we add constructors to SequenceFile.Writer which take in the
'classname' of the compressor's input/output stream classes (e.g. DeflaterOutputStream/InflaterInputStream
or GZIPOutputStream/GZIPInputStream), which acts as the hook for future compressors/decompressors.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message