hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-474) support compressed text files as input and output
Date Fri, 08 Sep 2006 17:30:25 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-474?page=comments#action_12433449 ] 
Doug Cutting commented on HADOOP-474:

> I think having the [gs]etCompressOutput methods in JobConf is reasonable since they are
intended to be used by all of the OutputFormats. 

Yes, they're used by the output formats we supply, but probably not by all output formats
that folks might define.  To be consistent, I think we should adopt the following rule for
JobConf: it should only set properties which are used by mapreduce kernel code, not by code
in pluggable user classes.  This has obviously not been the policy prior to this.  There are
exceptions in the current code.  But I think this will lead to better code organization. 
Long-term, access to things like mapred.input.dir should move to InputFormatBase.  Glancing
through JobConf, there are only a few such things that would need to move: the vast majority
of parameters set in JobConf are already kernel stuff.  So we're close to implementing this
rule and, if we elect to observe it, shouldn't stray further.

Do you think this is a reasonable rule?

Also, I think compressor should be spelled with an 'o'.

> support compressed text files as input and output
> -------------------------------------------------
>                 Key: HADOOP-474
>                 URL: http://issues.apache.org/jira/browse/HADOOP-474
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.5.0
>            Reporter: Owen O'Malley
>         Assigned To: Owen O'Malley
>             Fix For: 0.6.0
>         Attachments: text-gz-2.patch, text-gz.patch
> I'd like TextInputFomat and TextOutputFormat to automatically compress and uncompress
text files when they are read and written. Furthermore, I'd like to be able to use custom
compressors as defined in HADOOP-441. Therefore, I propose:
> Adding a map of compression codecs in the server config files:
> io.compression.codecs = "<suffix>=<codec class>,..."
> so the default would be something like:
> <property>
>   <name>io.compression.codecs</name>
>   <value>.gz=org.apache.hadoop.io.GZipCodec,.Z=org.apache.hadoop.io.ZipCodec</value>
>   <description>A list of file suffixes and the codecs for them.</description>
> </property>
> note that the suffix can include multiple "." so you could support suffixes like ".tar.gz",
but they are just treated as literals against the end of the filename.
> If the TextInputFormat is dealing with such a file, it:
>   1. makes a single split
>   2. decompresses automatically
> On the output side, if mapred.output.compress is true, then TextOutputFormat would use
a new property mapred.output.compression.codec that would define the codec to use to compress
the outputs,  defaulting to gzip. 

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message