hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yoram Arnon (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-374) native support for gzipped text files
Date Thu, 20 Jul 2006 18:15:13 GMT
native support for gzipped text files 
--------------------------------------

                 Key: HADOOP-374
                 URL: http://issues.apache.org/jira/browse/HADOOP-374
             Project: Hadoop
          Issue Type: New Feature
          Components: mapred
            Reporter: Yoram Arnon


in many cases it is convenient to store text files in dfs as gzip compressed files.
It would be good to have built in support for processing these files in a mapreduce job.

The getSplits implementation should return a single split per input file, ignoring the numSplits
parameter.
One can probably subclass InputFormatBase, and the getSplits method can simply call listPaths()

and then construct and return a single split per path returned.

The code for reading would look something like (courtesy of Vijay Murthy):

   public RecordReader getRecordReader(FileSystem fs, FileSplit split,
                                       JobConf job, Reporter reporter)
     throws IOException {
     final BufferedReader in =
       new BufferedReader(new InputStreamReader
         (new GZIPInputStream(fs.open(split.getPath()))));
     return new RecordReader() {
         long position;
         public synchronized boolean next(Writable key, Writable value)
           throws IOException {
           String line = in.readLine();
           if (line != null) {
             position += line.length();
             ((UTF8)value).set(line);
             return true;
           }
           return false;
         }
         public synchronized long getPos() throws IOException {
           return position;
         }
        public synchronized void close() throws IOException {
           in.close();
         }
       };
   }



-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message