hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-522) MapFile should support block compression
Date Mon, 11 Sep 2006 23:16:24 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-522?page=all ]

Doug Cutting updated HADOOP-522:
--------------------------------

    Attachment: block-compress-map-file.patch

This is a quick hack to try to test whether MapFile & SetFile will work with block compression.
 It currently fails.

To illustrate the problem:

ant compile-core-test
bin/hadoop org.apache.hadoop.io.TestSetFile -local foo


> MapFile should support block compression
> ----------------------------------------
>
>                 Key: HADOOP-522
>                 URL: http://issues.apache.org/jira/browse/HADOOP-522
>             Project: Hadoop
>          Issue Type: New Feature
>          Components: io
>            Reporter: Doug Cutting
>         Attachments: block-compress-map-file.patch
>
>
> MapFile is layered on SequenceFile and permits random-access to sorted data files (typically
reduce output) through a parallel index file.  This is used widely in Nutch (e.g. at search
time for displaying cached pages, incoming links, etc).  Such sorted data should benefit from
block compression, but the current MapFile API does not support specification of block compression.
 Also, even if it did, the semantics of SequenceFile methods like seek() and getPosition()
are changed under block compression so that MapFile may not work.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message