hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arun C Murthy <ar...@yahoo-inc.com>
Subject Re: FileUtil.copyMerge and SequenceFiles. 0-15.1
Date Sat, 29 Dec 2007 10:15:58 GMT
On Fri, Dec 28, 2007 at 11:53:42AM -0800, Jason Venner wrote:
>Is it safe to use this to generate a single SequenceFile out of a set of 
>sequence files produced by reduce?
>

Nope.

FileUtil.copyMerge just copies bytes of src files into one large heap of a destination file.
This will break if src files are SequenceFiles since we now have multiple headers mixed with
data.

I've opened http://issues.apache.org/jira/browse/HADOOP-2501 to cover _merge_ and other useful
utilities for SequenceFiles.

>this seems to be the source of my damaged sequence files.
>

Arun


Mime
View raw message