Return-Path: Delivered-To: apmail-lucene-hadoop-user-archive@locus.apache.org Received: (qmail 90628 invoked from network); 29 Dec 2007 10:18:29 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 29 Dec 2007 10:18:29 -0000 Received: (qmail 67891 invoked by uid 500); 29 Dec 2007 10:18:10 -0000 Delivered-To: apmail-lucene-hadoop-user-archive@lucene.apache.org Received: (qmail 67874 invoked by uid 500); 29 Dec 2007 10:18:10 -0000 Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-user@lucene.apache.org Delivered-To: mailing list hadoop-user@lucene.apache.org Received: (qmail 67865 invoked by uid 99); 29 Dec 2007 10:18:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Dec 2007 02:18:10 -0800 X-ASF-Spam-Status: No, hits=1.2 required=10.0 tests=SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [216.145.54.173] (HELO mrout3.yahoo.com) (216.145.54.173) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 29 Dec 2007 10:17:45 +0000 Received: from arunc.eglbp.corp.yahoo.com (arunc.eglbp.corp.yahoo.com [10.66.74.226]) by mrout3.yahoo.com (8.13.6/8.13.6/y.out) with ESMTP id lBTAFxYx021560 for ; Sat, 29 Dec 2007 02:16:00 -0800 (PST) DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns; h=received:x-authentication-warning:date:from:to:subject: message-id:references:mime-version:content-type: content-disposition:in-reply-to:user-agent; b=kBU0AxW6b93Pac/FY2oPwk1nZTDAx22O84TfWbHlXEvjWb6JDeGuHGZ+hdK1xv0s Received: (from arunc@localhost) by arunc.eglbp.corp.yahoo.com (8.13.3/8.13.3) id lBTAFwnI039503 for hadoop-user@lucene.apache.org; Sat, 29 Dec 2007 15:45:58 +0530 (IST) (envelope-from arunc@yahoo-inc.com) X-Authentication-Warning: arunc.eglbp.corp.yahoo.com: arunc set sender to arunc@yahoo-inc.com using -f Date: Sat, 29 Dec 2007 15:45:58 +0530 From: Arun C Murthy To: hadoop-user@lucene.apache.org Subject: Re: FileUtil.copyMerge and SequenceFiles. 0-15.1 Message-ID: <20071229101558.GA94666@yahoo-inc.com> References: <47755446.4000404@attributor.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <47755446.4000404@attributor.com> User-Agent: Mutt/1.4.2.1i X-Virus-Checked: Checked by ClamAV on apache.org On Fri, Dec 28, 2007 at 11:53:42AM -0800, Jason Venner wrote: >Is it safe to use this to generate a single SequenceFile out of a set of >sequence files produced by reduce? > Nope. FileUtil.copyMerge just copies bytes of src files into one large heap of a destination file. This will break if src files are SequenceFiles since we now have multiple headers mixed with data. I've opened http://issues.apache.org/jira/browse/HADOOP-2501 to cover _merge_ and other useful utilities for SequenceFiles. >this seems to be the source of my damaged sequence files. > Arun