Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Message-ID: <4D3A1A0D.2010604@yieldbuild.com>
Date: Fri, 21 Jan 2011 15:43:09 -0800
From: Alan Malloy <alan.malloy@yieldbuild.com>
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US;
 rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: Re: Losing Records with Block Compressed Sequence File
References: <AANLkTi=pkUenYZ5+91y7YP7ceymVrLjGH0RY-W1ScD91@mail.gmail.com>
In-Reply-To: <AANLkTi=pkUenYZ5+91y7YP7ceymVrLjGH0RY-W1ScD91@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Make sure to close the output writer? I had similar problems in a 
different scenario and it turned out I was neglecting to close/flush my 
output.

On 01/21/2011 01:04 PM, David Sinclair wrote:
> Hi, I am seeing an odd problem when writing block compressed sequence files.
> If I write 400,000 records into a sequence file w/o compression, all 400K
> end up in the file. If I write with block, regardless if it is bz2 or
> deflate, I start losing records. Not a ton, but a couple hundred.
>
> Here are the exact numbers
>
> bz2      399,734
> deflate  399,770
> none     400,000
>
> Conf settings
> io.file.buffer.size - 4K, io.seqfile.compress.blocksize - 1MB
>
> anyone ever see this behavior?
>
> thanks
>
> dave
>