Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MailScanner-NULL-Check: 1304507375.64077@zp2TNzIG1HHtBB3LPVv/wg
Message-ID: <4DB7F96F.3040307@apache.org>
Date: Wed, 27 Apr 2011 12:09:35 +0100
From: Steve Loughran <stevel@apache.org>
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US;
 rv:1.9.2.14) Gecko/20110223 Thunderbird/3.1.8
MIME-Version: 1.0
To: common-user@hadoop.apache.org
Subject: Re: Unsplittable files on HDFS
References: <BANLkTi=dtS=W11ra5TVdwdYUXKCJJS=F2w@mail.gmail.com>
	<BANLkTinJ7Tv2qMwRn6zWgocbCOkFDEKHiA@mail.gmail.com>
 <BANLkTinRTF_PATq=RH=KtafLA-x3XLmsFw@mail.gmail.com>
In-Reply-To: <BANLkTinRTF_PATq=RH=KtafLA-x3XLmsFw@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 27/04/11 10:48, Niels Basjes wrote:
> Hi,
>
> I did the following with a 1.6GB file
>     hadoop fs -Ddfs.block.size=2147483648 -put
> /home/nbasjes/access-2010-11-29.log.gz /user/nbasjes
> and I got
>
> Total number of blocks: 1
> 4189183682512190568:	 	10.10.138.61:50010	 	10.10.138.62:50010
>
> Yes, that does the trick. Thank you.
>
> Niels
>
> 2011/4/27 Harsh J<harsh@cloudera.com>:
>> Hey Niels,
>>
>> The block size is a per-file property. Would putting/creating these
>> gzip files on the DFS with a very high block size (such that it
>> doesn't split across for such files) be a valid solution to your
>> problem here?
>>

Don't set a block size >2GB, not all the bits of the code that use 
signed 32 bit integers have been eliminated yet.