hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <owen.omal...@gmail.com>
Subject Re: large block size problem
Date Mon, 16 Mar 2009 13:59:20 GMT
Since it is per a file, you'd need to check at file create too.

-- Owen

On Mar 16, 2009, at 4:29, Steve Loughran <stevel@apache.org> wrote:

> Steve Loughran wrote:
>> Owen O'Malley wrote:
>>> I seem to remember someone saying that blocks over 2^31 don't  
>>> work. I don't know if there is a jira already.
>> Looking at the stack trace, int is being used everywhere, which  
>> implies an upper limit of (2^31)-1, for blocks. Easy to fix, though  
>> it may change APIs, and then there is the testing.
>
>
> thinking about this a bit more, a quick early patch would be to  
> print a warning whenever you try to bring up a namenode with a block  
> size >= 2GB ; have the system continue so that people can test and  
> fix the code, but at least it stops end users being surprised.
>
> I spoke with someone from the local university on their High Energy  
> Physics problems last week -their single event files are about 2GB,  
> so that's the only sensible block size to use when scheduling work.  
> He'll be at ApacheCon next week, to make his use cases known.
>
> -steve

Mime
View raw message