Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Received-SPF: neutral (asf.osuosl.org: local policy)
Message-ID: <44062D7C.8030604@apache.org>
Date: Wed, 01 Mar 2006 15:25:48 -0800
From: Doug Cutting <cutting@apache.org>
User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051013)
MIME-Version: 1.0
To: hadoop-dev@lucene.apache.org
Subject: Re: scalability limits getDetails, mapFile Readers?
References: <0C02C8B2-4C2F-4E49-B5D8-C84075C9CFC0@media-style.com>
In-Reply-To: <0C02C8B2-4C2F-4E49-B5D8-C84075C9CFC0@media-style.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Stefan,

I think you meant to send this to nutch-dev, not hadoop-dev.

Doug

Stefan Groschupf wrote:
> Hi,
> 
> We run into a problem with nutch using  MapFileOutputFormat#getReaders  
> and getEntry.
> In detail this happens until summary generation where we open for  each 
> segment as much readers as much parts (part-0000 to part-n) we  have.
> Having 80 tasktracker and 80 segments means:
> 80 x 80 x 4 (parseData, parseText, content, crawl). A search server   
> also needs to open as much files as required for the index searcher.
> So the problem is a FileNotFoundException, (Too many open files).
> 
> Opening and closing Readers for each Detail makes no sense. We may  can 
> limit the number of readers somehow and close the readers that  wasn't 
> used since the longest time.
> But I'm not that happy with this solution, so any thoughts how we can  
> solve this problem in general?
> 
> Thanks.
> Stefan
> 
> 
> ---------------------------------------------
> blog: http://www.find23.org
> company: http://www.media-style.com
> 
>