hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrzej Bialecki ...@getopt.org>
Subject Re: "Too many open files" error after running a number of jobs
Date Tue, 17 Jul 2007 07:10:00 GMT
Shailendra Mudgal wrote:
> Hi ,
> We have upgraded our code to nutch-0.9 with hadoop-0.12.2-core.jar. After
> running say 50 nutch jobs(which includes inject/generate/fetch/parse etc.)
> we start getting "Too many open files" error on our cluster. We are using
> Linux box with kernel 2.6.9 and the open files number is 1024 on these
> machine which is default. I read several mails from nutch-user, or
> hadoop-user mailing lists. And i found only way was to increase the number
> of open files using ulimit. Is there any other solution for this problem at
> code level. BTW the value for io.sort.factor is 8 in our hadoop-site.xml.
> Is anybody having any idea in this regard? Any help will be appreciated.

Apparently datanodes that perform intensive IO operations need a higher 
limit. Try increasing this number to 16k or so.

Best regards,
Andrzej Bialecki     <><
  ___. ___ ___ ___ _ _   __________________________________
[__ || __|__/|__||\/|  Information Retrieval, Semantic Web
___|||__||  \|  ||  |  Embedded Unix, System Integration
http://www.sigram.com  Contact: info at sigram dot com

View raw message