hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ellis H. Wilson III" <el...@cse.psu.edu>
Subject Re: Error: Too Many Fetch Failures
Date Fri, 29 Jun 2012 00:30:46 GMT
On 06/19/12 23:10, Ellis H. Wilson III wrote:
> On 06/19/12 20:42, Raj Vishwanathan wrote:
>> You are probably having a very low somaxconn parameter ( default
>> centos has it at 128 , if I remember correctly). You can check the
>> value under /proc/sys/net/core/somaxconn
>
> Aha! Excellent, it does seem it's at the default, and that particular
> sysctl item had slipped my notice:
> [ellis@pool100 ~]$ cat /proc/sys/net/core/somaxconn
> 128
>
>> Can you also check the value of ulimit -n? It could be low.
>
> I did look for and alter this already, but it is set fairly high from
> what I can tell:
> [ellis@pool100 ~]$ ulimit -n
> 16384
>
> I altered both of these in /etc/sysctl.conf and have forced them to be
> re-read with `sysctl -p` on all nodes. I will report back if this fixes
> the issues tomorrow.

To anyone who runs into this problem in the future, I found that 
increasing the somaxconn parameter fixed the fetch failures issue 
completely (from 3 tests run so far on largish datasets).  This should 
be particularly useful for others who are dealing with an extremely high 
TaskTracker to DataNode ratio (10:1 in my case).

Thanks again to Raj for this solution, and others for their suggestions.

Best,

ellis

Mime
View raw message