hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Gutierrez <este...@cloudera.com>
Subject Re: Region server not accept connections intermittently
Date Mon, 14 Jul 2014 04:32:31 GMT
Hello Rural,

Thats interesting, unless you have changed ipc.server.listen.queue.size in
the HBase Region Server (and other Hadoop daemons) to value higher than
128, you might have worked around the issue by increasing the listen queue
(globally) for a service that doesn't explicitly set the queue size. See
the notes section in http://linux.die.net/man/2/listen

cheers,
esteban.



--
Cloudera, Inc.



On Fri, Jul 11, 2014 at 6:40 PM, Rural Hunter <ruralhunter@gmail.com> wrote:

> the max number of files has already been set to 32768 for the user running
> hbase/hadoop. I think there should be errors in log if it's the file number
> problem. The count of connections in SYN_RECV state is about 100. I also
> checked the source of those connections and they are from the hosts of my
> hbase clients. Anyway, I just changed these kernel settings:
> net.core.somaxconn=1024 (original 128)
> net.ipv4.tcp_synack_retries=2 (original 5)
>
> I will wait and see if this will happen again.
>
> 于 2014/7/12 1:57, Esteban Gutierrez 写道:
>
>  For how long you noticed that connections? when you say "many" do you mean
>> 1000s? the problem with having too many syn_recv is that you could end
>> running out of file descriptors, which makes me wonder know what is the
>> maximum number of open files that you have configured for the RS process
>> (see all the root->hbase user->hbase proc chain of ulimits) and if there
>> could be a rogue client connecting to 60020 (maybe a port scanner?) and
>> triggering that problem.
>>
>> cheers,
>> esteban.
>>
>>
>>
>>
>> --
>> Cloudera, Inc.
>>
>>
>>
>> On Fri, Jul 11, 2014 at 3:02 AM, Rural Hunter <ruralhunter@gmail.com>
>> wrote:
>>
>>  One additional info, I did 'netstat -an |grep 60020' when the problem
>>> happened, I saw many connections from remote to local port 60020 are on
>>> state "SYN_RECV". Not sure if that indicates anything.
>>>
>>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message