Mailing-List: contact zookeeper-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: zookeeper-user@hadoop.apache.org
MIME-Version: 1.0
In-Reply-To: <AANLkTinFajLQR-VW4CA9g64xmnvN53=CiOP2JuOPF593@mail.gmail.com>
References: <AANLkTimtQM5v0-6VkuJr9ZF-xd6sU_0_Ovut=4HyKiTQ@mail.gmail.com>
	<AANLkTim77fuHseuw5WxnUmhWkvo3FHk=+y2xB6o0tFcN@mail.gmail.com>
	<AANLkTinFajLQR-VW4CA9g64xmnvN53=CiOP2JuOPF593@mail.gmail.com>
Date: Tue, 5 Oct 2010 09:47:30 -0700
Message-ID: <AANLkTimrp5KwsGWfjDYzfOLGqrar2s5ortHYaoCHToRP@mail.gmail.com>
Subject: Re: Too many connections
From: Patrick Hunt <phunt@apache.org>
To: zookeeper-user <zookeeper-user@hadoop.apache.org>,
 avinash.lakshman@gmail.com
Content-Type: multipart/alternative; boundary=001636416ae3ac43d60491e16aea

--001636416ae3ac43d60491e16aea
Content-Type: text/plain; charset=ISO-8859-1

"A" (/10.30.84.207 a zookeeper client) is attempting to establish more then
10 sessions to the ZooKeeper server where you got the log. This can be
caused by a bug in user code (we've seen bugs where incorrectly implemented
ZK clients attempt to create an infinite number of sessions, which
essentially DOS the service, so we added the maxClientCnxn default limit of
10).

Often users see this problem when they are trying to simulate a real
environment - they run a simulated set of clients sessions (>10) from a
single host (ip) hitting the servers. However in your case I'm guessing that
it has something to do with this

"The observers are running in a different data center from where the ZK
non-observers are running."

Could you have a NAT or some other networking configuration that makes all
the observers seem to be coming from the same IP address?

Patrick

On Tue, Oct 5, 2010 at 9:33 AM, Avinash Lakshman <avinash.lakshman@gmail.com
> wrote:

> Thanks Patrick. But what does this mean? I see the log on server A telling
> me "Too many connections from A - default is 10". Too many connection from
> A
> to whom? I do not see who the other end of the connection is.
>
> Cheers
> Avinash
>
> On Tue, Oct 5, 2010 at 9:27 AM, Patrick Hunt <phunt@apache.org> wrote:
>
> > See this configuration param in the docs "maxClientCnxns":
> >
> >
> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
> >
> > <
> >
> http://hadoop.apache.org/zookeeper/docs/current/zookeeperAdmin.html#sc_advancedConfiguration
> > >
> > Patrick
> >
> > On Tue, Oct 5, 2010 at 8:10 AM, Avinash Lakshman <
> > avinash.lakshman@gmail.com
> > > wrote:
> >
> > > I find this happening in my observers node in the logs. The observers
> are
> > > running in a different data center from where the ZK non-observers are
> > > running. The only way to fix this seems to be restarting. How can I
> start
> > > addressing this? Here is the stack trace.
> > >
> > > Too many connections from /10.30.84.207 - max is 10
> > > WARN - Session 0x0 for server mybox.mydomain.com/10.30.84.207:5001,
> > > unexpected error, closing socket connection and attempting reconnect
> > > java.io.IOException: Connection reset by peer
> > >        at sun.nio.ch.FileDispatcher.read0(Native Method)
> > >        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
> > >        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
> > >        at sun.nio.ch.IOUtil.read(IOUtil.java:200)
> > >        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> > >        at
> > > org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:817)
> > >        at
> > > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1089)
> > >
> > > Please advice.
> > >
> > > Cheers
> > > Avinash
> > >
> >
>

--001636416ae3ac43d60491e16aea--