zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Hunt <ph...@apache.org>
Subject Re: chatty error: Exception causing close of session 0x0 due to java.io.IOException: Len error 1835955314
Date Wed, 18 Jan 2012 02:00:58 GMT
Hi Alexius, it looks like localhost (127.0.0.1) is making a socket
connection and then sending a request with a supposed size of 1.8GB.
(the first 4 bytes of each zk request are the length)

It's likely that this is a "4letterword" attempt gone wrong (or some
other host incorrectly pinging this port). Which application are you
pointing at the ZK server's client port? Could this be an invalid
monitoring request? (say someone trying to do a "stat" but spelling it
wrong?)

Patrick

On Tue, Jan 17, 2012 at 5:00 PM, Alexius Ludeman <lex@lexinator.com> wrote:
>  hi,
>
> I am getting the error from all 5 zookeeperd, "Exception causing close of
> session 0x0 due to java.io.IOException: Len error 1835955314" every
> microsecond(?).
>
> I have a 5 node cluster on ubuntu maverick in aws. I've backported
> zookeeperd and now am running 3.3.4+dfsg1-1.  My apps are
> using zookeeper-3.3.3.jar, but within the next 3-4 weeks I will be
> upgrading to zookeeper-3.3.4.jar.  This problem also existed with
> zookeeperd on 3.3.3, in hopes of upgrading would solve my problem.
>
> I have stopped the whole world, rm -rf /var/lib/zookeeper and restarted
> without any change to this log message.  I have several other environments
> without this error showing, same versions of client and server but only 3
> zookeeperd, and much fewer clients.
>
> Our use of zookeeper is: appName is a key and the value is a server:port.
>  After the app reads in its required appName(s), it puts watches on these
> appNames looking for changes.  So we have a bunch of apps with many
> watches.  The client count is below.
>
> I would be grateful if anyone has some insight on how to debug further.
>
> thanks
> lex
>
> zookeeper-0.p2:/var/log/zookeeper/zookeeper.log:
>
> 2012-01-17 22:23:59,097 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38653
> 2012-01-17 22:23:59,097 - INFO  [Thread-2309197:NIOServerCnxn@1435][] -
> Closed socket connection for client /127.0.0.1:38652 (no session
> established for client)
> 2012-01-17 22:23:59,097 - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
> session 0x0 due to java.io.IOException: Len error 1835955314
> 2012-01-17 22:23:59,097 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection for
> client /127.0.0.1:38653 (no session established for client)
> 2012-01-17 22:23:59,098 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38654
> 2012-01-17 22:23:59,098 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command from /
> 127.0.0.1:38654
> 2012-01-17 22:23:59,098 - INFO
>  [Thread-2309198:NIOServerCnxn$StatCommand@1153][] - Stat command output
> 2012-01-17 22:23:59,098 - INFO  [Thread-2309198:NIOServerCnxn@1435][] -
> Closed socket connection for client /127.0.0.1:38654 (no session
> established for client)
> 2012-01-17 22:23:59,098 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38655
> 2012-01-17 22:23:59,099 - WARN  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639][] - Exception causing close of
> session 0x0 due to java.io.IOException: Len error 1835955314
> 2012-01-17 22:23:59,099 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435][] - Closed socket connection for
> client /127.0.0.1:38655 (no session established for client)
> 2012-01-17 22:23:59,099 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251][] - Accepted socket
> connection from /127.0.0.1:38656
> 2012-01-17 22:23:59,099 - INFO  [NIOServerCxn.Factory:
> 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1237][] - Processing stat command from /
> 127.0.0.1:38656
>
>
> Connected to zookeeper-0.p2
> envi
>
> Environment:
> zookeeper.version=3.3.4--1, built on 12/09/2011 00:03 GMT
> host.name=zookeeper-0.p2.use.color.internal
> java.version=1.6.0_20
> java.vendor=Sun Microsystems Inc.
> java.home=/usr/lib/jvm/java-6-openjdk/jre
> java.class.path=/etc/zookeeper/conf:/usr/share/java/jline.jar:/usr/share/java/log4j-1.2.jar:/usr/share/java/xercesImpl.jar:/usr/share/java/xmlParserAPIs.jar:/usr/share/java/zookeeper.jar
> java.library.path=/usr/lib/jvm/java-6-openjdk/jre/lib/amd64/server:/usr/lib/jvm/java-6-openjdk/jre/lib/amd64:/usr/lib/jvm/java-6-openjdk/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib/jni:/lib:/usr/lib
> java.io.tmpdir=/tmp
> java.compiler=<NA>
> os.name=Linux
> os.arch=amd64
> os.version=2.6.35-24-virtual
> user.name=zookeeper
> user.home=/var/lib/zookeeper
> user.dir=/
>
> Trying 10.118.145.79...Connected to zookeeper-0.p2
> stats
>
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
>  /10.40.50.163:45526[1](queued=0,recved=2819,sent=2820)
>  /10.124.143.97:46512[0](queued=0,recved=1,sent=0)
>  /10.40.37.34:33308[1](queued=0,recved=2795,sent=2796)
>  /10.205.49.18:35339[1](queued=0,recved=2671,sent=2672)
>  /10.205.49.18:42419[1](queued=0,recved=2718,sent=2719)
>  /10.80.82.184:43535[1](queued=0,recved=2819,sent=2820)
>  /10.84.133.213:52533[1](queued=0,recved=2817,sent=2844)
>
> Latency min/avg/max: 0/0/200
> Received: 7330810
> Sent: 7331408
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197
>
> Trying 10.34.37.214...Connected to zookeeper-1.p2
> stat
>
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
>  /10.79.6.245:38198[1](queued=0,recved=2680,sent=2681)
>  /10.98.119.91:37301[1](queued=0,recved=2689,sent=2690)
>  /10.79.35.125:59744[1](queued=0,recved=2681,sent=2682)
>  /10.39.81.28:38047[1](queued=0,recved=2666,sent=2667)
>  /10.124.143.97:48969[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/81
> Received: 4984011
> Sent: 4984267
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197
>
> Trying 10.38.85.114...Connected to zookeeper-2.p2
>
> stat
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
>  /10.84.133.213:51499[1](queued=0,recved=2655,sent=2656)
>  /10.124.143.97:39412[0](queued=0,recved=1,sent=0)
>  /10.39.81.28:56431[1](queued=0,recved=2715,sent=2716)
>  /10.204.183.195:40180[1](queued=0,recved=2664,sent=2665)
>  /10.32.51.156:59390[1](queued=0,recved=2634,sent=2635)
>  /10.79.6.245:42161[1](queued=0,recved=2731,sent=2732)
>  /10.118.14.138:40355[1](queued=0,recved=2643,sent=2644)
>
> Latency min/avg/max: 0/0/33
> Received: 6650278
> Sent: 6651369
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: leader
> Node count: 197
>
> Trying 10.101.42.61...Connected to zookeeper-3.p2
>
> stat
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
>  /10.204.230.212:57523[1](queued=0,recved=2719,sent=2720)
>  /10.85.19.23:52916[1](queued=0,recved=2836,sent=2837)
>  /10.79.35.125:54742[1](queued=0,recved=2734,sent=2735)
>  /10.124.143.97:60909[0](queued=0,recved=1,sent=0)
>  /10.204.230.212:57918[1](queued=0,recved=2674,sent=2675)
>  /10.100.90.101:35977[1](queued=0,recved=2680,sent=2681)
>  /10.98.119.91:42147[1](queued=0,recved=2745,sent=2746)
>
> Latency min/avg/max: 0/0/123
> Received: 8430215
> Sent: 8430724
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197
>
> Trying 10.204.59.55...Connected to zookeeper-4.p2
>
> stat
> Zookeeper version: 3.3.4--1, built on 12/09/2011 00:03 GMT
> Clients:
>  /10.100.90.101:34437[1](queued=0,recved=2728,sent=2729)
>  /10.204.183.195:49961[1](queued=0,recved=2706,sent=2707)
>  /10.124.143.97:34797[0](queued=0,recved=1,sent=0)
>
> Latency min/avg/max: 0/0/19
> Received: 46410
> Sent: 46411
> Outstanding: 0
> Zxid: 0x100014e31
> Mode: follower
> Node count: 197

Mime
View raw message