Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 397C21187E for ; Thu, 17 Jul 2014 23:32:00 +0000 (UTC) Received: (qmail 15424 invoked by uid 500); 17 Jul 2014 23:31:59 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 15380 invoked by uid 500); 17 Jul 2014 23:31:59 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 15370 invoked by uid 99); 17 Jul 2014 23:31:59 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2014 23:31:59 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of arsneilson@gmail.com designates 209.85.217.179 as permitted sender) Received: from [209.85.217.179] (HELO mail-lb0-f179.google.com) (209.85.217.179) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Jul 2014 23:31:57 +0000 Received: by mail-lb0-f179.google.com with SMTP id v6so2255559lbi.24 for ; Thu, 17 Jul 2014 16:31:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=e4lrugyu2genIz6dFyaAdE90JjfW5xCBvEBpBkTbNps=; b=JJDTXPpFnxHEDrmYKfGQriAdunmUdxKUQG7VpgqrNyzq6l80gy2FrF24giSWFqApiX /leYB6M4HsKD6cyh0dNnsgEhZ+hTT+2CLvzcR4Dz4tg+p7Y1DAjRF81dGf5RCjD/NocT q22cPgfRukU2HxEO0iZqaMdBg3lopPcjF//wA0HwVivOmXhUvvIzE1SWfRibfy/wPznF 13SuueUz4pJQri7AjXj0W0w0KR0eTCKC96bGbcffgt/otheCq0giGdt8Zj7aYhGB4hhn OKScoBK6aDX/S7LvgT/cTnhrMvH8ZkpUADCdH1MO5XGCIZbS1fF1nXOiFbHpkSbewc4v hOaQ== MIME-Version: 1.0 X-Received: by 10.112.151.164 with SMTP id ur4mr414700lbb.13.1405639892559; Thu, 17 Jul 2014 16:31:32 -0700 (PDT) Received: by 10.152.127.66 with HTTP; Thu, 17 Jul 2014 16:31:32 -0700 (PDT) In-Reply-To: <53C65495.5050302@gmail.com> References: <53C65495.5050302@gmail.com> Date: Thu, 17 Jul 2014 16:31:32 -0700 Message-ID: Subject: Re: Storm Supervisors are failing From: Andrew Neilson To: user@storm.incubator.apache.org Content-Type: multipart/alternative; boundary=047d7beb9ff47a6c2c04fe6c0a1f X-Virus-Checked: Checked by ClamAV on apache.org --047d7beb9ff47a6c2c04fe6c0a1f Content-Type: text/plain; charset=UTF-8 https://issues.apache.org/jira/browse/STORM-307 the workaround is to stop your supervisor daemon, delete the contents of your Storm data directory (under the location specified by "storm.local.dir" in your storm config), and restart the supervisor. On Wed, Jul 16, 2014 at 3:31 AM, Suraj Nayak M wrote: > Hi, > > Help me resolve the error : > > Storm supervisors are failing with below error : > > 2014-07-16 15:42:35 c.n.c.ConnectionState [ERROR] Authentication failed > 2014-07-16 15:42:35 o.a.z.ClientCnxn [INFO] Session establishment complete > on server slave1.hdp.somedomain.com/10.255.1.209:2181, sessionid = > 0x2473de21afd0009, negotiated timeout = 20000 > 2014-07-16 15:42:35 c.n.c.f.s.ConnectionStateManager [INFO] State change: > CONNECTED > 2014-07-16 15:42:35 c.n.c.f.s.ConnectionStateManager [WARN] There are no > ConnectionStateListeners registered. > 2014-07-16 15:42:36 b.s.d.supervisor [INFO] Starting supervisor with id > 811b3a1c-d440-4c64-9fbc-239b8ef4c471 at host slave1.hdp.somedomain.com > 2014-07-16 15:42:36 b.s.event [ERROR] Error when processing event > java.lang.RuntimeException: java.io.EOFException > at backtype.storm.utils.Utils.deserialize(Utils.java:86) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at backtype.storm.utils.LocalState.snapshot(LocalState.java:45) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at backtype.storm.utils.LocalState.get(LocalState.java:56) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:207) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.4.0.jar:na] > at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] > at clojure.core$apply.invoke(core.clj:603) ~[clojure-1.4.0.jar:na] > at clojure.core$partial$fn__4070.doInvoke(core.clj:2343) > ~[clojure-1.4.0.jar:na] > at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.4.0.jar:na] > at backtype.storm.event$event_manager$fn__4187.invoke(event.clj:39) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55] > Caused by: java.io.EOFException: null > at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > ~[na:1.7.0_55] > at java.io.ObjectInputStream$BlockDataInputStream. > readShort(ObjectInputStream.java:2794) ~[na:1.7.0_55] > at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > ~[na:1.7.0_55] > at java.io.ObjectInputStream.(ObjectInputStream.java:299) > ~[na:1.7.0_55] > at backtype.storm.utils.Utils.deserialize(Utils.java:81) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > ... 11 common frames omitted > 2014-07-16 15:42:36 b.s.event [ERROR] Error when processing event > java.lang.RuntimeException: java.io.EOFException > at backtype.storm.utils.Utils.deserialize(Utils.java:86) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at backtype.storm.utils.LocalState.snapshot(LocalState.java:45) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at backtype.storm.utils.LocalState.get(LocalState.java:56) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at backtype.storm.daemon.supervisor$mk_synchronize_ > supervisor$this__6559.invoke(supervisor.clj:307) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at backtype.storm.event$event_manager$fn__4187.invoke(event.clj:39) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55] > Caused by: java.io.EOFException: null > at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2325) > ~[na:1.7.0_55] > at java.io.ObjectInputStream$BlockDataInputStream. > readShort(ObjectInputStream.java:2794) ~[na:1.7.0_55] > at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:801) > ~[na:1.7.0_55] > at java.io.ObjectInputStream.(ObjectInputStream.java:299) > ~[na:1.7.0_55] > at backtype.storm.utils.Utils.deserialize(Utils.java:81) > ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] > ... 6 common frames omitted > 2014-07-16 15:42:36 b.s.util [INFO] Halting process: ("Error when > processing an event") > 2014-07-16 15:42:36 b.s.util [INFO] Halting process: ("Error when > processing an event") > > --047d7beb9ff47a6c2c04fe6c0a1f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
https://issues.apache.org/jira/browse/STORM-307

th= e workaround is to stop your supervisor daemon, delete the contents of your= Storm data directory (under the location specified by "storm.local.di= r" in your storm config), and restart the supervisor.


On Wed,= Jul 16, 2014 at 3:31 AM, Suraj Nayak M <snayakm@gmail.com> = wrote:
Hi,

Help me resolve the error :

Storm supervisors are failing with below error :

2014-07-16 15:42:35 c.n.c.ConnectionState [ERROR] Authentication failed
2014-07-16 15:42:35 o.a.z.ClientCnxn [INFO] Session establishment complete = on server slave1.hdp.somedomain.com/10.255.1.209:2181, ses= sionid =3D 0x2473de21afd0009, negotiated timeout =3D 20000
2014-07-16 15:42:35 c.n.c.f.s.ConnectionStateManager [INFO] State ch= ange: CONNECTED
2014-07-16 15:42:35 c.n.c.f.s.ConnectionStateManager [WARN] There ar= e no ConnectionStateListeners registered.
2014-07-16 15:42:36 b.s.d.supervisor [INFO] Starting supervisor with id 811= b3a1c-d440-4c64-9fbc-239b8ef4c471 at host slave1.hdp.somedomain.com
2014-07-16 15:42:36 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.EOFException
=C2=A0 =C2=A0 at backtype.storm.utils.Utils.deserialize(Utils.java:8= 6) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563]
=C2=A0 =C2=A0 at backtype.storm.utils.LocalState.snapshot(Loc= alState.java:45) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-56= 3]
=C2=A0 =C2=A0 at backtype.storm.utils.LocalState.get(LocalState.<= /u>java:56) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] =C2=A0 =C2=A0 at backtype.storm.daemon.supervisor$sync_processes.= invoke(supervisor.clj:207) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.= 9.1.2.1.3.0-563]
=C2=A0 =C2=A0 at clojure.lang.AFn.applyToHelper(AFn.java:161) [cloju= re-1.4.0.jar:na]
=C2=A0 =C2=A0 at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4= .0.jar:na]
=C2=A0 =C2=A0 at clojure.core$apply.invoke(core.clj:603) ~[clojure-1= .4.0.jar:na]
=C2=A0 =C2=A0 at clojure.core$partial$fn__4070.doInvoke(core.clj:234= 3) ~[clojure-1.4.0.jar:na]
=C2=A0 =C2=A0 at clojure.lang.RestFn.invoke(RestFn.java:397) ~[cloju= re-1.4.0.jar:na]
=C2=A0 =C2=A0 at backtype.storm.event$event_manager$fn__4187.invoke(= event.clj:39) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.= 0-563]
=C2=A0 =C2=A0 at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.ja= r:na]
=C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]=
Caused by: java.io.EOFException: null
=C2=A0 =C2=A0 at java.io.ObjectInputStream$PeekInputStream.readFully= (ObjectInputStream.java:2325) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at java.io.ObjectInputStream$BlockDataInputStream.<= /u>readShort(ObjectInputStream.java:2794) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at java.io.ObjectInputStream.readStreamHeader(O= bjectInputStream.java:801) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at java.io.ObjectInputStream.<init>(ObjectInputS= tream.java:299) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at backtype.storm.utils.Utils.deserialize(Utils.java:8= 1) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563]
=C2=A0 =C2=A0 ... 11 common frames omitted
2014-07-16 15:42:36 b.s.event [ERROR] Error when processing event
java.lang.RuntimeException: java.io.EOFException
=C2=A0 =C2=A0 at backtype.storm.utils.Utils.deserialize(Utils.java:8= 6) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563]
=C2=A0 =C2=A0 at backtype.storm.utils.LocalState.snapshot(Loc= alState.java:45) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-56= 3]
=C2=A0 =C2=A0 at backtype.storm.utils.LocalState.get(LocalState.<= /u>java:56) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563] =C2=A0 =C2=A0 at backtype.storm.daemon.supervisor$mk_synchronize_= supervisor$this__6559.invoke(supervisor.clj:307) ~[storm-core-0.= 9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563]
=C2=A0 =C2=A0 at backtype.storm.event$event_manager$fn__4187.invoke(= event.clj:39) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.= 0-563]
=C2=A0 =C2=A0 at clojure.lang.AFn.run(AFn.java:24) [clojure-1.4.0.ja= r:na]
=C2=A0 =C2=A0 at java.lang.Thread.run(Thread.java:745) [na:1.7.0_55]=
Caused by: java.io.EOFException: null
=C2=A0 =C2=A0 at java.io.ObjectInputStream$PeekInputStream.readFully= (ObjectInputStream.java:2325) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at java.io.ObjectInputStream$BlockDataInputStream.<= /u>readShort(ObjectInputStream.java:2794) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at java.io.ObjectInputStream.readStreamHeader(O= bjectInputStream.java:801) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at java.io.ObjectInputStream.<init>(ObjectInputS= tream.java:299) ~[na:1.7.0_55]
=C2=A0 =C2=A0 at backtype.storm.utils.Utils.deserialize(Utils.java:8= 1) ~[storm-core-0.9.1.2.1.3.0-563.jar:0.9.1.2.1.3.0-563]
=C2=A0 =C2=A0 ... 6 common frames omitted
2014-07-16 15:42:36 b.s.util [INFO] Halting process: ("Error when proc= essing an event")
2014-07-16 15:42:36 b.s.util [INFO] Halting process: ("Error when proc= essing an event")


--047d7beb9ff47a6c2c04fe6c0a1f--