Return-Path: X-Original-To: apmail-storm-user-archive@minotaur.apache.org Delivered-To: apmail-storm-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id A5CC611A02 for ; Tue, 8 Jul 2014 11:56:45 +0000 (UTC) Received: (qmail 27352 invoked by uid 500); 8 Jul 2014 11:56:45 -0000 Delivered-To: apmail-storm-user-archive@storm.apache.org Received: (qmail 27315 invoked by uid 500); 8 Jul 2014 11:56:45 -0000 Mailing-List: contact user-help@storm.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.incubator.apache.org Delivered-To: mailing list user@storm.incubator.apache.org Received: (qmail 27301 invoked by uid 99); 8 Jul 2014 11:56:45 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jul 2014 11:56:45 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of kristian@livesystems.info designates 209.85.217.177 as permitted sender) Received: from [209.85.217.177] (HELO mail-lb0-f177.google.com) (209.85.217.177) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 08 Jul 2014 11:56:39 +0000 Received: by mail-lb0-f177.google.com with SMTP id u10so3801919lbd.22 for ; Tue, 08 Jul 2014 04:56:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:date:from:to:message-id:subject:mime-version :content-type; bh=8oVa6hG7t0Xnn0k4rzocXyzsu28c19bf4r0s/Vesa5g=; b=TqjVIQ8DDuggD8YaoonIayT9zbfsr+Hdg55udR1/AX4BYgjn4oiikp7I5nHZKQQw/t cztwwnkd63aAOxeiRsMde2Oj/MLatpk2oGV+2azs9teK3YpWi+xKMBVPgX1TEc3NhG6K MfQ8h9tsrRgVtN5xzCDJkUXHgU0WoHe/gjrmeJLY6vjHagzH4y4Lhc0rsXnNABIYYtJo BqjlP7j78uMnUXt3GnHBK0ADXbwSG34dQfUms11aw/I2sHAlo5X7YzwecQFdeBIOE4tC 9pSZjCkr6wicYPT6m5GWK13uMGXHxsAViLl/ee1UawHxtvrN2kQo8z1s0nyfg7QXMLb8 H7BA== X-Gm-Message-State: ALoCoQkGydWXlVk7YU2Ag48h5YQ/7CgJlYbHpArblum8neyfCBvWR21wdvqfBXnMU/DMDfyoC/Yd X-Received: by 10.152.25.132 with SMTP id c4mr1017840lag.88.1404820575148; Tue, 08 Jul 2014 04:56:15 -0700 (PDT) Received: from [172.16.1.2] (93-160-104-114-static.dk.customer.tdc.net. [93.160.104.114]) by mx.google.com with ESMTPSA id ar1sm56202481lbc.3.2014.07.08.04.56.13 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Tue, 08 Jul 2014 04:56:14 -0700 (PDT) Date: Tue, 8 Jul 2014 13:56:12 +0200 From: =?utf-8?Q?Kristian_=C3=98llegaard?= To: user@storm.incubator.apache.org Message-ID: Subject: Storm hostnames and AWS X-Mailer: sparrow 1.6.4 (build 1178) MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="53bbdc5c_71482545_17c" X-Virus-Checked: Checked by ClamAV on apache.org --53bbdc5c_71482545_17c Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi everyone, I=E2=80=99m currently setting up a storm cluster on AWS, but I have some = problems with how storm supervisor connects to the peer nodes. On AWS I=E2= =80=99m running storm in an auto scaling group, meaning that if one of th= e nodes go down, it automatically re-spawns another node with the same im= age. On AWS any node can disappear at any time, therefore it=E2=80=99s im= portant to run a setup similar to this. All nodes have a hostname like ip= -XX-XX-XX-XX representing their IP. However, they are not able to connect= to the others hostnames, e.g. hostname ip-10-0-0-1 will not know about t= he hostname ip-10-0-0-2, although it could access 10.0.0.2. This is resul= ting in errors like this on the supervisors: 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loaded executor tasks cou= nt-bolt:=5B3 3=5D 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D =46inished loading execut= or count-bolt:=5B3 3=5D 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loading executor word-spo= ut:=5B5 5=5D 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loaded executor tasks wor= d-spout:=5B5 5=5D 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Preparing bolt count-bolt= :(3) 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D =46inished loading execut= or word-spout:=5B5 5=5D 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Opening spout word-spout:= (5) 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loading executor =5F=5Fsy= stem:=5B-1 -1=5D 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loaded executor tasks =5F= =5Fsystem:=5B-1 -1=5D 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D =46inished loading execut= or =5F=5Fsystem:=5B-1 -1=5D 2014-07-08 11:43:05 b.s.u.ShellProcess =5BIN=46O=5D Storm multilang seria= lizer: backtype.storm.multilang.JsonSerializer 2014-07-08 11:43:05 b.s.u.ShellProcess =5BIN=46O=5D Storm multilang seria= lizer: backtype.storm.multilang.JsonSerializer 2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Preparing bolt =5F=5Fsyst= em:(-1) 2014-07-08 11:43:05 b.s.m.n.Client =5BIN=46O=5D Reconnect started for Net= ty-Client-ip-10-0-11-36:6703... =5B4=5D 2014-07-08 11:43:05 b.s.m.n.StormClientErrorHandler =5BIN=46O=5D Connecti= on failed Netty-Client-ip-10-0-11-36:6703 java.nio.channels.UnresolvedAddressException: null at sun.nio.ch.Net.checkAddress(Net.java:127) =7E=5Bna:1.7.0=5F55=5D at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:644) =7E=5B= na:1.7.0=5F55=5D at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.connect= (NioClientSocketPipelineSink.java:140) =7E=5Bnetty-3.2.2.=46inal.jar:na=5D= at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSu= nk(NioClientSocketPipelineSink.java:103) =7E=5Bnetty-3.2.2.=46inal.jar:na= =5D at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(= OneToOneEncoder.java:60) =7E=5Bnetty-3.2.2.=46inal.jar:na=5D at org.jboss.netty.channel.Channels.connect(Channels.java:541) =5Bnetty-3= .2.2.=46inal.jar:na=5D at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:2= 18) =5Bnetty-3.2.2.=46inal.jar:na=5D at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java= :227) =5Bnetty-3.2.2.=46inal.jar:na=5D at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java= :188) =5Bnetty-3.2.2.=46inal.jar:na=5D at backtype.storm.messaging.netty.Client.connect(Client.java:147) =5Bstor= m-core-0.9.2-incubating.jar:0.9.2-incubating=5D at backtype.storm.messaging.netty.Client.access=24000(Client.java:42) =5B= storm-core-0.9.2-incubating.jar:0.9.2-incubating=5D at backtype.storm.messaging.netty.Client=241.run(Client.java:104) =5Bstor= m-core-0.9.2-incubating.jar:0.9.2-incubating=5D at java.util.concurrent.Executors=24RunnableAdapter.call(Executors.java:4= 71) =5Bna:1.7.0=5F55=5D at java.util.concurrent.=46utureTask.run(=46utureTask.java:262) =5Bna:1.7= .0=5F55=5D at java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTa= sk.access=24201(ScheduledThreadPoolExecutor.java:178) =5Bna:1.7.0=5F55=5D= at java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utureTa= sk.run(ScheduledThreadPoolExecutor.java:292) =5Bna:1.7.0=5F55=5D at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j= ava:1145) =5Bna:1.7.0=5F55=5D at java.util.concurrent.ThreadPoolExecutor=24Worker.run(ThreadPoolExecuto= r.java:615) =5Bna:1.7.0=5F55=5D at java.lang.Thread.run(Thread.java:744) =5Bna:1.7.0=5F55=5D I=E2=80=99m a bit unsure how to proceed. Is it possible to form to regist= er the IP instead of the hostname somehow=3F Or is there another way to d= o this=3F Kind regards, Kristian --53bbdc5c_71482545_17c Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline
Hi everyone,

I=E2=80=99m currently setting = up a storm cluster on AWS, but I have some problems with how storm superv= isor connects to the peer nodes. On AWS I=E2=80=99m running storm in an a= uto scaling group, meaning that if one of the nodes go down, it automatic= ally re-spawns another node with the same image. On AWS any node can disa= ppear at any time, therefore it=E2=80=99s important to run a setup simila= r to this. All nodes have a hostname like ip-XX-XX-XX-XX representing the= ir IP. However, they are not able to connect to the others hostnames, e.g= . hostname ip-10-0-0-1 will not know about the hostname ip-10-0-0-2, alth= ough it could access 10.0.0.2. This is resulting in errors like this on t= he supervisors:

2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loaded executor tasks= count-bolt:=5B3 3=5D
201= 4-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D =46inished loading executor = count-bolt:=5B3 3=5D
2014= -07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loading executor word-spout:=5B= 5 5=5D
2014-07-08 11:43:0= 5 b.s.d.executor =5BIN=46O=5D Loaded executor tasks word-spout:=5B5 5=5D<= /font>
2014-07-08 11:43:05 b.s.d= .executor =5BIN=46O=5D Preparing bolt count-bolt:(3)
2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D= =46inished loading executor word-spout:=5B5 5=5D
2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D = Opening spout word-spout:(5)
2014-07-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loading executor =5F=5F= system:=5B-1 -1=5D
2014-0= 7-08 11:43:05 b.s.d.executor =5BIN=46O=5D Loaded executor tasks =5F=5Fsys= tem:=5B-1 -1=5D
2014-07-0= 8 11:43:05 b.s.d.executor =5BIN=46O=5D =46inished loading executor =5F=5F= system:=5B-1 -1=5D
2014-0= 7-08 11:43:05 b.s.u.ShellProcess =5BIN=46O=5D Storm multilang serializer:= backtype.storm.multilang.JsonSerializer
2014-07-08 11:43:05 b.s.u.ShellProcess =5BIN=46O=5D Storm = multilang serializer: backtype.storm.multilang.JsonSerializer
2014-07-08 11:43:05 b.s.d.executor =5B= IN=46O=5D Preparing bolt =5F=5Fsystem:(-1)
2014-07-08 11:43:05 b.s.m.n.Client =5BIN=46O=5D Reconne= ct started for Netty-Client-ip-10-0-11-36:6703... =5B4=5D
2014-07-08 11:43:05 b.s.m.n.StormClientE= rrorHandler =5BIN=46O=5D Connection failed Netty-Client-ip-10-0-11-36:670= 3
java.nio.channels.Unres= olvedAddressException: null
at sun.nio.ch.Net.checkAdd= ress(Net.java:127) =7E=5Bna:1.7.0=5F55=5D
at sun.nio.c= h.SocketChannelImpl.connect(SocketChannelImpl.java:644) =7E=5Bna:1.7.0=5F= 55=5D
at org.jboss.netty.channel.socket.nio.NioClien= tSocketPipelineSink.connect(NioClientSocketPipelineSink.java:140) =7E=5Bn= etty-3.2.2.=46inal.jar:na=5D
at org.jboss.netty.chan= nel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipel= ineSink.java:103) =7E=5Bnetty-3.2.2.=46inal.jar:na=5D
at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstrea= m(OneToOneEncoder.java:60) =7E=5Bnetty-3.2.2.=46inal.jar:na=5D
at org.jboss.netty.channel.Channels.connect(Channels.java:541) =5B= netty-3.2.2.=46inal.jar:na=5D
at org.jboss.netty.cha= nnel.AbstractChannel.connect(AbstractChannel.java:218) =5Bnetty-3.2.2.=46= inal.jar:na=5D
at org.jboss.netty.bootstrap.ClientBo= otstrap.connect(ClientBootstrap.java:227) =5Bnetty-3.2.2.=46inal.jar:na=5D=
at org.jboss.netty.bootstrap.ClientBootstrap.connec= t(ClientBootstrap.java:188) =5Bnetty-3.2.2.=46inal.jar:na=5D
=
at backtype.storm.messaging.netty.Client.connect(Client.java:147= ) =5Bstorm-core-0.9.2-incubating.jar:0.9.2-incubating=5D
at backtype.storm.messaging.netty.Client.access=24000(Client.java:42= ) =5Bstorm-core-0.9.2-incubating.jar:0.9.2-incubating=5D
at backtype.storm.messaging.netty.Client=241.run(Client.java:104) =5B= storm-core-0.9.2-incubating.jar:0.9.2-incubating=5D
= at java.util.concurrent.Executors=24RunnableAdapter.call(Executors.java:4= 71) =5Bna:1.7.0=5F55=5D
<= span style=3D=22white-space: pre;=22> at java.util.concurrent.=46u= tureTask.run(=46utureTask.java:262) =5Bna:1.7.0=5F55=5D
= at java.util.concurrent.ScheduledThreadPoolExecutor=24Scheduled=46utu= reTask.access=24201(ScheduledThreadPoolExecutor.java:178) =5Bna:1.7.0=5F5= 5=5D
at java.util.concurrent.ScheduledThreadPoolExec= utor=24Scheduled=46utureTask.run(ScheduledThreadPoolExecutor.java:292) =5B= na:1.7.0=5F55=5D
at java.util.concurrent.ThreadPoolE= xecutor.runWorker(ThreadPoolExecutor.java:1145) =5Bna:1.7.0=5F55=5D
at java.util.concurrent.ThreadPoolExecutor=24Worker.run(T= hreadPoolExecutor.java:615) =5Bna:1.7.0=5F55=5D
at j= ava.lang.Thread.run(Thread.java:744) =5Bna:1.7.0=5F55=5D

I=E2=80=99m a bit unsure how to proceed. Is it poss= ible to form to register the IP instead of the hostname somehow=3F Or is = there another way to do this=3F

Kind regards,

Kristian
--53bbdc5c_71482545_17c--