cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Himanshi Sharma <himanshi.sha...@tcs.com>
Subject Re: Cassandra nodes on EC2 in two different regions not communicating
Date Thu, 24 Feb 2011 07:04:38 GMT
<font face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size="2"><DIV>giving
private ip to rpc address gives the same exception</DIV>
<DIV>and the keeping it blank and providing public to listen also fails. I tried keeping
both blank and did telnet on 7000 so i get following o/p</DIV>
<DIV>&nbsp;</DIV>
<DIV>[root@ip-10-166-223-150 bin]# telnet 122.248.193.37 7000<BR>Trying 122.248.193.37...<BR>Connected
to 122.248.193.37.<BR>Escape character is '^]'.<BR></DIV>
<DIV>&nbsp;</DIV>
<DIV>Similarly from another achine</DIV>
<DIV>&nbsp;</DIV>
<DIV>[root@ip-10-136-75-201 bin]# telnet 184.72.22.87 7000<BR>Trying 184.72.22.87...<BR>Connected
to 184.72.22.87.<BR>Escape character is '^]'.<BR></DIV>
<DIV>&nbsp;</DIV>
<DIV><BR><BR><FONT color=#990099>-----Dave Viner <DAVEVINER@GMAIL.COM>wrote:
-----</FONT> </DIV>
<DIV>
<BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT:
black 2px solid; MARGIN-RIGHT: 0px">To: user@cassandra.apache.org<BR>From: Dave Viner
&lt;daveviner@gmail.com&gt;<BR>Date: 02/24/2011 11:59AM<BR>cc: Himanshi
Sharma &lt;himanshi.sharma@tcs.com&gt;<BR>Subject: Re: Cassandra nodes on EC2
in two different regions not communicating<BR><BR>Try using the private ipv4 address
in the rpc_address field, and the public ipv4 (NOT the elastic ip) in the listen_address.

<DIV><BR></DIV>
<DIV>If that fails, go back to rpc_address empty, and start up cassandra. </DIV>
<DIV><BR></DIV>
<DIV>Then from the other node, please telnet to port 7000 on the first node. &nbsp;And
show the output of that session in your reply. </DIV>
<DIV><BR></DIV>
<DIV>I haven't actually constructed a cross-region cluster nor have I used v0.7, but
this really sounds like it should be easy. <BR><BR>
<DIV class=gmail_quote>On Wed, Feb 23, 2011 at 10:22 PM, Himanshi Sharma <SPAN dir=ltr>&lt;
<A href="mailto:himanshi.sharma@tcs.com">himanshi.sharma@tcs.com </A>&gt;
</SPAN>wrote: <BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT:
#ccc 1px solid"><FONT face="Default Sans Serif,Verdana,Arial,Helvetica,sans-serif" size=2>
<DIV>Hi Dave, </DIV>
<DIV>&nbsp; </DIV>
<DIV>I tried with the public ips. If i mention the public ip in rpc address field, Cassandra
gives the same exception&nbsp;but if leave it blank then Cassandra runs but again in the
nodetool command with ring option it does'nt show the node in another region. </DIV>
<DIV>&nbsp; </DIV>
<DIV>Thanks, </DIV>
<DIV>Himanshi <BR><BR><BR><FONT color=#990099>-----Dave Viner
wrote: ----- </FONT></DIV>
<DIV>
<BLOCKQUOTE style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT:
black 2px solid; MARGIN-RIGHT: 0px">
<DIV class=im>To: <A href="mailto:user@cassandra.apache.org" target=_blank>user@cassandra.apache.org
</A><BR>From: Dave Viner &lt; <A href="mailto:daveviner@gmail.com" target=_blank>daveviner@gmail.com
</A>&gt; <BR></DIV>Date: 02/24/2011 10:43AM 
<DIV>
<DIV></DIV>
<DIV class=h5><BR>Subject: Re: Cassandra nodes on EC2 in two different regions
not communicating <BR><BR>That looks like it's not an issue of communicating between
nodes. &nbsp;It appears that the node can not bind to the address on the localhost that
you're asking for. 
<DIV><BR></DIV>
<DIV>" <SPAN><FONT style="FONT-SIZE: 13px; FONT-FAMILY: arial, sans-serif"
face=sans-serif size=2>java.net.BindException: Cannot assign requested address </FONT><FONT
face="arial, sans-serif">&nbsp;" </FONT></SPAN></DIV>
<DIV><FONT face="arial, sans-serif" color=#222222><SPAN style="BORDER-COLLAPSE:
collapse"><BR></SPAN></FONT></DIV>
<DIV><FONT face="arial, sans-serif"></FONT><FONT face="arial, sans-serif"
color=#222222><SPAN style="BORDER-COLLAPSE: collapse">I think the issue is that the
Elastic IP address is not actually an IP address that's on the localhost. &nbsp;So the
daemon can not bind to that IP. &nbsp;Instead of using the EIP, use the local IP address
for the rpc_address (i think that's what you need since that is what Thrift will bind to).
&nbsp;Then for the listen_address should be the ip address that is routable from the other
node. &nbsp;I would first try with the actual public IP address (not the Elastic IP).
&nbsp;Once you get that to work, then shutdown the cluster, change the listen_address
to the EIP, boot up and try again. </SPAN></FONT></DIV>
<DIV><FONT face="arial, sans-serif" color=#222222><SPAN style="BORDER-COLLAPSE:
collapse"><BR></SPAN></FONT></DIV>
<DIV><FONT face="arial, sans-serif" color=#222222><SPAN style="BORDER-COLLAPSE:
collapse">Dave Viner </SPAN></FONT></DIV>
<DIV><FONT face="arial, sans-serif" color=#222222><SPAN style="BORDER-COLLAPSE:
collapse"><BR></SPAN></FONT><BR>
<DIV class=gmail_quote>On Wed, Feb 23, 2011 at 8:54 PM, Himanshi Sharma <SPAN dir=ltr>&lt;
<A href="mailto:himanshi.sharma@tcs.com" target=_blank>himanshi.sharma@tcs.com </A>&gt;
</SPAN>wrote: <BR>
<BLOCKQUOTE class=gmail_quote style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT:
#ccc 1px solid"><BR><FONT face=sans-serif size=2>Hey Dave, </FONT><BR><BR><FONT
face=sans-serif size=2>Sorry i forgot to mention the Non-seed configuration. </FONT><BR><BR><FONT
face=sans-serif size=2>for first node in us-west its as below....i.e its own elastic ip
</FONT><BR><BR><FONT face=sans-serif size=2>listen_address: 50.18.60.117
</FONT><BR><FONT face=sans-serif size=2>rpc_address: 50.18.60.117 </FONT><BR><BR><FONT
face=sans-serif size=2>and for second node in ap-southeast-1 its as below....i.e again
its own elastic ip </FONT><BR><BR><FONT face=sans-serif size=2>listen_address:
175.41.143.192 </FONT><BR><FONT face=sans-serif size=2>rpc_address: 175.41.143.192
</FONT><BR><BR><FONT face=sans-serif size=2>Thanks, </FONT><BR><FONT
face=sans-serif size=2>Himanshi </FONT><BR><BR><FONT face=sans-serif
size=2><BR></FONT><BR><BR><BR>
<TABLE width="100%">
<TBODY>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>From: </FONT></TD>
<TD>
<DIV><FONT face=sans-serif size=1>Dave Viner &lt; <A href="mailto:daveviner@gmail.com"
target=_blank>daveviner@gmail.com </A>&gt; </FONT></DIV></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>To: </FONT></TD>
<TD><FONT face=sans-serif size=1><A href="mailto:user@cassandra.apache.org"
target=_blank>user@cassandra.apache.org </A></FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>Date: </FONT></TD>
<TD><FONT face=sans-serif size=1>02/23/2011 11:01 PM </FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>Subject: </FONT></TD>
<TD><FONT face=sans-serif size=1>Re: Cassandra nodes on EC2 in two different regions
not communicating </FONT></TD></TR></TBODY></TABLE><BR>
<HR noShade>
<BR><BR><BR><FONT size=3>internal EC2 ips (10.xxx.xxx.xxx) work across
availability zones (e.g., from us-east-1a to us-east-1b) but do not work across regions (e.g.,
us-east to us-west). &nbsp;To do regions, you must use the public ip address assigned
by amazon. </FONT><BR><BR><FONT size=3>Himanshi, when you log into
1 node, and telnet to port 7000 on the other node, which IP address did you use - the 10.x
address or the public ip address? </FONT><BR><FONT size=3>And what is the
seed/non-seed configuration in both cassandra.yaml files? </FONT><BR><BR><FONT
size=3>Dave Viner </FONT><BR><BR><BR><FONT size=3>On Wed,
Feb 23, 2011 at 8:12 AM, Frank LoVecchio &lt; </FONT><A href="mailto:frank@isidorey.com"
target=_blank><FONT color=blue size=3><U>frank@isidorey.com </U></FONT></A><FONT
size=3>&gt; wrote: </FONT><BR><FONT size=3>The internal Amazon IP
address is what you will want to use so you don't have to go through DNS anyways; not sure
if this works from US-East to US-West, but it does make things quicker in between zones, e.g.
us-east-1a to us-east-1b. </FONT><BR><FONT size=3><BR></FONT><BR><FONT
size=3>On Wed, Feb 23, 2011 at 9:09 AM, Dave Viner &lt; </FONT><A href="mailto:daveviner@gmail.com"
target=_blank><FONT color=blue size=3><U>daveviner@gmail.com </U></FONT></A><FONT
size=3>&gt; wrote: </FONT><BR><FONT size=3>Try using the IP address,
not the dns name in the cassandra.yaml. </FONT><BR><BR><FONT size=3>If
you can telnet from one to the other on port 7000, and both nodes have the other node in their
config, it should work. </FONT><BR><BR><FONT color=#8f8f8f size=3>Dave
Viner </FONT><BR><FONT size=3><BR></FONT><BR><FONT
size=3>On Wed, Feb 23, 2011 at 1:43 AM, Himanshi Sharma &lt; </FONT><A href="mailto:himanshi.sharma@tcs.com"
target=_blank><FONT color=blue size=3><U>himanshi.sharma@tcs.com </U></FONT></A><FONT
size=3>&gt; wrote: </FONT><BR><FONT face=sans-serif size=2><BR>Ya
they do. Have specified Public DNS in seed field of each node in Cassandra.yaml...nt able
to figure out what the problem is ??? </FONT><FONT size=3></FONT><FONT
face=sans-serif size=2><BR></FONT><FONT size=3><BR><BR><BR></FONT>
<TABLE width="100%">
<TBODY>
<TR vAlign=top>
<TD width="11%"><FONT face=sans-serif color=#5f5f5f size=1>From: </FONT><FONT
size=3></FONT></TD>
<TD width="88%"><FONT face=sans-serif size=1>Sasha Dolgy &lt; </FONT><A
href="mailto:sdolgy@gmail.com" target=_blank><FONT face=sans-serif color=blue size=1><U>sdolgy@gmail.com
</U></FONT></A><FONT face=sans-serif size=1>&gt; </FONT><FONT
size=3></FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>To: </FONT><FONT size=3></FONT></TD>
<TD><A href="mailto:user@cassandra.apache.org" target=_blank><FONT face=sans-serif
color=blue size=1><U>user@cassandra.apache.org </U></FONT></A><FONT
size=3></FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>Date: </FONT><FONT size=3></FONT></TD>
<TD><FONT face=sans-serif size=1>02/23/2011 02:56 PM </FONT><FONT size=3></FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>Subject: </FONT><FONT
size=3></FONT></TD>
<TD><FONT face=sans-serif size=1>Re: Cassandra nodes on EC2 in two different regions
not communicating </FONT></TD></TR></TBODY></TABLE><BR><FONT
size=3><BR></FONT>
<HR noShade>
<FONT size=3><BR><BR><BR>did you define the other host in the cassandra.yaml
?&nbsp; on both servers .... they need to know about each other <BR><BR>On
Wed, Feb 23, 2011 at 10:16 AM, Himanshi Sharma &lt; </FONT><A href="mailto:himanshi.sharma@tcs.com"
target=_blank><FONT color=blue size=3><U>himanshi.sharma@tcs.com </U></FONT></A><FONT
size=3>&gt; wrote: </FONT><FONT face=sans-serif size=2><BR><BR>Thanks
Dave but I am able to telnet to other instances on port 7000 <BR>and when i run &nbsp;./nodetool
--host </FONT><A href="http://ec2-50-18-60-117.us-west-1.compute.amazonaws.com/"
target=_blank><FONT face=sans-serif color=blue size=2><U>ec2-50-18-60-117.us-west-1.compute.amazonaws.com
</U></FONT></A><FONT face=sans-serif size=2>&nbsp;ring... I can
see only one node. </FONT><FONT size=3></FONT><FONT face=sans-serif size=2><BR><BR>Do
we need to configure anything else in Cassandra.yaml or Cassandra-env.sh ??? </FONT><FONT
size=3><BR><BR><BR><BR><BR><BR></FONT>
<TABLE width="100%">
<TBODY>
<TR vAlign=top>
<TD width="11%"><FONT face=sans-serif color=#5f5f5f size=1>From: </FONT><FONT
size=3></FONT></TD>
<TD width="88%"><FONT face=sans-serif size=1>Dave Viner &lt; </FONT><A
href="mailto:daveviner@gmail.com" target=_blank><FONT face=sans-serif color=blue size=1><U>daveviner@gmail.com
</U></FONT></A><FONT face=sans-serif size=1>&gt; </FONT><FONT
size=3></FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>To: </FONT><FONT size=3></FONT></TD>
<TD><A href="mailto:user@cassandra.apache.org" target=_blank><FONT face=sans-serif
color=blue size=1><U>user@cassandra.apache.org </U></FONT></A><FONT
size=3></FONT></TD></TR>
<TR>
<TD vAlign=top><FONT face=sans-serif color=#5f5f5f size=1>Cc: </FONT><FONT
size=3></FONT></TD>
<TD><FONT face=sans-serif size=1>Himanshi Sharma &lt; </FONT><A href="mailto:himanshi.sharma@tcs.com"
target=_blank><FONT face=sans-serif color=blue size=1><U>himanshi.sharma@tcs.com
</U></FONT></A><FONT face=sans-serif size=1>&gt; </FONT><FONT
size=3></FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>Date: </FONT><FONT size=3></FONT></TD>
<TD><FONT face=sans-serif size=1>02/23/2011 11:36 AM </FONT><FONT size=3></FONT></TD></TR>
<TR vAlign=top>
<TD><FONT face=sans-serif color=#5f5f5f size=1>Subject: </FONT><FONT
size=3></FONT></TD>
<TD><FONT face=sans-serif size=1>Re: Cassandra nodes on EC2 in two different regions
not communicating </FONT></TD></TR></TBODY></TABLE><BR><FONT
size=3><BR><BR></FONT>
<HR noShade>
<FONT size=3><BR><BR><BR>If you login to one of the nodes, can you
telnet to port 7000 on the other node? <BR><BR>If not, then almost certainly it's
a firewall/Security Group issue. <BR><BR>You can find out the security groups
for any node by logging in, and then running: <BR><BR>%&nbsp;curl " </FONT><A
href="http://169.254.169.254/latest/meta-data/security-groups" target=_blank><FONT color=blue
size=3><U>http://169.254.169.254/latest/meta-data/security-groups </U></FONT></A><FONT
size=3>" <BR><BR>Assuming that both nodes are in the same security group, ensure
that the SG is configured to allow other members of the SG to communicate on port 7000 to
each other. <BR><BR>HTH, <BR>Dave Viner <BR><BR><BR>On
Tue, Feb 22, 2011 at 8:59 PM, Himanshi Sharma &lt; </FONT><A href="mailto:himanshi.sharma@tcs.com"
target=_blank><FONT color=blue size=3><U>himanshi.sharma@tcs.com </U></FONT></A><FONT
size=3>&gt; wrote: </FONT><FONT face=sans-serif size=2><BR><BR>Hi,
</FONT><FONT size=3></FONT><FONT face=sans-serif size=2><BR><BR>I
am new to Cassandra. I m running Cassandra on EC2. I configured Cassandra cluster on two instances
in different regions. </FONT><FONT size=3></FONT><FONT face=sans-serif
size=2><BR>But when I am trying the nodetool command with ring option, I am getting
only single node. <BR><BR>How to make these two nodes communicate with each other.
I have already opened required ports. i.e 7000, 8080, 9160 in respective <BR>security
groups. Plz help me with this. </FONT><FONT size=3></FONT><FONT face=sans-serif
size=2><BR><BR>Regards, </FONT><FONT size=3></FONT><FONT
face=sans-serif size=2><BR>Himanshi Sharma </FONT><FONT size=3><BR></FONT><TT><FONT
size=3><BR><BR>=====-----=====-----===== <BR>Notice: The information
contained in this e-mail <BR>message and/or attachments to it may contain <BR>confidential
or privileged information. If you are <BR><BR>not the intended recipient, any
dissemination, use, <BR>review, distribution, printing or copying of the <BR>information
contained in this e-mail message <BR>and/or attachments to it are strictly prohibited.
If <BR>you have received this communication in error, <BR><BR>please notify
us by reply e-mail or telephone and <BR>immediately and permanently delete the message
<BR>and any attachments. Thank you </FONT></TT><FONT size=3><BR><BR><BR><BR></FONT><TT><FONT
size=3><BR>=====-----=====-----===== <BR><BR><BR>Notice: The information
contained in this e-mail <BR>message and/or attachments to it may contain <BR>confidential
or privileged information. If you are <BR>not the intended recipient, any dissemination,
use, <BR>review, distribution, printing or copying of the <BR><BR><BR>information
contained in this e-mail message <BR>and/or attachments to it are strictly prohibited.
If <BR>you have received this communication in error, <BR>please notify us by
reply e-mail or telephone and <BR>immediately and permanently delete the message <BR><BR><BR>and
any attachments. Thank you <BR></FONT></TT><FONT size=3><BR><BR><BR><BR><BR>--
<BR>Sasha Dolgy </FONT><FONT color=blue size=3><U><BR></U></FONT><A
href="mailto:sasha.dolgy@gmail.com" target=_blank><FONT color=blue size=3><U>sasha.dolgy@gmail.com
</U></FONT></A><FONT size=3><BR></FONT><BR><TT><FONT
size=3>=====-----=====-----===== <BR>Notice: The information contained in this e-mail
<BR>message and/or attachments to it may contain <BR>confidential or privileged
information. If you are <BR><BR><BR><BR>not the intended recipient,
any dissemination, use, <BR>review, distribution, printing or copying of the <BR>information
contained in this e-mail message <BR>and/or attachments to it are strictly prohibited.
If <BR>you have received this communication in error, <BR><BR><BR><BR>please
notify us by reply e-mail or telephone and <BR>immediately and permanently delete the
message <BR>and any attachments. Thank you <BR><BR></FONT></TT><BR><BR><FONT
size=3><BR><BR></FONT><BR><FONT color=#8f8f8f size=3>-- </FONT><BR><FONT
color=#8f8f8f size=3>Frank LoVecchio </FONT><BR><FONT color=#8f8f8f size=3>Senior
Software Engineer | Isidorey, LLC </FONT><BR><FONT color=#8f8f8f size=3>Google
Voice +1.720.295.9179 </FONT><BR><A href="http://isidorey.com/" target=_blank><FONT
color=blue size=3><U>isidorey.com </U></FONT></A><FONT color=#8f8f8f
size=3>| </FONT><A href="http://facebook.com/franklovecchio" target=_blank><FONT
color=blue size=3><U>facebook.com/franklovecchio </U></FONT></A><FONT
color=#8f8f8f size=3>| </FONT><A href="http://franklovecchio.com/" target=_blank><FONT
color=blue size=3><U>franklovecchio.com </U></FONT></A><BR><BR><BR><BR><PRE
style="WHITE-SPACE: normal">=====-----=====-----=====
<BR>Notice: The information contained in this e-mail
<BR>message and/or attachments to it may contain 
<BR>confidential or privileged information. If you are 
<BR>
not the intended recipient, any dissemination, use, 
<BR>review, distribution, printing or copying of the 
<BR>information contained in this e-mail message 
<BR>and/or attachments to it are strictly prohibited. If 
<BR>you have received this communication in error, 
<BR>
please notify us by reply e-mail or telephone and 
<BR>immediately and permanently delete the message 
<BR>and any attachments. Thank you
<BR><BR><BR></PRE></BLOCKQUOTE></DIV><BR></DIV></DIV></DIV></BLOCKQUOTE></DIV>
<DIV></DIV></FONT>
<DIV>
<DIV></DIV>
<DIV class=h5><PRE style="WHITE-SPACE: normal">=====-----=====-----=====
<BR>Notice: The information contained in this e-mail
<BR>message and/or attachments to it may contain 
<BR>confidential or privileged information. If you are 
<BR>
not the intended recipient, any dissemination, use, 
<BR>review, distribution, printing or copying of the 
<BR>information contained in this e-mail message 
<BR>and/or attachments to it are strictly prohibited. If 
<BR>you have received this communication in error, 
<BR>
please notify us by reply e-mail or telephone and 
<BR>immediately and permanently delete the message 
<BR>and any attachments. Thank you
<BR><BR><BR></PRE></DIV></DIV></BLOCKQUOTE></DIV><BR></DIV></BLOCKQUOTE></DIV>
<DIV></DIV></font><pre style="white-space:normal">=====-----=====-----=====<br>Notice:
The information contained in this e-mail<br>message and/or attachments to it may contain
<br>confidential or privileged information. If you are <br>not the intended recipient,
any dissemination, use, <br>review, distribution, printing or copying of the <br>information
contained in this e-mail message <br>and/or attachments to it are strictly prohibited.
If <br>you have received this communication in error, <br>please notify us by
reply e-mail or telephone and <br>immediately and permanently delete the message <br>and
any attachments. Thank you<br><br><br></pre>

Mime
View raw message