cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Frank LoVecchio <fr...@isidorey.com>
Subject Re: Cassandra nodes on EC2 in two different regions not communicating
Date Thu, 24 Feb 2011 18:18:18 GMT
Not sure if there is a particular reason for you using different regions,
but Amazon states that each zone is a different physical location completely
separate from others, e.g. us-east-1a and us-east-1b.  Using the Amazon
internal IPs (10.x. etc) reduces latency greatly by not going outbound
through DNS (though us-east-1c is twice as latent), you have an RF 4
possible (in different physical locations!), and most importantly.... sh*t
just works.

Maybe this won't help you, but it may be useful for others :)

On Thu, Feb 24, 2011 at 10:53 AM, Dave Viner <daveviner@gmail.com> wrote:

> Another possibility is this:
>
> why not setup 2 nodes in 1 region in 1 az, and get that to work.
> Then, open a third node in the same region, but different AZ, and get that
> to work.
> Then, once you have that working, open a fourth node in a different region
> and get that to work.
>
> Seems like taking a piece-meal approach would be beneficial here.
>
> Dave Viner
>
>
> On Thu, Feb 24, 2011 at 6:11 AM, Daniel van Ham Colchete <
> daniel.colchete@gmail.com> wrote:
>
>> Himanshi,
>>
>> my bad, try this for iptables:
>>
>> # SNAT outgoing connections
>> iptables -t nat -A POSTROUTING -p tcp --dport 7000 -d 175.41.143.192 -j
>> SNAT --to-source INTERNALIP
>>
>> As for tcpdump the argument for the -i option is the interface name (eth0,
>> cassth0, etc...), and not the IP. So, it should be
>> tcpdump -i cassth0 -n port 7000
>> or
>> tcpdump -i eth0 -n port 7000
>>
>> I`m assuming your main network card is eth0, but that should be the case.
>>
>> Does it work?
>>
>> Best,
>> Daniel
>>
>>
>> On Thu, Feb 24, 2011 at 9:27 AM, Himanshi Sharma <himanshi.sharma@tcs.com
>> > wrote:
>>
>>>
>>> Thanks Daniel.
>>>
>>> But SNAT command is not working and when i try tcpdump it gives
>>>
>>> [root@ip-10-136-75-201 ~]# tcpdump -i 50.18.60.117 -n port 7000
>>> tcpdump: Invalid adapter index
>>>
>>> Not able to figure out wats this ??
>>>
>>> Thanks,
>>> Himanshi
>>>
>>>
>>>
>>>  From: Daniel van Ham Colchete <daniel.colchete@gmail.com> To:
>>> user@cassandra.apache.org Date: 02/24/2011 04:27 PM Subject: Re:
>>> Cassandra nodes on EC2 in two different regions not communicating
>>> ------------------------------
>>>
>>>
>>>
>>> Himanshi,
>>>
>>> you could try adding your public IP address to an internal interface and
>>> DNAT the packets to it. This shouldn't give you any problems with your
>>> normal traffic. Tell Cassandra on listen on the public IPs and it should
>>> work.
>>>
>>> Linux commands would be:
>>>
>>> # Create an internal interface using bridge-utils
>>> brctl addbr cassth0
>>>
>>> # add the ip
>>> ip addr add dev cassth0 *50.18.60.117/32* <http://50.18.60.117/32>
>>>
>>> # DNAT incoming connections
>>> iptables -t nat -A PREROUTING -p tcp --dport 7000 -d INTERNALIP -j DNAT
>>> --to-destination 50.18.60.117
>>>
>>> # SNAT outgoing connections
>>> iptables -t nat -A OUTPUT -p tcp --dport 7000 -d 175.41.143.192 -j SNAT
>>> --to-source INTERNALIP
>>>
>>> This should work since Amazon you re-SNAT your outgoing packets to your
>>> public IP again, so the other cassandra instance will see your public IP as
>>> your source address.
>>>
>>> I didn't test this setup here but it should work unless I forgot some
>>> small detail. If you need to troubleshoot use the command "tcpdump -i
>>> INTERFACE -n port 7000" where INTERFACE should be your public interface or
>>> your cassth0.
>>>
>>> Please let me know if it worked.
>>>
>>> Best regards,
>>> Daniel Colchete
>>>
>>> On Thu, Feb 24, 2011 at 4:04 AM, Himanshi Sharma <*
>>> himanshi.sharma@tcs.com* <himanshi.sharma@tcs.com>> wrote:
>>> giving private ip to rpc address gives the same exception
>>> and the keeping it blank and providing public to listen also fails. I
>>> tried keeping both blank and did telnet on 7000 so i get following o/p
>>>
>>> [root@ip-10-166-223-150 bin]# telnet 122.248.193.37 7000
>>> Trying 122.248.193.37...
>>> Connected to 122.248.193.37.
>>> Escape character is '^]'.
>>>
>>> Similarly from another achine
>>>
>>> [root@ip-10-136-75-201 bin]# telnet 184.72.22.87 7000
>>> Trying 184.72.22.87...
>>> Connected to 184.72.22.87.
>>> Escape character is '^]'.
>>>
>>>
>>>
>>> -----Dave Viner wrote: -----
>>> To: *user@cassandra.apache.org* <user@cassandra.apache.org>
>>> From: Dave Viner <*daveviner@gmail.com* <daveviner@gmail.com>>
>>> Date: 02/24/2011 11:59AM
>>> cc: Himanshi Sharma <*himanshi.sharma@tcs.com* <himanshi.sharma@tcs.com>
>>> >
>>>
>>> Subject: Re: Cassandra nodes on EC2 in two different regions not
>>> communicating
>>>
>>> Try using the private ipv4 address in the rpc_address field, and the
>>> public ipv4 (NOT the elastic ip) in the listen_address.
>>>
>>> If that fails, go back to rpc_address empty, and start up cassandra.
>>>
>>> Then from the other node, please telnet to port 7000 on the first node.
>>>  And show the output of that session in your reply.
>>>
>>> I haven't actually constructed a cross-region cluster nor have I used
>>> v0.7, but this really sounds like it should be easy.
>>>
>>> On Wed, Feb 23, 2011 at 10:22 PM, Himanshi Sharma < *himanshi.sharma@tcs.com
>>> * <himanshi.sharma@tcs.com>> wrote:
>>> Hi Dave,
>>>
>>> I tried with the public ips. If i mention the public ip in rpc address
>>> field, Cassandra gives the same exception but if leave it blank then
>>> Cassandra runs but again in the nodetool command with ring option it does'nt
>>> show the node in another region.
>>>
>>> Thanks,
>>> Himanshi
>>>
>>>
>>> -----Dave Viner wrote: -----
>>> To: *user@cassandra.apache.org * <user@cassandra.apache.org>
>>> From: Dave Viner < *daveviner@gmail.com * <daveviner@gmail.com>>
>>> Date: 02/24/2011 10:43AM
>>>
>>> Subject: Re: Cassandra nodes on EC2 in two different regions not
>>> communicating
>>>
>>> That looks like it's not an issue of communicating between nodes.  It
>>> appears that the node can not bind to the address on the localhost that
>>> you're asking for.
>>>
>>> " java.net.BindException: Cannot assign requested address  "
>>>
>>> I think the issue is that the Elastic IP address is not actually an IP
>>> address that's on the localhost.  So the daemon can not bind to that IP.
>>>  Instead of using the EIP, use the local IP address for the rpc_address (i
>>> think that's what you need since that is what Thrift will bind to).  Then
>>> for the listen_address should be the ip address that is routable from the
>>> other node.  I would first try with the actual public IP address (not the
>>> Elastic IP).  Once you get that to work, then shutdown the cluster, change
>>> the listen_address to the EIP, boot up and try again.
>>>
>>> Dave Viner
>>>
>>>
>>> On Wed, Feb 23, 2011 at 8:54 PM, Himanshi Sharma < *himanshi.sharma@tcs.com
>>> * <himanshi.sharma@tcs.com>> wrote:
>>>
>>> Hey Dave,
>>>
>>> Sorry i forgot to mention the Non-seed configuration.
>>>
>>> for first node in us-west its as below....i.e its own elastic ip
>>>
>>> listen_address: 50.18.60.117
>>> rpc_address: 50.18.60.117
>>>
>>> and for second node in ap-southeast-1 its as below....i.e again its own
>>> elastic ip
>>>
>>> listen_address: 175.41.143.192
>>> rpc_address: 175.41.143.192
>>>
>>> Thanks,
>>> Himanshi
>>>
>>>
>>>
>>>
>>>   From: Dave Viner < *daveviner@gmail.com * <daveviner@gmail.com>>
 To:
>>> *user@cassandra.apache.org * <user@cassandra.apache.org> Date: 02/23/2011
>>> 11:01 PM  Subject: Re: Cassandra nodes on EC2 in two different regions
>>> not communicating
>>>
>>>  ------------------------------
>>>
>>>
>>>
>>> internal EC2 ips (10.xxx.xxx.xxx) work across availability zones (e.g.,
>>> from us-east-1a to us-east-1b) but do not work across regions (e.g., us-east
>>> to us-west).  To do regions, you must use the public ip address assigned by
>>> amazon.
>>>
>>> Himanshi, when you log into 1 node, and telnet to port 7000 on the other
>>> node, which IP address did you use - the 10.x address or the public ip
>>> address?
>>> And what is the seed/non-seed configuration in both cassandra.yaml files?
>>>
>>>
>>> Dave Viner
>>>
>>>
>>> On Wed, Feb 23, 2011 at 8:12 AM, Frank LoVecchio < *frank@isidorey.com *<frank@isidorey.com>>
>>> wrote:
>>> The internal Amazon IP address is what you will want to use so you don't
>>> have to go through DNS anyways; not sure if this works from US-East to
>>> US-West, but it does make things quicker in between zones, e.g. us-east-1a
>>> to us-east-1b.
>>>
>>>
>>> On Wed, Feb 23, 2011 at 9:09 AM, Dave Viner < *daveviner@gmail.com *<daveviner@gmail.com>>
>>> wrote:
>>> Try using the IP address, not the dns name in the cassandra.yaml.
>>>
>>> If you can telnet from one to the other on port 7000, and both nodes have
>>> the other node in their config, it should work.
>>>
>>> Dave Viner
>>>
>>>
>>> On Wed, Feb 23, 2011 at 1:43 AM, Himanshi Sharma < *himanshi.sharma@tcs.com
>>> * <himanshi.sharma@tcs.com>> wrote:
>>>
>>> Ya they do. Have specified Public DNS in seed field of each node in
>>> Cassandra.yaml...nt able to figure out what the problem is ???
>>>
>>>
>>>   From: Sasha Dolgy < *sdolgy@gmail.com * <sdolgy@gmail.com>>  To:
*user@cassandra.apache.org
>>> * <user@cassandra.apache.org> Date: 02/23/2011 02:56 PM  Subject: Re:
>>> Cassandra nodes on EC2 in two different regions not communicating
>>>
>>>
>>>  ------------------------------
>>>
>>>
>>>
>>> did you define the other host in the cassandra.yaml ?  on both servers
>>> .... they need to know about each other
>>>
>>> On Wed, Feb 23, 2011 at 10:16 AM, Himanshi Sharma < *himanshi.sharma@tcs.com
>>> * <himanshi.sharma@tcs.com>> wrote:
>>>
>>> Thanks Dave but I am able to telnet to other instances on port 7000
>>> and when i run  ./nodetool --host *ec2-50-18-60-117.us-west-1.compute.amazonaws.com
>>> * <http://ec2-50-18-60-117.us-west-1.compute.amazonaws.com/> ring... I
>>> can see only one node.
>>>
>>> Do we need to configure anything else in Cassandra.yaml or
>>> Cassandra-env.sh ???
>>>
>>>
>>>
>>>
>>>   From: Dave Viner < *daveviner@gmail.com * <daveviner@gmail.com>>
 To:
>>> *user@cassandra.apache.org * <user@cassandra.apache.org> Cc: Himanshi
>>> Sharma < *himanshi.sharma@tcs.com * <himanshi.sharma@tcs.com>>  Date:
02/23/2011
>>> 11:36 AM  Subject: Re: Cassandra nodes on EC2 in two different regions
>>> not communicating
>>>
>>>
>>>
>>>  ------------------------------
>>>
>>>
>>>
>>> If you login to one of the nodes, can you telnet to port 7000 on the
>>> other node?
>>>
>>> If not, then almost certainly it's a firewall/Security Group issue.
>>>
>>> You can find out the security groups for any node by logging in, and then
>>> running:
>>>
>>> % curl " *http://169.254.169.254/latest/meta-data/security-groups *<http://169.254.169.254/latest/meta-data/security-groups>"
>>>
>>>
>>> Assuming that both nodes are in the same security group, ensure that the
>>> SG is configured to allow other members of the SG to communicate on port
>>> 7000 to each other.
>>>
>>> HTH,
>>> Dave Viner
>>>
>>>
>>> On Tue, Feb 22, 2011 at 8:59 PM, Himanshi Sharma < *himanshi.sharma@tcs.com
>>> * <himanshi.sharma@tcs.com>> wrote:
>>>
>>> Hi,
>>>
>>> I am new to Cassandra. I m running Cassandra on EC2. I configured
>>> Cassandra cluster on two instances in different regions.
>>> But when I am trying the nodetool command with ring option, I am getting
>>> only single node.
>>>
>>> How to make these two nodes communicate with each other. I have already
>>> opened required ports. i.e 7000, 8080, 9160 in respective
>>> security groups. Plz help me with this.
>>>
>>> Regards,
>>> Himanshi Sharma
>>>
>>>
>>> =====-----=====-----=====
>>> Notice: The information contained in this e-mail
>>> message and/or attachments to it may contain
>>> confidential or privileged information. If you are
>>>
>>> not the intended recipient, any dissemination, use,
>>> review, distribution, printing or copying of the
>>> information contained in this e-mail message
>>> and/or attachments to it are strictly prohibited. If
>>> you have received this communication in error,
>>>
>>> please notify us by reply e-mail or telephone and
>>> immediately and permanently delete the message
>>> and any attachments. Thank you
>>>
>>>
>>>
>>>
>>> =====-----=====-----=====
>>>
>>>
>>> Notice: The information contained in this e-mail
>>> message and/or attachments to it may contain
>>> confidential or privileged information. If you are
>>> not the intended recipient, any dissemination, use,
>>> review, distribution, printing or copying of the
>>>
>>>
>>> information contained in this e-mail message
>>> and/or attachments to it are strictly prohibited. If
>>> you have received this communication in error,
>>> please notify us by reply e-mail or telephone and
>>> immediately and permanently delete the message
>>>
>>>
>>> and any attachments. Thank you
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Sasha Dolgy *
>>> **sasha.dolgy@gmail.com * <sasha.dolgy@gmail.com>
>>>
>>> =====-----=====-----=====
>>> Notice: The information contained in this e-mail
>>> message and/or attachments to it may contain
>>> confidential or privileged information. If you are
>>>
>>>
>>>
>>> not the intended recipient, any dissemination, use,
>>> review, distribution, printing or copying of the
>>> information contained in this e-mail message
>>> and/or attachments to it are strictly prohibited. If
>>> you have received this communication in error,
>>>
>>>
>>>
>>> please notify us by reply e-mail or telephone and
>>> immediately and permanently delete the message
>>> and any attachments. Thank you
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Frank LoVecchio
>>> Senior Software Engineer | Isidorey, LLC
>>> Google Voice +1.720.295.9179 *
>>> **isidorey.com * <http://isidorey.com/>| *facebook.com/franklovecchio *<http://facebook.com/franklovecchio>|
>>> *franklovecchio.com * <http://franklovecchio.com/>
>>>
>>>
>>>
>>> =====-----=====-----=====
>>>
>>> Notice: The information contained in this e-mail
>>>
>>> message and/or attachments to it may contain
>>>
>>> confidential or privileged information. If you are
>>>
>>>
>>> not the intended recipient, any dissemination, use,
>>>
>>> review, distribution, printing or copying of the
>>>
>>> information contained in this e-mail message
>>>
>>> and/or attachments to it are strictly prohibited. If
>>>
>>> you have received this communication in error,
>>>
>>>
>>> please notify us by reply e-mail or telephone and
>>>
>>> immediately and permanently delete the message
>>>
>>> and any attachments. Thank you
>>>
>>>
>>>
>>>
>>> =====-----=====-----=====
>>>
>>> Notice: The information contained in this e-mail
>>>
>>> message and/or attachments to it may contain
>>>
>>> confidential or privileged information. If you are
>>>
>>>
>>> not the intended recipient, any dissemination, use,
>>>
>>> review, distribution, printing or copying of the
>>>
>>> information contained in this e-mail message
>>>
>>> and/or attachments to it are strictly prohibited. If
>>>
>>> you have received this communication in error,
>>>
>>>
>>> please notify us by reply e-mail or telephone and
>>>
>>> immediately and permanently delete the message
>>>
>>> and any attachments. Thank you
>>>
>>>
>>>
>>>
>>> =====-----=====-----=====
>>>
>>> Notice: The information contained in this e-mail
>>> message and/or attachments to it may contain
>>> confidential or privileged information. If you are
>>>
>>>
>>> not the intended recipient, any dissemination, use,
>>> review, distribution, printing or copying of the
>>> information contained in this e-mail message
>>> and/or attachments to it are strictly prohibited. If
>>> you have received this communication in error,
>>>
>>>
>>> please notify us by reply e-mail or telephone and
>>> immediately and permanently delete the message
>>> and any attachments. Thank you
>>>
>>>
>>>
>>>
>>> =====-----=====-----=====
>>>
>>> Notice: The information contained in this e-mail
>>> message and/or attachments to it may contain
>>> confidential or privileged information. If you are
>>>
>>>
>>>
>>> not the intended recipient, any dissemination, use,
>>> review, distribution, printing or copying of the
>>> information contained in this e-mail message
>>> and/or attachments to it are strictly prohibited. If
>>> you have received this communication in error,
>>>
>>>
>>>
>>> please notify us by reply e-mail or telephone and
>>> immediately and permanently delete the message
>>> and any attachments. Thank you
>>>
>>>
>>>
>>
>


-- 
Frank LoVecchio
Senior Software Engineer | Isidorey, LLC
Google Voice +1.720.295.9179
isidorey.com | facebook.com/franklovecchio | franklovecchio.com

Mime
View raw message