cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dave Viner <davevi...@gmail.com>
Subject Re: Cassandra nodes on EC2 in two different regions not communicating
Date Thu, 24 Feb 2011 17:53:36 GMT
Another possibility is this:

why not setup 2 nodes in 1 region in 1 az, and get that to work.
Then, open a third node in the same region, but different AZ, and get that
to work.
Then, once you have that working, open a fourth node in a different region
and get that to work.

Seems like taking a piece-meal approach would be beneficial here.

Dave Viner


On Thu, Feb 24, 2011 at 6:11 AM, Daniel van Ham Colchete <
daniel.colchete@gmail.com> wrote:

> Himanshi,
>
> my bad, try this for iptables:
>
> # SNAT outgoing connections
> iptables -t nat -A POSTROUTING -p tcp --dport 7000 -d 175.41.143.192 -j
> SNAT --to-source INTERNALIP
>
> As for tcpdump the argument for the -i option is the interface name (eth0,
> cassth0, etc...), and not the IP. So, it should be
> tcpdump -i cassth0 -n port 7000
> or
> tcpdump -i eth0 -n port 7000
>
> I`m assuming your main network card is eth0, but that should be the case.
>
> Does it work?
>
> Best,
> Daniel
>
>
> On Thu, Feb 24, 2011 at 9:27 AM, Himanshi Sharma <himanshi.sharma@tcs.com>wrote:
>
>>
>> Thanks Daniel.
>>
>> But SNAT command is not working and when i try tcpdump it gives
>>
>> [root@ip-10-136-75-201 ~]# tcpdump -i 50.18.60.117 -n port 7000
>> tcpdump: Invalid adapter index
>>
>> Not able to figure out wats this ??
>>
>> Thanks,
>> Himanshi
>>
>>
>>
>>  From: Daniel van Ham Colchete <daniel.colchete@gmail.com> To:
>> user@cassandra.apache.org Date: 02/24/2011 04:27 PM Subject: Re:
>> Cassandra nodes on EC2 in two different regions not communicating
>> ------------------------------
>>
>>
>>
>> Himanshi,
>>
>> you could try adding your public IP address to an internal interface and
>> DNAT the packets to it. This shouldn't give you any problems with your
>> normal traffic. Tell Cassandra on listen on the public IPs and it should
>> work.
>>
>> Linux commands would be:
>>
>> # Create an internal interface using bridge-utils
>> brctl addbr cassth0
>>
>> # add the ip
>> ip addr add dev cassth0 *50.18.60.117/32* <http://50.18.60.117/32>
>>
>> # DNAT incoming connections
>> iptables -t nat -A PREROUTING -p tcp --dport 7000 -d INTERNALIP -j DNAT
>> --to-destination 50.18.60.117
>>
>> # SNAT outgoing connections
>> iptables -t nat -A OUTPUT -p tcp --dport 7000 -d 175.41.143.192 -j SNAT
>> --to-source INTERNALIP
>>
>> This should work since Amazon you re-SNAT your outgoing packets to your
>> public IP again, so the other cassandra instance will see your public IP as
>> your source address.
>>
>> I didn't test this setup here but it should work unless I forgot some
>> small detail. If you need to troubleshoot use the command "tcpdump -i
>> INTERFACE -n port 7000" where INTERFACE should be your public interface or
>> your cassth0.
>>
>> Please let me know if it worked.
>>
>> Best regards,
>> Daniel Colchete
>>
>> On Thu, Feb 24, 2011 at 4:04 AM, Himanshi Sharma <*
>> himanshi.sharma@tcs.com* <himanshi.sharma@tcs.com>> wrote:
>> giving private ip to rpc address gives the same exception
>> and the keeping it blank and providing public to listen also fails. I
>> tried keeping both blank and did telnet on 7000 so i get following o/p
>>
>> [root@ip-10-166-223-150 bin]# telnet 122.248.193.37 7000
>> Trying 122.248.193.37...
>> Connected to 122.248.193.37.
>> Escape character is '^]'.
>>
>> Similarly from another achine
>>
>> [root@ip-10-136-75-201 bin]# telnet 184.72.22.87 7000
>> Trying 184.72.22.87...
>> Connected to 184.72.22.87.
>> Escape character is '^]'.
>>
>>
>>
>> -----Dave Viner wrote: -----
>> To: *user@cassandra.apache.org* <user@cassandra.apache.org>
>> From: Dave Viner <*daveviner@gmail.com* <daveviner@gmail.com>>
>> Date: 02/24/2011 11:59AM
>> cc: Himanshi Sharma <*himanshi.sharma@tcs.com* <himanshi.sharma@tcs.com>>
>>
>> Subject: Re: Cassandra nodes on EC2 in two different regions not
>> communicating
>>
>> Try using the private ipv4 address in the rpc_address field, and the
>> public ipv4 (NOT the elastic ip) in the listen_address.
>>
>> If that fails, go back to rpc_address empty, and start up cassandra.
>>
>> Then from the other node, please telnet to port 7000 on the first node.
>>  And show the output of that session in your reply.
>>
>> I haven't actually constructed a cross-region cluster nor have I used
>> v0.7, but this really sounds like it should be easy.
>>
>> On Wed, Feb 23, 2011 at 10:22 PM, Himanshi Sharma < *himanshi.sharma@tcs.com
>> * <himanshi.sharma@tcs.com>> wrote:
>> Hi Dave,
>>
>> I tried with the public ips. If i mention the public ip in rpc address
>> field, Cassandra gives the same exception but if leave it blank then
>> Cassandra runs but again in the nodetool command with ring option it does'nt
>> show the node in another region.
>>
>> Thanks,
>> Himanshi
>>
>>
>> -----Dave Viner wrote: -----
>> To: *user@cassandra.apache.org * <user@cassandra.apache.org>
>> From: Dave Viner < *daveviner@gmail.com * <daveviner@gmail.com>>
>> Date: 02/24/2011 10:43AM
>>
>> Subject: Re: Cassandra nodes on EC2 in two different regions not
>> communicating
>>
>> That looks like it's not an issue of communicating between nodes.  It
>> appears that the node can not bind to the address on the localhost that
>> you're asking for.
>>
>> " java.net.BindException: Cannot assign requested address  "
>>
>> I think the issue is that the Elastic IP address is not actually an IP
>> address that's on the localhost.  So the daemon can not bind to that IP.
>>  Instead of using the EIP, use the local IP address for the rpc_address (i
>> think that's what you need since that is what Thrift will bind to).  Then
>> for the listen_address should be the ip address that is routable from the
>> other node.  I would first try with the actual public IP address (not the
>> Elastic IP).  Once you get that to work, then shutdown the cluster, change
>> the listen_address to the EIP, boot up and try again.
>>
>> Dave Viner
>>
>>
>> On Wed, Feb 23, 2011 at 8:54 PM, Himanshi Sharma < *himanshi.sharma@tcs.com
>> * <himanshi.sharma@tcs.com>> wrote:
>>
>> Hey Dave,
>>
>> Sorry i forgot to mention the Non-seed configuration.
>>
>> for first node in us-west its as below....i.e its own elastic ip
>>
>> listen_address: 50.18.60.117
>> rpc_address: 50.18.60.117
>>
>> and for second node in ap-southeast-1 its as below....i.e again its own
>> elastic ip
>>
>> listen_address: 175.41.143.192
>> rpc_address: 175.41.143.192
>>
>> Thanks,
>> Himanshi
>>
>>
>>
>>
>>   From: Dave Viner < *daveviner@gmail.com * <daveviner@gmail.com>>  To:
*user@cassandra.apache.org
>> * <user@cassandra.apache.org> Date: 02/23/2011 11:01 PM  Subject: Re:
>> Cassandra nodes on EC2 in two different regions not communicating
>>
>>  ------------------------------
>>
>>
>>
>> internal EC2 ips (10.xxx.xxx.xxx) work across availability zones (e.g.,
>> from us-east-1a to us-east-1b) but do not work across regions (e.g., us-east
>> to us-west).  To do regions, you must use the public ip address assigned by
>> amazon.
>>
>> Himanshi, when you log into 1 node, and telnet to port 7000 on the other
>> node, which IP address did you use - the 10.x address or the public ip
>> address?
>> And what is the seed/non-seed configuration in both cassandra.yaml files?
>>
>> Dave Viner
>>
>>
>> On Wed, Feb 23, 2011 at 8:12 AM, Frank LoVecchio < *frank@isidorey.com *<frank@isidorey.com>>
>> wrote:
>> The internal Amazon IP address is what you will want to use so you don't
>> have to go through DNS anyways; not sure if this works from US-East to
>> US-West, but it does make things quicker in between zones, e.g. us-east-1a
>> to us-east-1b.
>>
>>
>> On Wed, Feb 23, 2011 at 9:09 AM, Dave Viner < *daveviner@gmail.com *<daveviner@gmail.com>>
>> wrote:
>> Try using the IP address, not the dns name in the cassandra.yaml.
>>
>> If you can telnet from one to the other on port 7000, and both nodes have
>> the other node in their config, it should work.
>>
>> Dave Viner
>>
>>
>> On Wed, Feb 23, 2011 at 1:43 AM, Himanshi Sharma < *himanshi.sharma@tcs.com
>> * <himanshi.sharma@tcs.com>> wrote:
>>
>> Ya they do. Have specified Public DNS in seed field of each node in
>> Cassandra.yaml...nt able to figure out what the problem is ???
>>
>>
>>   From: Sasha Dolgy < *sdolgy@gmail.com * <sdolgy@gmail.com>>  To: *user@cassandra.apache.org
>> * <user@cassandra.apache.org> Date: 02/23/2011 02:56 PM  Subject: Re:
>> Cassandra nodes on EC2 in two different regions not communicating
>>
>>
>>  ------------------------------
>>
>>
>>
>> did you define the other host in the cassandra.yaml ?  on both servers
>> .... they need to know about each other
>>
>> On Wed, Feb 23, 2011 at 10:16 AM, Himanshi Sharma < *himanshi.sharma@tcs.com
>> * <himanshi.sharma@tcs.com>> wrote:
>>
>> Thanks Dave but I am able to telnet to other instances on port 7000
>> and when i run  ./nodetool --host *ec2-50-18-60-117.us-west-1.compute.amazonaws.com
>> * <http://ec2-50-18-60-117.us-west-1.compute.amazonaws.com/> ring... I
>> can see only one node.
>>
>> Do we need to configure anything else in Cassandra.yaml or
>> Cassandra-env.sh ???
>>
>>
>>
>>
>>   From: Dave Viner < *daveviner@gmail.com * <daveviner@gmail.com>>  To:
*user@cassandra.apache.org
>> * <user@cassandra.apache.org> Cc: Himanshi Sharma < *himanshi.sharma@tcs.com
>> * <himanshi.sharma@tcs.com>>  Date: 02/23/2011 11:36 AM  Subject: Re:
>> Cassandra nodes on EC2 in two different regions not communicating
>>
>>
>>
>>  ------------------------------
>>
>>
>>
>> If you login to one of the nodes, can you telnet to port 7000 on the other
>> node?
>>
>> If not, then almost certainly it's a firewall/Security Group issue.
>>
>> You can find out the security groups for any node by logging in, and then
>> running:
>>
>> % curl " *http://169.254.169.254/latest/meta-data/security-groups *<http://169.254.169.254/latest/meta-data/security-groups>"
>>
>>
>> Assuming that both nodes are in the same security group, ensure that the
>> SG is configured to allow other members of the SG to communicate on port
>> 7000 to each other.
>>
>> HTH,
>> Dave Viner
>>
>>
>> On Tue, Feb 22, 2011 at 8:59 PM, Himanshi Sharma < *himanshi.sharma@tcs.com
>> * <himanshi.sharma@tcs.com>> wrote:
>>
>> Hi,
>>
>> I am new to Cassandra. I m running Cassandra on EC2. I configured
>> Cassandra cluster on two instances in different regions.
>> But when I am trying the nodetool command with ring option, I am getting
>> only single node.
>>
>> How to make these two nodes communicate with each other. I have already
>> opened required ports. i.e 7000, 8080, 9160 in respective
>> security groups. Plz help me with this.
>>
>> Regards,
>> Himanshi Sharma
>>
>>
>> =====-----=====-----=====
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>>
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>>
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>>
>>
>>
>>
>> =====-----=====-----=====
>>
>>
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>>
>>
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>>
>>
>> and any attachments. Thank you
>>
>>
>>
>>
>>
>> --
>> Sasha Dolgy *
>> **sasha.dolgy@gmail.com * <sasha.dolgy@gmail.com>
>>
>> =====-----=====-----=====
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>>
>>
>>
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>>
>>
>>
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>>
>>
>>
>>
>>
>>
>> --
>> Frank LoVecchio
>> Senior Software Engineer | Isidorey, LLC
>> Google Voice +1.720.295.9179 *
>> **isidorey.com * <http://isidorey.com/>| *facebook.com/franklovecchio *<http://facebook.com/franklovecchio>|
>> *franklovecchio.com * <http://franklovecchio.com/>
>>
>>
>>
>> =====-----=====-----=====
>>
>> Notice: The information contained in this e-mail
>>
>> message and/or attachments to it may contain
>>
>> confidential or privileged information. If you are
>>
>>
>> not the intended recipient, any dissemination, use,
>>
>> review, distribution, printing or copying of the
>>
>> information contained in this e-mail message
>>
>> and/or attachments to it are strictly prohibited. If
>>
>> you have received this communication in error,
>>
>>
>> please notify us by reply e-mail or telephone and
>>
>> immediately and permanently delete the message
>>
>> and any attachments. Thank you
>>
>>
>>
>>
>> =====-----=====-----=====
>>
>> Notice: The information contained in this e-mail
>>
>> message and/or attachments to it may contain
>>
>> confidential or privileged information. If you are
>>
>>
>> not the intended recipient, any dissemination, use,
>>
>> review, distribution, printing or copying of the
>>
>> information contained in this e-mail message
>>
>> and/or attachments to it are strictly prohibited. If
>>
>> you have received this communication in error,
>>
>>
>> please notify us by reply e-mail or telephone and
>>
>> immediately and permanently delete the message
>>
>> and any attachments. Thank you
>>
>>
>>
>>
>> =====-----=====-----=====
>>
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>>
>>
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>>
>>
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>>
>>
>>
>>
>> =====-----=====-----=====
>>
>> Notice: The information contained in this e-mail
>> message and/or attachments to it may contain
>> confidential or privileged information. If you are
>>
>>
>> not the intended recipient, any dissemination, use,
>> review, distribution, printing or copying of the
>> information contained in this e-mail message
>> and/or attachments to it are strictly prohibited. If
>> you have received this communication in error,
>>
>>
>> please notify us by reply e-mail or telephone and
>> immediately and permanently delete the message
>> and any attachments. Thank you
>>
>>
>>
>

Mime
View raw message