incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hanna <jeremy.hanna1...@gmail.com>
Subject Re: EC2 - 2 regions
Date Tue, 22 Mar 2011 03:42:34 GMT
Sorry if I was presumptuous earlier.  I created a ticket so that the patch could be submitted
and reviewed - that is if it can be generalized so that it works across regions and doesn't
adversely affect the common case.
https://issues.apache.org/jira/browse/CASSANDRA-2362

On Mar 21, 2011, at 10:41 PM, Jeremy Hanna wrote:

> Sorry if I was presumptuous earlier.  I created a ticket so that the patch could be submitted
and reviewed - that is if it can be generalized so that it works across regions and doesn't
adversely affect the common case.
> https://issues.apache.org/jira/browse/CASSANDRA-2362
> 
> On Mar 21, 2011, at 12:20 PM, Jeremy Hanna wrote:
> 
>> I talked to Matt Dennis in the channel about it and I think everyone would like to
make sure that cassandra works great across multiple regions.  He sounded like he didn't know
why it wouldn't work after having looked at the patches.  I would like to try it both ways
- with and without the patches later today if I can and I'd like to help out with getting
it working out of the box.
>> 
>> Thanks for the investigative work and documentation Milind!
>> 
>> Jeremy
>> 
>> On Mar 21, 2011, at 12:12 PM, Dave Viner wrote:
>> 
>>> Hi Milind,
>>> 
>>> Great work here.  Can you provide the patch against the 2 files?
>>> 
>>> Perhaps there's some way to incorporate it into the trunk of cassandra so that
this is feasible (in a future release) without patching the source code.
>>> 
>>> Dave Viner
>>> 
>>> 
>>> On Mon, Mar 21, 2011 at 9:41 AM, A J <s5alye@gmail.com> wrote:
>>> Thanks for sharing the document, Milind !
>>> Followed the instructions and it worked for me.
>>> 
>>> On Mon, Mar 21, 2011 at 5:01 AM, Milind Parikh <milindparikh@gmail.com>
wrote:
>>>> Here's the document on Cassandra (0.7.4) across EC2 regions. Clearly this
is
>>>> work in progress.... but wanted to share what I have. PDF is the working
>>>> copy.
>>>> 
>>>> 
>>>> https://docs.google.com/document/d/175duUNIx7m5mCDa2sjXVI04ekyMa5bdiWdu-AFgisaY/edit?hl=en
>>>> 
>>>> On Sun, Mar 20, 2011 at 7:49 PM, aaron morton <aaron@thelastpickle.com>
>>>> wrote:
>>>>> 
>>>>> Recent discussion on the dev list
>>>>> http://www.mail-archive.com/dev@cassandra.apache.org/msg01832.html
>>>>> Aaron
>>>>> On 19 Mar 2011, at 06:46, A J wrote:
>>>>> 
>>>>> Just to add, all the telnet (port 7000) and cassandra-cli (port 9160)
>>>>> connections are done using the public DNS (that goes like
>>>>> ec2-.....compute.amazonaws.com)
>>>>> 
>>>>> On Fri, Mar 18, 2011 at 1:37 PM, A J <s5alye@gmail.com> wrote:
>>>>> 
>>>>> I am able to telnet from one region to another on 7000 port without
>>>>> 
>>>>> issues. (I get the expected Connected to .....Escape character is
>>>>> 
>>>>> '^]'.)
>>>>> 
>>>>> Also I am able to execute cassandra client on 9160 port from one
>>>>> 
>>>>> region to another without issues (this is when I run cassandra
>>>>> 
>>>>> separately on each region without forming a cluster).
>>>>> 
>>>>> So I think the ports 7000 and 9160 are not the issue.
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Mar 18, 2011 at 1:26 PM, Dave Viner <daveviner@gmail.com>
wrote:
>>>>> 
>>>>> From the us-west instance, are you able to connect to the us-east instance
>>>>> 
>>>>> using telnet on port 7000 and 9160?
>>>>> 
>>>>> If not, then you need to open those ports for communication (via your
>>>>> 
>>>>> Security Group)
>>>>> 
>>>>> Dave Viner
>>>>> 
>>>>> On Fri, Mar 18, 2011 at 10:20 AM, A J <s5alye@gmail.com> wrote:
>>>>> 
>>>>> Thats exactly what I am doing.
>>>>> 
>>>>> I was able to do the first two scenarios without any issues (i.e. 2
>>>>> 
>>>>> nodes in same availability zone. Followed by an additional node in a
>>>>> 
>>>>> different zone but same region)
>>>>> 
>>>>> I am stuck at the third scenario of separate regions.
>>>>> 
>>>>> (I did read the "Cassandra nodes on EC2 in two different regions not
>>>>> 
>>>>> communicating" thread but it did not seem to end with resolution)
>>>>> 
>>>>> 
>>>>> On Fri, Mar 18, 2011 at 1:15 PM, Dave Viner <daveviner@gmail.com>
wrote:
>>>>> 
>>>>> Hi AJ,
>>>>> 
>>>>> I'd suggest getting to a multi-region cluster step-by-step.  First, get
>>>>> 
>>>>> 2
>>>>> 
>>>>> nodes running in the same availability zone.  Make sure that works
>>>>> 
>>>>> properly.
>>>>> 
>>>>> Second, add a node in a separate availability zone, but in the same
>>>>> 
>>>>> region.
>>>>> 
>>>>> Make sure that's working properly.  Third, add a node that's in a
>>>>> 
>>>>> separate
>>>>> 
>>>>> region.
>>>>> 
>>>>> Taking it step-by-step will ensure that any issues are specific to the
>>>>> 
>>>>> region-to-region communication, rather than intra-zone connectivity or
>>>>> 
>>>>> cassandra cluster configuration.
>>>>> 
>>>>> Dave Viner
>>>>> 
>>>>> On Fri, Mar 18, 2011 at 8:34 AM, A J <s5alye@gmail.com> wrote:
>>>>> 
>>>>> Hello,
>>>>> 
>>>>> I am trying to setup a cassandra cluster across regions.
>>>>> 
>>>>> For testing I am keeping it simple and just having one node in US-EAST
>>>>> 
>>>>> (say ec2-1-2-3-4.compute-1.amazonaws.com) and one node in US-WEST (say
>>>>> 
>>>>> ec2-2-2-3-4.us-west-1.compute.amazonaws.com).
>>>>> 
>>>>> Using Cassandra 0.7.4
>>>>> 
>>>>> 
>>>>> The one in east region is the seed node and has the values as:
>>>>> 
>>>>> auto_bootstrap: false
>>>>> 
>>>>> seeds: ec2-1-2-3-4.compute-1.amazonaws.com
>>>>> 
>>>>> listen_address: ec2-1-2-3-4.compute-1.amazonaws.com
>>>>> 
>>>>> rpc_address: 0.0.0.0
>>>>> 
>>>>> The one in west region is non seed and has the values as:
>>>>> 
>>>>> auto_bootstrap: true
>>>>> 
>>>>> seeds: ec2-1-2-3-4.compute-1.amazonaws.com
>>>>> 
>>>>> listen_address: ec2-2-2-3-4.us-west-1.compute.amazonaws.com
>>>>> 
>>>>> rpc_address: 0.0.0.0
>>>>> 
>>>>> I first fire the seed node (east region instance) and it comes up
>>>>> 
>>>>> without issues.
>>>>> 
>>>>> When I fire the non-seed node (west region instance) it fails after
>>>>> 
>>>>> sometime with the error:
>>>>> 
>>>>> DEBUG 15:09:08,844 Created HHOM instance, registered MBean.
>>>>> 
>>>>> INFO 15:09:08,844 Joining: getting load information
>>>>> 
>>>>> INFO 15:09:08,845 Sleeping 90000 ms to wait for load information...
>>>>> 
>>>>> DEBUG 15:09:09,822 attempting to connect to
>>>>> 
>>>>> ec2-1-2-3-4.compute-1.amazonaws.com/1.2.3.4
>>>>> 
>>>>> DEBUG 15:09:10,825 Disseminating load info ...
>>>>> 
>>>>> DEBUG 15:10:10,826 Disseminating load info ...
>>>>> 
>>>>> DEBUG 15:10:38,845 ... got load info
>>>>> 
>>>>> INFO 15:10:38,845 Joining: getting bootstrap token
>>>>> 
>>>>> ERROR 15:10:38,847 Exception encountered during startup.
>>>>> 
>>>>> java.lang.RuntimeException: No other nodes seen!  Unable to bootstrap
>>>>> 
>>>>>      at
>>>>> 
>>>>> 
>>>>> org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper.java:164)
>>>>> 
>>>>>      at
>>>>> 
>>>>> 
>>>>> org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:146)
>>>>> 
>>>>>      at
>>>>> 
>>>>> 
>>>>> org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.java:141)
>>>>> 
>>>>>      at
>>>>> 
>>>>> 
>>>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:450)
>>>>> 
>>>>>      at
>>>>> 
>>>>> 
>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:404)
>>>>> 
>>>>>      at
>>>>> 
>>>>> 
>>>>> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:192)
>>>>> 
>>>>>      at
>>>>> 
>>>>> 
>>>>> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
>>>>> 
>>>>>      at
>>>>> 
>>>>> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
>>>>> 
>>>>> 
>>>>> The seed node seems to somewhat acknowledge the non-seed node:
>>>>> 
>>>>> attempting to connect to /2.2.3.4
>>>>> 
>>>>> attempting to connect to /10.170.190.31
>>>>> 
>>>>> Can you suggest how can I fix it (I did see a few threads on similar
>>>>> 
>>>>> issue but did not really follow the chain)
>>>>> 
>>>>> Thanks, AJ
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>> 
> 


Mime
View raw message