incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From A J <s5a...@gmail.com>
Subject Re: EC2 - 2 regions
Date Tue, 22 Mar 2011 17:58:00 GMT
Milind,
Among the limitation you might want to add that 'nodetool repair' does
not work with this patch.
I tried several times and the repair hangs.
When I run it directly on the trunk of 0.7.4 (without the patch) it
completes successfully within reasonable time.

Thanks.

On Tue, Mar 22, 2011 at 1:07 PM, Jeremy Hanna
<jeremy.hanna1234@gmail.com> wrote:
> Never mind - I had thought it was more generalizable but since it's just going against
the public IP between regions, that's not going to be something that makes it into trunk.
 I had just wanted to see if there was a way that it could be done, but it sounds like since
amazon doesn't provide decent information between regions, something like this workaround
patch is required.
>
> Anyway - thanks for the work on this.
>
> On Mar 22, 2011, at 8:33 AM, Jeremy Hanna wrote:
>
>> Milind,
>>
>> Thank you for attaching the patch here, but it would be really nice if you could
create a jira account so you could participate in the discussion on the ticket and put the
patch on there - that is the way people license their contributions with the apache 2 license.
 You just need to create an account with the public jira inked off of the ticket at the top.
>>
>> Understandable that it would necessarily be a general solution now - but it's a start
to understanding what would need to be done so that if possible, something general could be
derived.  I'm just trying to help get the discussion started so it could be something that
people could do out of the box.  Not only that, but also so that it could be tested and evolve
with the codebase so that people could know that it is hardened and used by others.
>>
>> Any limitations would be nice to note when you attach the patch to the ticket.
>>
>> Thanks so much for your work on this!
>>
>> Jeremy
>>
>> On Mar 21, 2011, at 11:29 PM, Milind Parikh wrote:
>>
>>> Patch is attached... I don't have access to Jira.
>>>
>>> A cautionery note: This is NOT a general solution and is not intended as such.
It could be included as a part of larger patch. I will explain in the limitation sections
about why it is not a general solution; as I find time.
>>>
>>> Regards
>>> Milind
>>>
>>> On Mon, Mar 21, 2011 at 11:42 PM, Jeremy Hanna <jeremy.hanna1234@gmail.com>
wrote:
>>> Sorry if I was presumptuous earlier.  I created a ticket so that the patch could
be submitted and reviewed - that is if it can be generalized so that it works across regions
and doesn't adversely affect the common case.
>>> https://issues.apache.org/jira/browse/CASSANDRA-2362
>>>
>>> On Mar 21, 2011, at 10:41 PM, Jeremy Hanna wrote:
>>>
>>>> Sorry if I was presumptuous earlier.  I created a ticket so that the patch
could be submitted and reviewed - that is if it can be generalized so that it works across
regions and doesn't adversely affect the common case.
>>>> https://issues.apache.org/jira/browse/CASSANDRA-2362
>>>>
>>>> On Mar 21, 2011, at 12:20 PM, Jeremy Hanna wrote:
>>>>
>>>>> I talked to Matt Dennis in the channel about it and I think everyone
would like to make sure that cassandra works great across multiple regions.  He sounded like
he didn't know why it wouldn't work after having looked at the patches.  I would like to
try it both ways - with and without the patches later today if I can and I'd like to help
out with getting it working out of the box.
>>>>>
>>>>> Thanks for the investigative work and documentation Milind!
>>>>>
>>>>> Jeremy
>>>>>
>>>>> On Mar 21, 2011, at 12:12 PM, Dave Viner wrote:
>>>>>
>>>>>> Hi Milind,
>>>>>>
>>>>>> Great work here.  Can you provide the patch against the 2 files?
>>>>>>
>>>>>> Perhaps there's some way to incorporate it into the trunk of cassandra
so that this is feasible (in a future release) without patching the source code.
>>>>>>
>>>>>> Dave Viner
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 21, 2011 at 9:41 AM, A J <s5alye@gmail.com> wrote:
>>>>>> Thanks for sharing the document, Milind !
>>>>>> Followed the instructions and it worked for me.
>>>>>>
>>>>>> On Mon, Mar 21, 2011 at 5:01 AM, Milind Parikh <milindparikh@gmail.com>
wrote:
>>>>>>> Here's the document on Cassandra (0.7.4) across EC2 regions.
Clearly this is
>>>>>>> work in progress.... but wanted to share what I have. PDF is
the working
>>>>>>> copy.
>>>>>>>
>>>>>>>
>>>>>>> https://docs.google.com/document/d/175duUNIx7m5mCDa2sjXVI04ekyMa5bdiWdu-AFgisaY/edit?hl=en
>>>>>>>
>>>>>>> On Sun, Mar 20, 2011 at 7:49 PM, aaron morton <aaron@thelastpickle.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Recent discussion on the dev list
>>>>>>>> http://www.mail-archive.com/dev@cassandra.apache.org/msg01832.html
>>>>>>>> Aaron
>>>>>>>> On 19 Mar 2011, at 06:46, A J wrote:
>>>>>>>>
>>>>>>>> Just to add, all the telnet (port 7000) and cassandra-cli
(port 9160)
>>>>>>>> connections are done using the public DNS (that goes like
>>>>>>>> ec2-.....compute.amazonaws.com)
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2011 at 1:37 PM, A J <s5alye@gmail.com>
wrote:
>>>>>>>>
>>>>>>>> I am able to telnet from one region to another on 7000 port
without
>>>>>>>>
>>>>>>>> issues. (I get the expected Connected to .....Escape character
is
>>>>>>>>
>>>>>>>> '^]'.)
>>>>>>>>
>>>>>>>> Also I am able to execute cassandra client on 9160 port from
one
>>>>>>>>
>>>>>>>> region to another without issues (this is when I run cassandra
>>>>>>>>
>>>>>>>> separately on each region without forming a cluster).
>>>>>>>>
>>>>>>>> So I think the ports 7000 and 9160 are not the issue.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2011 at 1:26 PM, Dave Viner <daveviner@gmail.com>
wrote:
>>>>>>>>
>>>>>>>> From the us-west instance, are you able to connect to the
us-east instance
>>>>>>>>
>>>>>>>> using telnet on port 7000 and 9160?
>>>>>>>>
>>>>>>>> If not, then you need to open those ports for communication
(via your
>>>>>>>>
>>>>>>>> Security Group)
>>>>>>>>
>>>>>>>> Dave Viner
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2011 at 10:20 AM, A J <s5alye@gmail.com>
wrote:
>>>>>>>>
>>>>>>>> Thats exactly what I am doing.
>>>>>>>>
>>>>>>>> I was able to do the first two scenarios without any issues
(i.e. 2
>>>>>>>>
>>>>>>>> nodes in same availability zone. Followed by an additional
node in a
>>>>>>>>
>>>>>>>> different zone but same region)
>>>>>>>>
>>>>>>>> I am stuck at the third scenario of separate regions.
>>>>>>>>
>>>>>>>> (I did read the "Cassandra nodes on EC2 in two different
regions not
>>>>>>>>
>>>>>>>> communicating" thread but it did not seem to end with resolution)
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2011 at 1:15 PM, Dave Viner <daveviner@gmail.com>
wrote:
>>>>>>>>
>>>>>>>> Hi AJ,
>>>>>>>>
>>>>>>>> I'd suggest getting to a multi-region cluster step-by-step.
 First, get
>>>>>>>>
>>>>>>>> 2
>>>>>>>>
>>>>>>>> nodes running in the same availability zone.  Make sure
that works
>>>>>>>>
>>>>>>>> properly.
>>>>>>>>
>>>>>>>> Second, add a node in a separate availability zone, but in
the same
>>>>>>>>
>>>>>>>> region.
>>>>>>>>
>>>>>>>> Make sure that's working properly.  Third, add a node that's
in a
>>>>>>>>
>>>>>>>> separate
>>>>>>>>
>>>>>>>> region.
>>>>>>>>
>>>>>>>> Taking it step-by-step will ensure that any issues are specific
to the
>>>>>>>>
>>>>>>>> region-to-region communication, rather than intra-zone connectivity
or
>>>>>>>>
>>>>>>>> cassandra cluster configuration.
>>>>>>>>
>>>>>>>> Dave Viner
>>>>>>>>
>>>>>>>> On Fri, Mar 18, 2011 at 8:34 AM, A J <s5alye@gmail.com>
wrote:
>>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> I am trying to setup a cassandra cluster across regions.
>>>>>>>>
>>>>>>>> For testing I am keeping it simple and just having one node
in US-EAST
>>>>>>>>
>>>>>>>> (say ec2-1-2-3-4.compute-1.amazonaws.com) and one node in
US-WEST (say
>>>>>>>>
>>>>>>>> ec2-2-2-3-4.us-west-1.compute.amazonaws.com).
>>>>>>>>
>>>>>>>> Using Cassandra 0.7.4
>>>>>>>>
>>>>>>>>
>>>>>>>> The one in east region is the seed node and has the values
as:
>>>>>>>>
>>>>>>>> auto_bootstrap: false
>>>>>>>>
>>>>>>>> seeds: ec2-1-2-3-4.compute-1.amazonaws.com
>>>>>>>>
>>>>>>>> listen_address: ec2-1-2-3-4.compute-1.amazonaws.com
>>>>>>>>
>>>>>>>> rpc_address: 0.0.0.0
>>>>>>>>
>>>>>>>> The one in west region is non seed and has the values as:
>>>>>>>>
>>>>>>>> auto_bootstrap: true
>>>>>>>>
>>>>>>>> seeds: ec2-1-2-3-4.compute-1.amazonaws.com
>>>>>>>>
>>>>>>>> listen_address: ec2-2-2-3-4.us-west-1.compute.amazonaws.com
>>>>>>>>
>>>>>>>> rpc_address: 0.0.0.0
>>>>>>>>
>>>>>>>> I first fire the seed node (east region instance) and it
comes up
>>>>>>>>
>>>>>>>> without issues.
>>>>>>>>
>>>>>>>> When I fire the non-seed node (west region instance) it fails
after
>>>>>>>>
>>>>>>>> sometime with the error:
>>>>>>>>
>>>>>>>> DEBUG 15:09:08,844 Created HHOM instance, registered MBean.
>>>>>>>>
>>>>>>>> INFO 15:09:08,844 Joining: getting load information
>>>>>>>>
>>>>>>>> INFO 15:09:08,845 Sleeping 90000 ms to wait for load information...
>>>>>>>>
>>>>>>>> DEBUG 15:09:09,822 attempting to connect to
>>>>>>>>
>>>>>>>> ec2-1-2-3-4.compute-1.amazonaws.com/1.2.3.4
>>>>>>>>
>>>>>>>> DEBUG 15:09:10,825 Disseminating load info ...
>>>>>>>>
>>>>>>>> DEBUG 15:10:10,826 Disseminating load info ...
>>>>>>>>
>>>>>>>> DEBUG 15:10:38,845 ... got load info
>>>>>>>>
>>>>>>>> INFO 15:10:38,845 Joining: getting bootstrap token
>>>>>>>>
>>>>>>>> ERROR 15:10:38,847 Exception encountered during startup.
>>>>>>>>
>>>>>>>> java.lang.RuntimeException: No other nodes seen!  Unable
to bootstrap
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cassandra.dht.BootStrapper.getBootstrapSource(BootStrapper.java:164)
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cassandra.dht.BootStrapper.getBalancedToken(BootStrapper.java:146)
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cassandra.dht.BootStrapper.getBootstrapToken(BootStrapper.java:141)
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:450)
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:404)
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:192)
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>>
>>>>>>>> org.apache.cassandra.service.AbstractCassandraDaemon.activate(AbstractCassandraDaemon.java:314)
>>>>>>>>
>>>>>>>>     at
>>>>>>>>
>>>>>>>> org.apache.cassandra.thrift.CassandraDaemon.main(CassandraDaemon.java:79)
>>>>>>>>
>>>>>>>>
>>>>>>>> The seed node seems to somewhat acknowledge the non-seed
node:
>>>>>>>>
>>>>>>>> attempting to connect to /2.2.3.4
>>>>>>>>
>>>>>>>> attempting to connect to /10.170.190.31
>>>>>>>>
>>>>>>>> Can you suggest how can I fix it (I did see a few threads
on similar
>>>>>>>>
>>>>>>>> issue but did not really follow the chain)
>>>>>>>>
>>>>>>>> Thanks, AJ
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>> <cassec2regions.patch>
>>
>
>

Mime
View raw message