incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ondřej Černoš <cern...@gmail.com>
Subject Re: Repair Freeze / Gossip Invisibility / EC2 Public IP configuration
Date Thu, 25 Apr 2013 09:08:34 GMT
Hi,

I have similar issue  with stuck repair. Similar multiregion setup, only
between us-east and private cloud at rackspace. The log mentiones merkle
tree exchanges and I see a lot of dropped communication:

I will comment on your ticket in Jira.

regards,

ondrej cernos


On Fri, Apr 19, 2013 at 4:50 AM, Arya Goudarzi <goudarzi@gmail.com> wrote:

> We don't use default ports. Woops! Now I advertised mine. I did try
> disabling internode compression for all in cassandra.yaml but still it did
> not work. I have to open the insecure storage port to public ips.
>
>
> On Tue, Apr 16, 2013 at 4:59 PM, Edward Capriolo <edlinuxguru@gmail.com>wrote:
>
>> So cassandra does inter node compression. I have not checked but this
>> might be accidentally getting turned on by default. Because the storage
>> port is typically 7000. Not sure why you are allowing 7100. In any case try
>> allowing 7000 or with internode compression off.
>>
>>
>> On Tue, Apr 16, 2013 at 6:42 PM, Arya Goudarzi <goudarzi@gmail.com>wrote:
>>
>>> TL;DR; An EC2 Multi-Region Setup's Repair/Gossip Works with 1.1.10 but
>>> with 1.2.4, gossip does not see the nodes after restarting all nodes at
>>> once, and repair gets stuck.
>>>
>>> This is a working configuration:
>>> Cassandra 1.1.10 Cluster with 12 nodes in us-east-1 and 12 nodes in
>>> us-west-2
>>> Using Ec2MultiRegionSnitch and SSL enabled for DC_ONLY and
>>> NetworkTopologyStrategy with strategy_options: us-east-1:3;us-west-2:3;
>>> C* instances have a security group called 'cluster1'
>>> security group 'cluster1' in each region is configured as such
>>> Allow TCP:
>>> 7199 from cluster1 (JMX)
>>> 1024 - 65535 from cluster1 (JMX Random Ports - This supersedes all
>>> specific ports, but I have the specific ports just for clarity )
>>> 7100 from cluster1 (Configured Normal Storage)
>>> 7103 from cluster1 (Configured SSL Storage)
>>> 9160 from cluster1 (Configured Thrift RPC Port)
>>> 9160 from <client_group>
>>> foreach node's public IP we also have this rule set to enable cross
>>> region comminication:
>>> 7103 from public_ip (Open SSL storage)
>>>
>>> The above is a functioning and happy setup. You run repair, and it
>>> finishes successfully.
>>>
>>> Broken Setup:
>>>
>>> Upgrade to 1.2.4 without changing any of the above security group
>>> settings:
>>>
>>> Run repair. The repair will get stuck. Thus hanging.
>>>
>>> Now for each public_ip add a security group rule as such to cluster1
>>> security group:
>>>
>>> Allow TCP: 7100 from public_ip
>>>
>>> Run repair. Things will work now. Also after restarting all nodes at the
>>> same time, gossip will see everyone again.
>>>
>>> I was told on https://issues.apache.org/jira/browse/CASSANDRA-5432 that
>>> nothing in terms of networking was changed. If nothing in terms of port and
>>> networking was changed in 1.2, then why the above is happening? I can
>>> constantly reproduce it.
>>>
>>> Please advice.
>>>
>>> -Arya
>>>
>>>
>>
>

Mime
View raw message