cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: C* in multiple AWS AZ's
Date Tue, 03 Jul 2018 11:12:47 GMT
Hi,

Nice that you solved the issue. I had some thoughts while reading:


> My original thought was a new DC parallel to the current, and then
> decommission the other DC.
>

I also think this is the best way to go when possible. It can be reverted
any time in the process, respect distributions, you can switch app per app,
and some other advantages.

Also, I am a bit late on this topic, but using this:

AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel..
> I'm wanting to move the current US-EAST from AZ 1a to 1e.


You could have kept Ec2Snitch and use the 'dc_suffix' option in
'cassandra-rackdc.properties'
file to create a non-conflicting logical datacenter in the same region. For
example with a 'dc_suffix' set to '-WHATEVER' the new data center created
would have been called 'US-EAST-WHATEVER' :-).

I know all docs say use ec2multiregion for multi-DC.
>

Using EC2RegionSnitch brought nothing but troubles to me in the past. You
have to open security groups for both public and private IPs, the number of
rules is limited (or was for me when I did it) and after some headaches, we
did the same, a tunnel between VPCs and used ec2_snitch.

Yet, now I would keep GPFS if that works fully for you now. It will even
allow you to hybrid the cluster with non AWS hardware or transition out of
AWS if you need at some point in the future.

As a side note, using i3, you might have to use a recent operating system
(such as Ubuntu 16.04) to have the latest drivers for NVMe. NVMe support in
Ubuntu 14.04 AMI is not reliable. It might be absent or lead to data
corruption. Make sure the OS in use works well with this hardware.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html

C*heers,
-----------------------
Alain Rodriguez - @arodream - alain@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-06-29 16:35 GMT+01:00 Pradeep Chhetri <pradeep@stashaway.com>:

> Ohh i see now. It makes sense. Thanks a lot.
>
> On Fri, Jun 29, 2018 at 9:17 PM, Randy Lynn <rlynn@getavail.com> wrote:
>
>> data is only lost if you stop the node. between restarts the storage is
>> fine.
>>
>> On Fri, Jun 29, 2018 at 10:39 AM, Pradeep Chhetri <pradeep@stashaway.com>
>> wrote:
>>
>>> Isnt NVMe storage an instance storage ie. the data will be lost in case
>>> the instance restarts. How are you going to make sure that there is no data
>>> loss in case instance gets rebooted?
>>>
>>> On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn <rlynn@getavail.com> wrote:
>>>
>>>> GPFS - Rahul FTW! Thank you for your help!
>>>>
>>>> Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did
>>>> not have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS
>>>> so I intrinsically know we'll get better throughput.
>>>>
>>>> On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh <
>>>> rahul.xavier.singh@gmail.com> wrote:
>>>>
>>>>> Totally agree. GPFS for the win. EC2 multi region snitch is an
>>>>> automation tool like Ansible or Puppet. Unless you have two orders of
>>>>> magnitude more servers than you do now, you don’t need it.
>>>>>
>>>>> Rahul
>>>>> On Jun 29, 2018, 6:18 AM -0400, kurt greaves <kurt@instaclustr.com>,
>>>>> wrote:
>>>>>
>>>>> Yes. You would just end up with a rack named differently to the AZ.
>>>>> This is not a problem as racks are just logical. I would recommend
>>>>> migrating all your DCs to GPFS though for consistency.
>>>>>
>>>>> On Fri., 29 Jun. 2018, 09:04 Randy Lynn, <rlynn@getavail.com> wrote:
>>>>>
>>>>>> So we have two data centers already running..
>>>>>>
>>>>>> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site
>>>>>> tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e..
>>>>>> I know all docs say use ec2multiregion for multi-DC.
>>>>>>
>>>>>> I like the GPFS idea. would that work with the multi-DC too?
>>>>>> What's the downside? status would report rack of 1a, even though
in
>>>>>> 1e?
>>>>>>
>>>>>> Thanks in advance for the help/thoughts!!
>>>>>>
>>>>>>
>>>>>> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves <kurt@instaclustr.com>
>>>>>> wrote:
>>>>>>
>>>>>>> There is a need for a repair with both DCs as rebuild will not
>>>>>>> stream all replicas, so unless you can guarantee you were perfectly
>>>>>>> consistent at time of rebuild you'll want to do a repair after
rebuild.
>>>>>>>
>>>>>>> On another note you could just replace the nodes but use GPFS
>>>>>>> instead of EC2 snitch, using the same rack name.
>>>>>>>
>>>>>>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, <
>>>>>>> rahul.xavier.singh@gmail.com> wrote:
>>>>>>>
>>>>>>>> Parallel load is the best approach and then switch your Data
access
>>>>>>>> code to only access the new hardware. After you verify that
there are no
>>>>>>>> local read / writes on the OLD dc and that the updates are
only via Gossip,
>>>>>>>> then go ahead and change the replication factor on the key
space to have
>>>>>>>> zero replicas in the old DC. Then you can decommissioned.
>>>>>>>>
>>>>>>>> This way you are hundred percent sure that you aren’t missing
any
>>>>>>>> new data. No need for a DC to DC repair but a repair is always
healthy.
>>>>>>>>
>>>>>>>> Rahul
>>>>>>>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn <rlynn@getavail.com>,
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Already running with Ec2.
>>>>>>>>
>>>>>>>> My original thought was a new DC parallel to the current,
and then
>>>>>>>> decommission the other DC.
>>>>>>>>
>>>>>>>> Also my data load is small right now.. I know small is relative
>>>>>>>> term.. each node is carrying about 6GB..
>>>>>>>>
>>>>>>>> So given the data size, would you go with parallel DC or
let the
>>>>>>>> new AZ carry a heavy load until the others are migrated over?
>>>>>>>> and then I think "repair" to cleanup the replications?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
>>>>>>>> rahul.xavier.singh@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> You don’t have to use EC2 snitch on AWS but if you
have already
>>>>>>>>> started with it , it may put a node in a different DC.
>>>>>>>>>
>>>>>>>>> If your data density won’t be ridiculous You could
add 3 to
>>>>>>>>> different DC/ Region and then sync up. After the new
DC is operational you
>>>>>>>>> can remove one at a time on the old DC and at the same
time add to the new
>>>>>>>>> one.
>>>>>>>>>
>>>>>>>>> Rahul
>>>>>>>>> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn <rlynn@getavail.com>,
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I have a 6-node cluster I'm migrating to the new i3 types.
>>>>>>>>> But at the same time I want to migrate to a different
AZ.
>>>>>>>>>
>>>>>>>>> What happens if I do the "running node replace method"
with 1 node
>>>>>>>>> at a time moving to the new AZ. Meaning, I'll have temporarily;
>>>>>>>>>
>>>>>>>>> 5 nodes in AZ 1c
>>>>>>>>> 1 new node in AZ 1e.
>>>>>>>>>
>>>>>>>>> I'll wash-rinse-repeat till all 6 are on the new machine
type and
>>>>>>>>> in the new AZ.
>>>>>>>>>
>>>>>>>>> Any thoughts about whether this gets weird with the Ec2Snitch
and
>>>>>>>>> a RF 3?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Randy Lynn
>>>>>>>>> rlynn@getavail.com
>>>>>>>>>
>>>>>>>>> office:
>>>>>>>>> 859.963.1616 <+1-859-963-1616> ext 202
>>>>>>>>> 163 East Main Street - Lexington, KY 40507 - USA
>>>>>>>>> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g>
>>>>>>>>>
>>>>>>>>> <https://www.getavail.com/> getavail.com
>>>>>>>>> <https://www.getavail.com/>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Randy Lynn
>>>>>>>> rlynn@getavail.com
>>>>>>>>
>>>>>>>> office:
>>>>>>>> 859.963.1616 <+1-859-963-1616> ext 202
>>>>>>>> 163 East Main Street - Lexington, KY 40507 - USA
>>>>>>>> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g>
>>>>>>>>
>>>>>>>> <https://www.getavail.com/> getavail.com
>>>>>>>> <https://www.getavail.com/>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Randy Lynn
>>>>>> rlynn@getavail.com
>>>>>>
>>>>>> office:
>>>>>> 859.963.1616 <+1-859-963-1616> ext 202
>>>>>> 163 East Main Street - Lexington, KY 40507 - USA
>>>>>> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g>
>>>>>>
>>>>>> <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Randy Lynn
>>>> rlynn@getavail.com
>>>>
>>>> office:
>>>> 859.963.1616 <+1-859-963-1616> ext 202
>>>> 163 East Main Street - Lexington, KY 40507 - USA
>>>> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g>
>>>>
>>>> <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
>>>>
>>>
>>
>>
>> --
>> Randy Lynn
>> rlynn@getavail.com
>>
>> office:
>> 859.963.1616 <+1-859-963-1616> ext 202
>> 163 East Main Street - Lexington, KY 40507 - USA
>> <https://maps.google.com/?q=163+East+Main+Street+-+Lexington,+KY+40507+-+USA&entry=gmail&source=g>
>>
>> <https://www.getavail.com/> getavail.com <https://www.getavail.com/>
>>
>
>

Mime
View raw message