cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Seth Edwards <s...@pubnub.com>
Subject Re: Adding disk capacity to a running node
Date Mon, 17 Oct 2016 21:06:49 GMT
Thanks for the detailed steps Ben! This gives me another option in case of
emergency.



On Mon, Oct 17, 2016 at 1:55 PM, Ben Bromhead <ben@instaclustr.com> wrote:

> yup you would need to copy the files across to the new volume from the dir
> you wanted to give additional space to. Rough steps would look like:
>
>    1. Create EBS volume (make it big... like 3TB)
>    2. Attach to instance
>    3. Mount/format EBS volume
>    4. Stop C*
>    5. Copy full/troublesome directory to the EBS volume
>    6. Remove copied files (using rsync for the copy / remove step can be
>    a good idea)
>    7. bind mount EBS volume with the same path as the troublesome
>    directory
>    8. Start C* back up
>    9. Let it finish compacting / streaming etc
>    10. Stop C*
>    11. remove bind mount
>    12. copy files back to ephemeral
>    13. start C* back up
>    14. repeat on other nodes
>    15. run repair
>
> You can use this process if you somehow end up in a full disk situation.
> If you end up in a low disk situation you'll have other issues (like
> corrupt / half written SSTable components), but it's better than nothing
>
> Also to maintain your read throughput during this whole thing, double
> check the EBS volumes read_ahead_kb setting on the block volume and reduce
> it to something sane like 0 or 16.
>
>
>
> On Mon, 17 Oct 2016 at 13:42 Seth Edwards <seth@pubnub.com> wrote:
>
>> @Ben
>>
>> Interesting idea, is this also an option for situations where the disk is
>> completely full and Cassandra has stopped? (Not that I want to go there).
>>
>> If this was the route taken, and we did
>>
>> mount --bind   /mnt/path/to/large/sstable   /mnt/newebs
>>
>> We would still need to do some manual copying of files? such as
>>
>> mv /mnt/path/to/large/sstable.sd /mnt/newebs ?
>>
>> Thanks!
>>
>> On Mon, Oct 17, 2016 at 12:59 PM, Ben Bromhead <ben@instaclustr.com>
>> wrote:
>>
>> Yup as everyone has mentioned ephemeral are fine if you run in multiple
>> AZs... which is pretty much mandatory for any production deployment in AWS
>> (and other cloud providers) . i2.2xls are generally your best bet for high
>> read throughput applications on AWS.
>>
>> Also on AWS ephemeral storage will generally survive a user initiated
>> restart. For the times that AWS retires an instance, you get plenty of
>> notice and it's generally pretty rare. We run over 1000 instances on AWS
>> and see one forced retirement a month if that. We've never had an instance
>> pulled from under our feet without warning.
>>
>> To add another option for the original question, one thing you can do is
>> to attach a large EBS drive to the instance and bind mount it to the
>> directory for the table that has the very large SSTables. You will need to
>> copy data across to the EBS volume. Let everything compact and then copy
>> everything back and detach EBS. Latency may be higher than normal on the
>> node you are doing this on (especially if you are used to i2.2xl
>> performance).
>>
>> This is something we often have to do, when we encounter pathological
>> compaction situations associated with bootstrapping, adding new DCs or STCS
>> with a dominant table or people ignore high disk usage warnings :)
>>
>> On Mon, 17 Oct 2016 at 12:43 Jeff Jirsa <jeff.jirsa@crowdstrike.com>
>> wrote:
>>
>> Ephemeral is fine, you just need to have enough replicas (in enough AZs
>> and enough regions) to tolerate instances being terminated.
>>
>>
>>
>>
>>
>>
>>
>> *From: *Vladimir Yudovin <vladyu@winguzone.com>
>> *Reply-To: *"user@cassandra.apache.org" <user@cassandra.apache.org>
>> *Date: *Monday, October 17, 2016 at 11:48 AM
>> *To: *user <user@cassandra.apache.org>
>>
>>
>> *Subject: *Re: Adding disk capacity to a running node
>>
>>
>>
>> It's extremely unreliable to use ephemeral (local) disks. Even if you
>> don't stop instance by yourself, it can be restarted on different server in
>> case of some hardware failure or AWS initiated update. So all node data
>> will be lost.
>>
>>
>>
>> Best regards, Vladimir Yudovin,
>>
>>
>> *Winguzone
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com-3Ffrom-3Dlist&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=ixOxpX-xpw1dJZNpaMT3mepToWX8gzmsVaXFizQLzoU&s=4q7P9fddEYpXwPR-h9yA_tk5JwR8l6c7cKJ-LQTVcGM&e=>
>> - Hosted Cloud Cassandra on Azure and SoftLayer.Launch your cluster in
>> minutes.*
>>
>>
>>
>>
>>
>> ---- On Mon, 17 Oct 2016 14:45:00 -0400*Seth Edwards <seth@pubnub.com
>> <seth@pubnub.com>>* wrote ----
>>
>>
>>
>> These are i2.2xlarge instances so the disks currently configured as
>> ephemeral dedicated disks.
>>
>>
>>
>> On Mon, Oct 17, 2016 at 11:34 AM, Laing, Michael <
>> michael.laing@nytimes.com> wrote:
>>
>>
>>
>> You could just expand the size of your ebs volume and extend the file
>> system. No data is lost - assuming you are running Linux.
>>
>>
>>
>>
>>
>> On Monday, October 17, 2016, Seth Edwards <seth@pubnub.com> wrote:
>>
>> We're running 2.0.16. We're migrating to a new data model but we've had
>> an unexpected increase in write traffic that has caused us some capacity
>> issues when we encounter compactions. Our old data model is on STCS. We'd
>> like to add another ebs volume (we're on aws) to our JBOD config and
>> hopefully avoid any situation where we run out of disk space during a large
>> compaction. It appears that the behavior we are hoping to get is actually
>> undesirable and removed in 3.2. It still might be an option for us until we
>> can finish the migration.
>>
>>
>>
>> I'm not familiar with LVM so it may be a bit risky to try at this point.
>>
>>
>>
>> On Mon, Oct 17, 2016 at 9:42 AM, Yabin Meng <yabinmeng@gmail.com> wrote:
>>
>> I assume you're talking about Cassandra JBOD (just a bunch of disk) setup
>> because you do mention it as adding it to the list of data directories. If
>> this is the case, you may run into issues, depending on your C* version.
>> Check this out: http://www.datastax.com/dev/blog/improving-jbod
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.datastax.com_dev_blog_improving-2Djbod&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=ixOxpX-xpw1dJZNpaMT3mepToWX8gzmsVaXFizQLzoU&s=e_rkkJ8RHJXe4KvyNfeRWQkdy-zZzOnaMDQle3nN808&e=>
>> .
>>
>>
>>
>> Or another approach is to use LVM to manage multiple devices into a
>> single mount point. If you do so, from what Cassandra can see is just
>> simply increased disk storage space and there should should have no problem.
>>
>>
>>
>> Hope this helps,
>>
>>
>>
>> Yabin
>>
>>
>>
>> On Mon, Oct 17, 2016 at 11:54 AM, Vladimir Yudovin <vladyu@winguzone.com>
>> wrote:
>>
>>
>>
>> Yes, Cassandra should keep percent of disk usage equal for all disk.
>> Compaction process and SSTable flushes will use new disk to distribute both
>> new and existing data.
>>
>>
>>
>> Best regards, Vladimir Yudovin,
>>
>>
>> *Winguzone
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__winguzone.com-3Ffrom-3Dlist&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=ixOxpX-xpw1dJZNpaMT3mepToWX8gzmsVaXFizQLzoU&s=4q7P9fddEYpXwPR-h9yA_tk5JwR8l6c7cKJ-LQTVcGM&e=>
>> - Hosted Cloud Cassandra on Azure and SoftLayer.Launch your cluster in
>> minutes.*
>>
>>
>>
>>
>>
>> ---- On Mon, 17 Oct 2016 11:43:27 -0400*Seth Edwards <seth@pubnub.com
>> <seth@pubnub.com>>* wrote ----
>>
>>
>>
>> We have a few nodes that are running out of disk capacity at the moment
>> and instead of adding more nodes to the cluster, we would like to add
>> another disk to the server and add it to the list of data directories. My
>> question, is, will Cassandra use the new disk for compactions on sstables
>> that already exist in the primary directory?
>>
>>
>>
>>
>>
>>
>>
>> Thanks!
>>
>>
>>
>>
>>
>>
>> ____________________________________________________________________
>> CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential
>> and may be legally privileged. If you are not the intended recipient, do
>> not disclose, copy, distribute, or use this email or any attachments. If
>> you have received this in error please let the sender know and then delete
>> the email and all attachments.
>>
>> --
>> Ben Bromhead
>> CTO | Instaclustr <https://www.instaclustr.com/>
>> +1 650 284 9692
>> Managed Cassandra / Spark on AWS, Azure and Softlayer
>>
>>
>> --
> Ben Bromhead
> CTO | Instaclustr <https://www.instaclustr.com/>
> +1 650 284 9692
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>

Mime
View raw message