Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CC833200BA1 for ; Mon, 17 Oct 2016 21:01:50 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CB00F160AEC; Mon, 17 Oct 2016 19:01:50 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id C3ADF160AE2 for ; Mon, 17 Oct 2016 21:01:49 +0200 (CEST) Received: (qmail 88383 invoked by uid 500); 17 Oct 2016 19:01:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 88372 invoked by uid 99); 17 Oct 2016 19:01:47 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 17 Oct 2016 19:01:47 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 1FD0AC0B20 for ; Mon, 17 Oct 2016 19:01:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.898 X-Spam-Level: * X-Spam-Status: No, score=1.898 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=winguzone.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Ns01oZtfFHkx for ; Mon, 17 Oct 2016 19:01:44 +0000 (UTC) Received: from sender163-mail.zoho.com (sender163-mail.zoho.com [74.201.84.163]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id F23495F30E for ; Mon, 17 Oct 2016 19:01:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1476730900; s=zoho; d=winguzone.com; i=vladyu@winguzone.com; h=Date:From:To:Message-Id:In-Reply-To:References:Subject:MIME-Version:Content-Type; l=11643; bh=cOTwORMwsPdjr5/UFYy415xsvqwbOimNI1H08nfLuWk=; b=TiWUVIksX4FQwrtPqXvuoNlJcuA8gRCE6rGqBoJ8XyqYaiqTxwroTTzwtCToQ620 aC3f8MP8E2wo4dcvtGruolbEHjmcfUkBpM2XfGz1AmbCsNS+J5oNFAmf4w++FdNGbEE Fz92vReT+Qeqrm3JKqvdrVIxBQSbpxre+vB3kbOk= Received: from mail.zoho.com by mx.zohomail.com with SMTP id 1476730900070315.15455965856665; Mon, 17 Oct 2016 12:01:40 -0700 (PDT) Date: Mon, 17 Oct 2016 15:01:40 -0400 From: Vladimir Yudovin To: "user" Message-Id: <157d4054e58.be159532437851.2039085735232966883@winguzone.com> In-Reply-To: References: <157d35a4996.111bf0c3c419737.893769067162731191@winguzone.com> <157d3f92405.afdb1151436704.4323668014528777643@winguzone.com> Subject: Re: Adding disk capacity to a running node MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_1360365_2098078878.1476730900066" X-Priority: Medium User-Agent: Zoho Mail X-Mailer: Zoho Mail archived-at: Mon, 17 Oct 2016 19:01:51 -0000 ------=_Part_1360365_2098078878.1476730900066 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit But after such restart node should be joined to cluster again and restore data, right? Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra on Azure and SoftLayer. Launch your cluster in minutes. ---- On Mon, 17 Oct 2016 14:55:49 -0400Jonathan Haddad <jon@jonhaddad.com> wrote ---- Vladimir, *Most* people are running Cassandra are doing so using ephemeral disks. Instances are not arbitrarily moved to different hosts. Yes, instances can be shut down, but that's why you distribute across AZs. On Mon, Oct 17, 2016 at 11:48 AM Vladimir Yudovin <vladyu@winguzone.com> wrote: It's extremely unreliable to use ephemeral (local) disks. Even if you don't stop instance by yourself, it can be restarted on different server in case of some hardware failure or AWS initiated update. So all node data will be lost. Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra on Azure and SoftLayer. Launch your cluster in minutes. ---- On Mon, 17 Oct 2016 14:45:00 -0400Seth Edwards <seth@pubnub.com> wrote ---- These are i2.2xlarge instances so the disks currently configured as ephemeral dedicated disks. On Mon, Oct 17, 2016 at 11:34 AM, Laing, Michael <michael.laing@nytimes.com> wrote: You could just expand the size of your ebs volume and extend the file system. No data is lost - assuming you are running Linux. On Monday, October 17, 2016, Seth Edwards <seth@pubnub.com> wrote: We're running 2.0.16. We're migrating to a new data model but we've had an unexpected increase in write traffic that has caused us some capacity issues when we encounter compactions. Our old data model is on STCS. We'd like to add another ebs volume (we're on aws) to our JBOD config and hopefully avoid any situation where we run out of disk space during a large compaction. It appears that the behavior we are hoping to get is actually undesirable and removed in 3.2. It still might be an option for us until we can finish the migration. I'm not familiar with LVM so it may be a bit risky to try at this point. On Mon, Oct 17, 2016 at 9:42 AM, Yabin Meng <yabinmeng@gmail.com> wrote: I assume you're talking about Cassandra JBOD (just a bunch of disk) setup because you do mention it as adding it to the list of data directories. If this is the case, you may run into issues, depending on your C* version. Check this out: http://www.datastax.com/dev/blog/improving-jbod. Or another approach is to use LVM to manage multiple devices into a single mount point. If you do so, from what Cassandra can see is just simply increased disk storage space and there should should have no problem. Hope this helps, Yabin On Mon, Oct 17, 2016 at 11:54 AM, Vladimir Yudovin <vladyu@winguzone.com> wrote: Yes, Cassandra should keep percent of disk usage equal for all disk. Compaction process and SSTable flushes will use new disk to distribute both new and existing data. Best regards, Vladimir Yudovin, Winguzone - Hosted Cloud Cassandra on Azure and SoftLayer. Launch your cluster in minutes. ---- On Mon, 17 Oct 2016 11:43:27 -0400Seth Edwards <seth@pubnub.com> wrote ---- We have a few nodes that are running out of disk capacity at the moment and instead of adding more nodes to the cluster, we would like to add another disk to the server and add it to the list of data directories. My question, is, will Cassandra use the new disk for compactions on sstables that already exist in the primary directory? Thanks! ------=_Part_1360365_2098078878.1476730900066 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable =
But after such restart node should be joined to cluster = again and restore data, right?

Be= st regards, Vladimir Yudovin,
Winguzone - Hosted Cloud Cassand= ra on Azure and SoftLayer.
Launch your cluster in minutes.
=


---- On Mon, 17 Oct 2016 14:55:49 -0400Jonathan Haddad <jon@jo= nhaddad.com> wrote ----

Vladimir,

*Most* p= eople are running Cassandra are doing so using ephemeral disks.  Insta= nces are not arbitrarily moved to different hosts.  Yes, instances can= be shut down, but that's why you distribute across AZs.  

On Mon, Oct 17, 2016 at 11:48 AM V= ladimir Yudovin <vladyu@winguzone.com> wrote:

It's extremely unreliable to use ephemeral (local) disks= . Even if you don't stop instance by yourself, it can be restarted on diffe= rent server in case of some hardware failure or AWS initiated update. So al= l node data will be lost.

<= /div>
Best regards, Vladimir Yudovin,
Winguzone - Hoste= d Cloud Cassandra on Azure and SoftLayer.
Launch your cluster in minutes= .


---- On Mon, 17 Oct 2016 14:45:00 -0400Seth Edwards <= ;seth@pubnub.com&g= t; wrote ----
<= div>
These are= i2.2xlarge instances so the disks currently configured as ephemeral dedica= ted disks. 

On Mon, Oct 17, 201= 6 at 11:34 AM, Laing, Michael <michael.laing@nytimes.com> wrote:=

You could just expand the size of your ebs volume and extend the fi= le system. No data is lost - assuming you are running Linux.
=


On Monday, October 17,= 2016, Seth Edwards <seth@pubnub.com> wrote:
We're running 2.0.16. We're migrating to a new data mode= l but we've had an unexpected increase in write traffic that has caused us = some capacity issues when we encounter compactions. Our old data model is o= n STCS. We'd like to add another ebs volume (we're on aws) to our JBOD conf= ig and hopefully avoid any situation where we run out of disk space during = a large compaction. It appears that the behavior we are hoping to get is ac= tually undesirable and removed in 3.2. It still might be an option for us u= ntil we can finish the migration. 

I'm no= t familiar with LVM so it may be a bit risky to try at this point. 

On Mon, Oct 17, 2016 at 9:42 AM,= Yabin Meng <yabinmeng@gmail.com> wrote:
I as= sume you're talking about Cassandra JBOD (just a bunch of disk) setup becau= se you do mention it as adding it to the list of data directories. If this = is the case, you may run into issues, depending on your C* version. Check t= his out: http://www.datastax.com/dev/blog/improving-jbod.

Or another approach is to use LVM to manage multipl= e devices into a single mount point. If you do so, from what Cassandra can = see is just simply increased disk storage space and there should should hav= e no problem.

Hope this helps,
<= br>
Yabin

= On Mon, Oct 17, 2016 at 11:54 AM, Vladimir Yudovin <vladyu@winguzone.com> wrote:

Yes, Cassandra s= hould keep percent of disk usage equal for all disk. Compaction process and= SSTable flushes will use new disk to distribute both new and existing data= .

Best regards, Vladimir Yudovin,
= Winguzone - Hosted Cloud Cassandra on Azure and SoftLayer.
Launch yo= ur cluster in minutes.


---- On Mon, 17 Oct 2016 11:43:27 -0400Seth Edwards <seth@pubnub.com> wrote ----

= We have a few nodes that are running out of disk capacity at the moment and= instead of adding more nodes to the cluster, we would like to add another = disk to the server and add it to the list of data directories. My question,= is, will Cassandra use the new disk for compactions on sstables that alrea= dy exist in the primary directory? 


<= /div>

Thanks!
<= br>


------=_Part_1360365_2098078878.1476730900066--