Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 71458200C7E for ; Tue, 23 May 2017 20:15:05 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6FAE9160BC3; Tue, 23 May 2017 18:15:05 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 91024160BA4 for ; Tue, 23 May 2017 20:15:03 +0200 (CEST) Received: (qmail 32966 invoked by uid 500); 23 May 2017 18:15:01 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 32956 invoked by uid 99); 23 May 2017 18:15:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 May 2017 18:15:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 76132C0DEC for ; Tue, 23 May 2017 18:15:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.296 X-Spam-Level: X-Spam-Status: No, score=-0.296 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-2.796, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=scylladb-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id A45drTUcyHmy for ; Tue, 23 May 2017 18:15:00 +0000 (UTC) Received: from mail-qk0-f176.google.com (mail-qk0-f176.google.com [209.85.220.176]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id D83E45F613 for ; Tue, 23 May 2017 18:14:59 +0000 (UTC) Received: by mail-qk0-f176.google.com with SMTP id u75so136629942qka.3 for ; Tue, 23 May 2017 11:14:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=scylladb-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=JQ1dzg+XFuIxRa7vSm/IBl83N0Ti4q3VPuO5hjkZivc=; b=qTj1tM7qVQ4oCu4n7OlQdmYBXFG7lf3braAl0jOUv4U2hVwgUcZW7JwXdXfPW+/AL4 EF2z3MdDnS48J7iGhEVXdUZcQsXo3Q7rvHXJLaN23ucEvgHkVJFmbB3BsHdMzPDYQCRJ yh1MjermExpK5HxZz1+csZRhWF6QxTR6NPrpZLNXjDt/AjiTLPN0pzPbj9MdkxRVfm4p We+XmT+qsX4OzbNrTOq38rxb4ZqurORbNVbQG/vGb+8CReF8f3UnzXLGCHBvcScKiUL2 auddHvlUUudBapgYCazoxxHAHWk7N6XEMedH9DNnzWSpT/zV0OluXU9aX8s81qshY9yc 8asQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=JQ1dzg+XFuIxRa7vSm/IBl83N0Ti4q3VPuO5hjkZivc=; b=jTz2Vkf+cEGgAaPTxKdlSOF5v/+FZkFRxRxrY7qrblGudzKAq/bik7rT/1ChKygKtW N7o5Ztw9oPwYeBx8hrlK9RLuI2tUbcJExKTEEu+k7tWwEFciiz7E2ZIGOngJpPuoRwXP lttwzEU20vAP3A6z0Fxuu4WNX226YGigZX8JDFHdaslR1Gh8ne4EwpQOL2YI0bk9NwPM lDBeyZj/pGEm1TtEprDkTHjicKZ6aVVfAxqhlqZbPdBWsNfqDXVOMidPZUdcJJsQkqAG xhy9SbrcRkYodem8qGXG2xVCgucj8JgU3l2c76UR2GL2KHGLH+II0enwxRqw8qE64byu bdRg== X-Gm-Message-State: AODbwcBjtlBF+yMOCt0Fbdw/IrV6dhISMVgS4/ZmbU6x+tGDNAQ8kHqy 3FvkYHbuVJ4CiKzWUbzW0Z3pF2a+EywH X-Received: by 10.55.164.149 with SMTP id n143mr29203731qke.205.1495563299402; Tue, 23 May 2017 11:14:59 -0700 (PDT) MIME-Version: 1.0 Received: by 10.237.45.3 with HTTP; Tue, 23 May 2017 11:14:18 -0700 (PDT) In-Reply-To: References: <3BAE6835-DB96-4243-B44A-C06430436DCB@aspect.com> <18D00A98-3E75-44F2-870D-1A6D69E90F58@aspect.com> From: Dor Laor Date: Tue, 23 May 2017 11:14:18 -0700 Message-ID: Subject: Re: EC2 instance recommendations To: Jonathan Haddad Cc: "Gopal, Dhruva" , "user@cassandra.apache.org" Content-Type: multipart/alternative; boundary="001a114fa4f83355c9055034f7e8" archived-at: Tue, 23 May 2017 18:15:05 -0000 --001a114fa4f83355c9055034f7e8 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Note that EBS durability isn't perfect, you cannot rely on them entirely: https://aws.amazon.com/ebs/details/ "Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume. This makes EBS volumes 20 times more reliable than typical commodity disk drives" It's rare but once in awhile we get an email notification that one of our EBS drives was corrupted. In this case a combined write through cache and EBS is certainly better than EBS w/o backup. On Tue, May 23, 2017 at 10:40 AM, Jonathan Haddad wrote= : > Another option that I like the idea of but never see used unfortunately i= s > using ZFS, with EBS for storage and the SSD ephemeral drive as L2Arc. > You'd get the performance of ephemeral storage with all the features of > EBS. Something to consider. > > On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva > wrote: > >> Thanks! So, I assume that as long we make sure we never explicitly >> =E2=80=9Cshutdown=E2=80=9D the instance, we are good. Are you also sayin= g we won=E2=80=99t be able >> to snapshot a directory with ephemeral storage and that is why EBS is >> better? We=E2=80=99re just finding that to get a reasonable amount of IO= PS (gp2) >> out of EBS at a reasonable rate, it gets more expensive than an I3. >> >> >> >> *From: *Jonathan Haddad >> >> >> *Date: *Tuesday, May 23, 2017 at 9:42 AM >> *To: *"Gopal, Dhruva" , Matija Gobec < >> matija0204@gmail.com>, Bhuvan Rawal >> >> *Cc: *"user@cassandra.apache.org" >> >> >> *Subject: *Re: EC2 instance recommendations >> >> >> >> > Oh, so all the data is lost if the instance is shutdown or restarted >> (for that instance)? >> >> >> >> When you restart the OS, you're technically not shutting down the >> instance. As long as the instance isn't stopped / terminated, your data= is >> fine. I ran my databases on ephemeral storage for years without issue. = In >> general, ephemeral storage is going to give you lower latency since ther= e's >> no network overhead. EBS is generally cheaper than ephemeral, is >> persistent, and you can take snapshots easily. >> >> >> >> On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva >> wrote: >> >> Oh, so all the data is lost if the instance is shutdown or restarted (fo= r >> that instance)? If we take a na=C3=AFve approach to backing up the direc= tory, >> and restoring it, if we ever have to bring down the instance and back up= , >> will that work as a strategy? Data is only kept around for 2 days and is >> TTL=E2=80=99d after. >> >> >> >> *From: *Matija Gobec >> *Date: *Tuesday, May 23, 2017 at 8:15 AM >> *To: *Bhuvan Rawal >> *Cc: *"Gopal, Dhruva" , " >> user@cassandra.apache.org" >> *Subject: *Re: EC2 instance recommendations >> >> >> >> We are running on I3s since they came out. NVMe SSDs are really fast and >> I managed to push them to 75k IOPs. >> >> As Bhuvan mentioned the i3 storage is ephemeral. If you can work around >> it and plan for failure recovery you are good to go. >> >> >> >> I ran Cassandra on m4s before and had no problems with EBS volumes (gp2) >> even in low latency use cases. With the cost of M4 instances and EBS >> volumes that make sense in IOPs, I would recommend going with more i3s a= nd >> working around the ephemeral issue (if its an issue). >> >> >> >> Best, >> >> Matija >> >> On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal >> wrote: >> >> i3 instances will undoubtedly give you more meat for buck - easily 40K+ >> iops whereas on the other hand EBS maxes out at 20K PIOPS which is highl= y >> expensive (at times they can cost you significantly more than cost of >> instance). >> >> But they have ephemeral local storage and data is lost once instance is >> stopped, you need to be prudent in case of i series, it is generally use= d >> for large persistent caches. >> >> >> >> Regards, >> >> Bhuvan >> >> On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva >> wrote: >> >> Hi =E2=80=93 >> >> We=E2=80=99ve been running M4.2xlarge EC2 instances with 2-3 TB of sto= rage and >> have been comparing this to I-3.2xlarge, which seems more cost effective >> when dealing with this amount of storage and from an IOPS perspective. D= oes >> anyone have any recommendations/ on the I-3s and how it performs overall= , >> compared to the M4 equivalent? On the surface, without us having taken i= t >> through its paces performance-wise, it does seem to be pretty powerful. = We >> just ran through an exercise with a RAIDed 200 TB volume (as opposed to = a >> non RAIDed 3 TB volume) and were seeing a 20-30% improvement with the >> RAIDed setup, on a 6 node Cassandra ring. Just looking for any >> feedback/experience folks may have had with the I-3s. >> >> >> >> Regards, >> >> *DHRUVA GOPAL* >> >> *sr. MANAGER, ENGINEERING* >> >> *REPORTING, ANALYTICS AND BIG DATA* >> >> *+1 408.325.2011 <+1%20408-325-2011>* *WORK* >> >> *+1 408.219.1094 <+1%20408-219-1094>* *MOBILE* >> >> *UNITED STATES* >> >> *dhruva.gopal@aspect.com * >> >> *aspect.com * >> >> [image: cription: http://webapp2.aspect.com/EmailSigLogo-rev.jpg] >> >> >> >> This email (including any attachments) is proprietary to Aspect Software= , >> Inc. and may contain information that is confidential. If you have recei= ved >> this message in error, please do not read, copy or forward this message. >> Please notify the sender immediately, delete it from your system and >> destroy any copies. You may not further disclose or distribute this emai= l >> or its attachments. >> >> This email (including any attachments) is proprietary to Aspect Software= , >> Inc. and may contain information that is confidential. If you have recei= ved >> this message in error, please do not read, copy or forward this message. >> Please notify the sender immediately, delete it from your system and >> destroy any copies. You may not further disclose or distribute this emai= l >> or its attachments. >> >> This email (including any attachments) is proprietary to Aspect Software= , >> Inc. and may contain information that is confidential. If you have recei= ved >> this message in error, please do not read, copy or forward this message. >> Please notify the sender immediately, delete it from your system and >> destroy any copies. You may not further disclose or distribute this emai= l >> or its attachments. >> > --001a114fa4f83355c9055034f7e8 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Note that EBS durability isn't perfect, you cannot rel= y on them entirely:
"Amazon EBS volumes are designed for an annual f= ailure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complet= e or partial loss of the volume, depending on the size and performance of t= he volume. This makes EBS volumes 20 times more reliable than typical commo= dity disk drives"
It's rare but once in awhile we get an email notification that one = of our EBS drives was corrupted.=C2=A0
In this case a combined write through cache and EBS is = certainly better than EBS w/o backup.



On Tue, May 23, 2017 at 10:40 AM, J= onathan Haddad <jon@jonhaddad.com> wrote:
Another option that I like the idea of but= never see used unfortunately is using ZFS, with EBS for storage and the SS= D ephemeral drive as L2Arc.=C2=A0 You'd get the performance of ephemera= l storage with all the features of EBS.=C2=A0 Something to consider.
<= div class=3D"HOEnZb">

On Tue, May 23, 2017 at 10:30 AM Gopal, Dhruva <Dhruva.Gopal@aspect.com> wrote:


Date: Tuesday, May 23, 2017 at 9:42 AM
To: "Gopal, Dhruva" <Dhruva.Gopal@Aspect.com>, Matij= a Gobec <matij= a0204@gmail.com>, Bhuvan Rawal <bhu1rawal@gmail.com>


Subject: Re: EC2 instance recommendations

=C2=A0

>=C2=A0Oh, so all the data is lost= if the instance is shutdown or restarted (for that instance)?=C2=A0

=C2=A0

When you restart the OS, you're t= echnically not shutting down the instance.=C2=A0 As long as the instance is= n't stopped / terminated, your data is fine.=C2=A0 I ran my databases on ephemeral storage for years without issue.=C2=A0 In general, = ephemeral storage is going to give you lower latency since there's no n= etwork overhead.=C2=A0 EBS is generally cheaper than ephemeral, is persiste= nt, and you can take snapshots easily.

=C2=A0

=

On Tue, May 23, 2017 at 9:35 AM Gopal, Dhruva <Dhruva.Gopal@asp= ect.com> wrote:

Oh, so all the data is lost if the instance is shut= down or restarted (for that instance)? If we take a na=C3=AFve approach to backing up the directory, and restoring it, if we ever have to bring do= wn the instance and back up, will that work as a strategy? Data is only kep= t around for 2 days and is TTL=E2=80=99d after.

=C2=A0

From: Matija Gobec <matija0204@gmail.com>
Date: Tuesday, May 23, 2017 at 8:15 AM
To: Bhuvan Rawal <bhu1rawal@gmail.com>
Cc: "Gopal, Dhruva" <Dhruva.Gopal@Aspect.com>, "= ;user@cassan= dra.apache.org" <user@cassandra.apache.org>
Subject: Re: EC2 instance recommendations

=C2=A0

We are running on I3s since they came out. NVMe SSDs= are really fast and I managed to push them to 75k IOPs.

As Bhuvan mentioned the i3 storage is ephemeral. If = you can work around it and plan for failure recovery you are good to go.=

=C2=A0

I ran Cassandra on m4s before and had no problems wi= th EBS volumes (gp2) even in low latency use cases. With the cost of M4 ins= tances and EBS volumes that make sense in IOPs, I would recommend going with more i3s and working around the ephemeral issue= (if its an issue).

=C2=A0

Best,

Matija

On Tue, May 23, 2017 at 2:13 AM, Bhuvan Rawal <bhu1rawal@gmail.com<= /a>> wrote:

i3 instances will undoubtedly give you more meat for= buck - easily 40K+ iops whereas on the other hand EBS maxes out at 20K PIO= PS which is highly expensive (at times they can cost you significantly more than cost of instance).=C2=A0

But they have ephemeral local storage and data is lo= st once instance is stopped, you need to be prudent in case of i series, it= is generally used for large persistent caches.

=C2=A0

Regards,

Bhuvan

On Tue, May 23, 2017 at 4:55 AM, Gopal, Dhruva <<= a href=3D"mailto:Dhruva.Gopal@aspect.com" target=3D"_blank">Dhruva.Gopal@as= pect.com> wrote:

Hi =E2=80=93=

=C2=A0 We=E2=80=99v= e been running M4.2xlarge EC2 instances with 2-3 TB of storage and have bee= n comparing this to I-3.2xlarge, which seems more cost effective when dealing with this amount of storage and from an IOPS perspective. Doe= s anyone have any recommendations/ on the I-3s and how it performs overall,= compared to the M4 equivalent? On the surface, without us having taken it = through its paces performance-wise, it does seem to be pretty powerful. We just ran through an exercise with a= RAIDed 200 TB volume (as opposed to a non RAIDed 3 TB volume) and were see= ing a 20-30% improvement with the RAIDed setup, on a 6 node Cassandra ring.= Just looking for any feedback/experience folks may have had with the I-3s.

=C2=A0<= /span>

Regards= ,

DHRUVA GOPAL

sr. MANAGER, ENGINEERING=

REPORTING, ANALYTICS AND BIG DATA

+1 408.325.2011=C2=A0WORK<= /span>=C2=A0=

+1 408.219.1094=C2=A0MOBILE<= /span>=C2=A0=

UNITED STATES<= /u>

dhruva.gopal@aspect.com=C2=A0

aspect.com

3D"c=<= /u>

=C2=A0

This email (including any attachments) is proprietar= y to Aspect Software, Inc. and may contain information that is confidential= . If you have received this message in error, please do not read, copy or forward this message. Please notify the sender immedi= ately, delete it from your system and destroy any copies. You may not furth= er disclose or distribute this email or its attachments.

This email (including any attachments) is proprietar= y to Aspect Software, Inc. and may contain information that is confidential= . If you have received this message in error, please do not read, copy or f= orward this message. Please notify the sender immediately, delete it from your system and destroy any copies.= You may not further disclose or distribute this email or its attachments.

This email (including any attachments) is proprietary to Aspect Software, I= nc. and may contain information that is confidential. If you have received = this message in error, please do not read, copy or forward this message. Pl= ease notify the sender immediately, delete it from your system and destroy any copies. You may not further dis= close or distribute this email or its attachments.

--001a114fa4f83355c9055034f7e8--