cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeff Jirsa <jeff.ji...@crowdstrike.com>
Subject Re: EC2 storage options for C*
Date Thu, 04 Feb 2016 00:34:28 GMT
I don’t want to be “that guy”, but there’s literally almost a dozen emails in this
thread answering exactly that question. Did you read the thread to which you replied? 


From:  James Rothering
Reply-To:  "user@cassandra.apache.org"
Date:  Wednesday, February 3, 2016 at 4:09 PM
To:  "user@cassandra.apache.org"
Subject:  Re: EC2 storage options for C*

Just curious here ... when did EBS become OK for C*? Didn't they always push towards using
ephemeral disks?

On Wed, Feb 3, 2016 at 12:17 PM, Ben Bromhead <ben@instaclustr.com> wrote:
For what it's worth we've tried d2 instances and they encourage terrible things like super
dense nodes (increases your replacement time). In terms of useable storage I would go with
gp2 EBS on a m4 based instance. 

On Mon, 1 Feb 2016 at 14:25 Jack Krupansky <jack.krupansky@gmail.com> wrote:
Ah, yes, the good old days of m1.large.

-- Jack Krupansky

On Mon, Feb 1, 2016 at 5:12 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
A lot of people use the old gen instances (m1 in particular) because they came with a ton
of effectively free ephemeral storage (up to 1.6TB). Whether or not they’re viable is a
decision for each user to make. They’re very, very commonly used for C*, though. At a time
when EBS was not sufficiently robust or reliable, a cluster of m1 instances was the de facto
standard. 

The canonical “best practice” in 2015 was i2. We believe we’ve made a compelling argument
to use m4 or c4 instead of i2. There exists a company we know currently testing d2 at scale,
though I’m not sure they have much in terms of concrete results at this time. 

- Jeff

From: Jack Krupansky
Reply-To: "user@cassandra.apache.org"
Date: Monday, February 1, 2016 at 1:55 PM 

To: "user@cassandra.apache.org"
Subject: Re: EC2 storage options for C*

Thanks. My typo - I referenced "C2 Dense Storage" which is really "D2 Dense Storage". 

The remaining question is whether any of the "Previous Generation Instances" should be publicly
recommended going forward.

And whether non-SSD instances should be recommended going forward as well. sure, technically,
someone could use the legacy instances, but the question is what we should be recommending
as best practice going forward.

Yeah, the i2 instances look like the sweet spot for any non-EBS clusters.

-- Jack Krupansky

On Mon, Feb 1, 2016 at 4:30 PM, Steve Robenalt <srobenalt@highwire.org> wrote:
Hi Jack, 

At the bottom of the instance-types page, there is a link to the previous generations, which
includes the older series (m1, m2, etc), many of which have HDD options. 

There are also the d2 (Dense Storage) instances in the current generation that include various
combos of local HDDs.

The i2 series has good sized SSDs available, and has the advanced networking option, which
is also useful for Cassandra. The enhanced networking is available with other instance types
as well, as you'll see on the feature list under each type. 

Steve



On Mon, Feb 1, 2016 at 1:17 PM, Jack Krupansky <jack.krupansky@gmail.com> wrote:
Thanks. Reading a little bit on AWS, and back to my SSD vs. magnetic question, it seems like
magnetic (HDD) is no longer a recommended storage option for databases on AWS. In particular,
only the C2 Dense Storage instances have local magnetic storage - all the other instance types
are SSD or EBS-only - and EBS Magnetic is only recommended for "Infrequent Data Access." 

For the record, that AWS doc has Cassandra listed as a use case for i2 instance types.

Also, the AWS doc lists EBS io2 for the NoSQL database use case and gp2 only for the "small
to medium databases" use case.

Do older instances with local HDD still exist on AWS (m1, m2, etc.)? Is the doc simply for
any newly started instances?

See:
https://aws.amazon.com/ec2/instance-types/
http://aws.amazon.com/ebs/details/


-- Jack Krupansky

On Mon, Feb 1, 2016 at 2:09 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
> My apologies if my questions are actually answered on the video or slides, I just did
a quick scan of the slide text.

Virtually all of them are covered.

> I'm curious where the EBS physical devices actually reside - are they in the same rack,
the same data center, same availability zone? I mean, people try to minimize network latency
between nodes, so how exactly is EBS able to avoid network latency?

Not published,and probably not a straight forward answer (probably have redundancy cross-az,
if it matches some of their other published behaviors). The promise they give you is ‘iops’,
with a certain block size. Some instance types are optimized for dedicated, ebs-only network
interfaces. Like most things in cassandra / cloud, the only way to know for sure is to test
it yourself and see if observed latency is acceptable (or trust our testing, if you assume
we’re sufficiently smart and honest). 

> Did your test use Amazon EBS–Optimized Instances?

We tested dozens of instance type/size combinations (literally). The best performance was
clearly with ebs-optimized instances that also have enhanced networking (c4, m4, etc) - slide
43

> SSD or magnetic or does it make any difference?

SSD, GP2 (slide 64)

> What info is available on EBS performance at peak times, when multiple AWS customers
have spikes of demand?

Not published, but experiments show that we can hit 10k iops all day every day with only trivial
noisy neighbor problems, not enough to impact a real cluster (slide 58)

> Is RAID much of a factor or help at all using EBS?

You can use RAID to get higher IOPS than you’d normally get by default (GP2 IOPS cap is
10k, which you get with a 3.333T volume – if you need more than 10k, you can stripe volumes
together up to the ebs network link max) (hinted at in slide 64)

> How exactly is EBS provisioned in terms of its own HA - I mean, with a properly configured
Cassandra cluster RF provides HA, so what is the equivalent for EBS? If I have RF=3, what
assurance is there that those three EBS volumes aren't all in the same physical rack?

There is HA, I’m not sure that AWS publishes specifics. Occasionally specific volumes will
have issues (hypervisor’s dedicated ethernet link to EBS network fails, for example). Occasionally
instances will have issues. The volume-specific issues seem to be less common than the instance-store
“instance retired” or “instance is running on degraded hardware” events. Stop/Start
and you’ve recovered (possible with EBS, not possible with instance store). The assurances
are in AWS’ SLA – if the SLA is insufficient (and it probably is insufficient), use more
than one AZ and/or AWS region or cloud vendor.

> For multi-data center operation, what configuration options assure that the EBS volumes
for each DC are truly physically separated?

It used to be true that EBS control plane for a given region spanned AZs. That’s no longer
true. AWS asserts that failure modes for each AZ are isolated (data may replicate between
AZs, but a full outage in us-east-1a shouldn’t affect running ebs volumes in us-east-1b
or us-east-1c). Slide 65

> In terms of syncing data for the commit log, if the OS call to sync an EBS volume returns,
is the commit log data absolutely 100% synced at the hardware level on the EBS end, such that
a power failure of the systems on which the EBS volumes reside will still guarantee availability
of the fsynced data. As well, is return from fsync an absolute guarantee of sstable durability
when Cassandra is about to delete the commit log, including when the two are on different
volumes? In practice, we would like some significant degree of pipelining of data, such as
during the full processing of flushing memtables, but for the fsync at the end a solid guarantee
is needed.

Most of the answers in this block are “probably not 100%, you should be writing to more
than one host/AZ/DC/vendor to protect your organization from failures”. AWS targets something
like 0.1% annual failure rate per volume and 99.999% availability (slide 66). We believe they’re
exceeding those goals (at least based with the petabytes of data we have on gp2 volumes).
 



From: Jack Krupansky
Reply-To: "user@cassandra.apache.org"
Date: Monday, February 1, 2016 at 5:51 AM 

To: "user@cassandra.apache.org"
Subject: Re: EC2 storage options for C*

I'm not a fan of guy - this appears to be the slideshare corresponding to the video: 
http://www.slideshare.net/AmazonWebServices/bdt323-amazon-ebs-cassandra-1-million-writes-per-second

My apologies if my questions are actually answered on the video or slides, I just did a quick
scan of the slide text.

I'm curious where the EBS physical devices actually reside - are they in the same rack, the
same data center, same availability zone? I mean, people try to minimize network latency between
nodes, so how exactly is EBS able to avoid network latency? 

Did your test use Amazon EBS–Optimized Instances?

SSD or magnetic or does it make any difference?

What info is available on EBS performance at peak times, when multiple AWS customers have
spikes of demand?

Is RAID much of a factor or help at all using EBS?

How exactly is EBS provisioned in terms of its own HA - I mean, with a properly configured
Cassandra cluster RF provides HA, so what is the equivalent for EBS? If I have RF=3, what
assurance is there that those three EBS volumes aren't all in the same physical rack?

For multi-data center operation, what configuration options assure that the EBS volumes for
each DC are truly physically separated?

In terms of syncing data for the commit log, if the OS call to sync an EBS volume returns,
is the commit log data absolutely 100% synced at the hardware level on the EBS end, such that
a power failure of the systems on which the EBS volumes reside will still guarantee availability
of the fsynced data. As well, is return from fsync an absolute guarantee of sstable durability
when Cassandra is about to delete the commit log, including when the two are on different
volumes? In practice, we would like some significant degree of pipelining of data, such as
during the full processing of flushing memtables, but for the fsync at the end a solid guarantee
is needed.


-- Jack Krupansky

On Mon, Feb 1, 2016 at 12:56 AM, Eric Plowe <eric.plowe@gmail.com> wrote:
Jeff, 

If EBS goes down, then EBS Gp2 will go down as well, no? I'm not discounting EBS, but prior
outages are worrisome. 


On Sunday, January 31, 2016, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
Free to choose what you'd like, but EBS outages were also addressed in that video (second
half, discussion by Dennis Opacki). 2016 EBS isn't the same as 2011 EBS. 

-- 
Jeff Jirsa


On Jan 31, 2016, at 8:27 PM, Eric Plowe <eric.plowe@gmail.com> wrote:

Thank you all for the suggestions. I'm torn between GP2 vs Ephemeral. GP2 after testing is
a viable contender for our workload. The only worry I have is EBS outages, which have happened.


On Sunday, January 31, 2016, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
Also in that video - it's long but worth watching

We tested up to 1M reads/second as well, blowing out page cache to ensure we weren't "just"
reading from memory



-- 
Jeff Jirsa


On Jan 31, 2016, at 9:52 AM, Jack Krupansky <jack.krupansky@gmail.com> wrote:

How about reads? Any differences between read-intensive and write-intensive workloads?

-- Jack Krupansky

On Sun, Jan 31, 2016 at 3:13 AM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
Hi John,

We run using 4T GP2 volumes, which guarantee 10k iops. Even at 1M writes per second on 60
nodes, we didn’t come close to hitting even 50% utilization (10k is more than enough for
most workloads). PIOPS is not necessary. 



From: John Wong
Reply-To: "user@cassandra.apache.org"
Date: Saturday, January 30, 2016 at 3:07 PM
To: "user@cassandra.apache.org"
Subject: Re: EC2 storage options for C*

For production I'd stick with ephemeral disks (aka instance storage) if you have running a
lot of transaction. 
However, for regular small testing/qa cluster, or something you know you want to reload often,
EBS is definitely good enough and we haven't had issues 99%. The 1% is kind of anomaly where
we have flush blocked.

But Jeff, kudo that you are able to use EBS. I didn't go through the video, do you actually
use PIOPS or just standard GP2 in your production cluster?

On Sat, Jan 30, 2016 at 1:28 PM, Bryan Cheng <bryan@blockcypher.com> wrote:
Yep, that motivated my question "Do you have any idea what kind of disk performance you need?".
If you need the performance, its hard to beat ephemeral SSD in RAID 0 on EC2, and its a solid,
battle tested configuration. If you don't, though, EBS GP2 will save a _lot_ of headache.

Personally, on small clusters like ours (12 nodes), we've found our choice of instance dictated
much more by the balance of price, CPU, and memory. We're using GP2 SSD and we find that for
our patterns the disk is rarely the bottleneck. YMMV, of course.

On Fri, Jan 29, 2016 at 7:32 PM, Jeff Jirsa <jeff.jirsa@crowdstrike.com> wrote:
If you have to ask that question, I strongly recommend m4 or c4 instances with GP2 EBS.  When
you don’t care about replacing a node because of an instance failure, go with i2+ephemerals.
Until then, GP2 EBS is capable of amazing things, and greatly simplifies life.

We gave a talk on this topic at both Cassandra Summit and AWS re:Invent: https://www.youtube.com/watch?v=1R-mgOcOSd4
It’s very much a viable option, despite any old documents online that say otherwise.



From: Eric Plowe
Reply-To: "user@cassandra.apache.org"
Date: Friday, January 29, 2016 at 4:33 PM
To: "user@cassandra.apache.org"
Subject: EC2 storage options for C*

My company is planning on rolling out a C* cluster in EC2. We are thinking about going with
ephemeral SSDs. The question is this: Should we put two in RAID 0 or just go with one? We
currently run a cluster in our data center with 2 250gig Samsung 850 EVO's in RAID 0 and we
are happy with the performance we are seeing thus far.

Thanks!

Eric








-- 
Steve Robenalt 
Software Architect
srobenalt@highwire.org 
(office/cell): 916-505-1785

HighWire Press, Inc.
425 Broadway St, Redwood City, CA 94063
www.highwire.org

Technology for Scholarly Communication


-- 
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692



Mime
View raw message