cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Molinaro <>
Subject Re: Effective allocation of multiple disks
Date Thu, 11 Mar 2010 03:31:36 GMT
Except major compactions are not that rare if you have a cluster which you
need to add capacity to.  Anytime to add nodes with bootstrap it is recommended
you run cleanup on nodes which you removed data from (and this is useful to
see how much space you are now using).  Cleanup does a major compaction and
if you happen to have one disk larger than others, most of the data ends
up there (this happened to me when I added some ebs's to some ec2 nodes, I
distributed the sstables and everything was cool, I had more io, things were
great, then I need to add another node, did a cleanup, boom everything is
on one disk and io sucks again).  I also don't quite know what happens when
a major compaction occurs which would combine sstables and fill up the
largest disk?

However after discussion I completely understand why things were done this
way, it's difficut to manage the space and really it should be relegated to
the disk subsystem of the OS (ie, RAID0, JBOD, LVM, etc).  I would almost
recommend just keeping things simple and removing multiple data directories
from the config altogether and just documenting that you should plan on using
OS level mechanisms for growing diskspace and io.


On Wed, Mar 10, 2010 at 04:43:36PM -0600, Stu Hood wrote:
> Yea, I suppose major compactions are the wildcard here. Nonetheless, the situation where
you only have 1 SSTable should be very rare.
> I'll open a ticket though, because we really ought to be able to utilize those disks
more thoroughly, and I have some ideas there.
> -----Original Message-----
> From: "Anthony Molinaro" <>
> Sent: Wednesday, March 10, 2010 3:38pm
> To:
> Subject: Re: Effective allocation of multiple disks
> This is incorrect, as discussed a few weeks ago.  I have a setup with multiple
> disks, and as soon as compaction occurs all the data ends up on one disk.  If
> you need the additional io, you will want raid0.  But simply listing multiple
> DataFileDirectories will not work.
> -Anthony
> On Wed, Mar 10, 2010 at 02:08:13AM -0600, Stu Hood wrote:
> > You can list multiple DataFileDirectories, and Cassandra will scatter files across
all of them. Use 1 disk for the commitlog, and 3 disks for data directories.
> > 
> > See
> > 
> > Thanks,
> > Stu
> > 
> > -----Original Message-----
> > From: "Eric Rosenberry" <>
> > Sent: Wednesday, March 10, 2010 2:00am
> > To:
> > Subject: Effective allocation of multiple disks
> > 
> > Based on the documentation, it is clear that with Cassandra you want to have
> > one disk for commitlog, and one disk for data.
> > 
> > My question is: If you think your workload is going to require more io
> > performance to the data disks than a single disk can handle, how would you
> > recommend effectively utilizing additional disks?
> > 
> > It would seem a number of vendors sell 1U boxes with four 3.5 inch disks.
> >  If we use one for commitlog, is there a way to have Cassandra itself
> > equally split data across the three remaining disks?  Or is this something
> > that needs to be handled by the hardware level, or operating system/file
> > system level?
> > 
> > Options include a hardware RAID controller in a RAID 0 stripe (this is more
> > $$$ and for what gain?), or utilizing a volume manager like LVM.
> > 
> > Along those same lines, if you do implement some type of striping, what RAID
> > stripe size is recommended?  (I think Todd Burruss asked this earlier but I
> > did not see a response)
> > 
> > Thanks for any input!
> > 
> > -Eric
> > 
> > 
> -- 
> ------------------------------------------------------------------------
> Anthony Molinaro                           <>

Anthony Molinaro                           <>

View raw message