Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.161.43 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:date:message-id:subject:from:to:content-type;
        b=qS9NQGCpjZyGnT9zuZayYKREysVsYrZbS1EiXEPVTJq7nRPt3V49/AGOFsS9UP26gT
         3/8j46eutbCJ+pRjzX2zdlLT6AF8CuHx3HoygQYoW4nru4b4DCq6TpAd6ib8ud0I9shy
         BhSA3XY/qwfCeFMkzbvb3cFoa449U2iHCPoRQ=
MIME-Version: 1.0
Date: Tue, 7 Dec 2010 21:25:02 -0500
Message-ID: <AANLkTimdNrvCSbRTP+gwKi=tieXiV5ZWbJOc1m5n_ehX@mail.gmail.com>
Subject: Running multiple instances on a single server --micrandra ??
From: Edward Capriolo <edlinuxguru@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I am quite ready to be stoned for this thread but I have been thinking
about this for a while and I just wanted to bounce these ideas of some
guru's.

Cassandra does allow multiple data directories, but as far as I can
tell no one runs in this configuration. This is something that is very
different between the hbase architecture and the Cassandra
architecture. HBase borrows the concept from hadoop of JBOD
configurations. HBase has many small ish (~256 MB) regions managed
with Zookeeper. Cassandra has a few (1 per node) large node sized
Token Ranges managed by Gossip consensus.

Lets say a node has 6 300 GB disks. You have the options of RAID5,
RAID6, RAID10, or RAID0. The problem I have found with these
configurations are major compactions (of even large minor ones) can
take a long time. Even if your disk is not heavily utilized this is a
lot of data to move through. Thus node joins take a long time. Node
moves take a long time.

The idea behind "micrandra" is for a 6 disk system run 6 instances of
Cassandra, one per disk. Use the RackAwareSnitch to make sure no
replicas live on the same node.

The downsides
1) we would have to manage 6x the instances of cassandra
2) we would have some overhead for each JVM.

The upsides ?
1) Since disk/instance failure only degrades the overall performance
1/6th (RAID0 you lost the entire node) (RAID5 still takes a hit when
down a disk)
2) Moves and joins have less work to do
3) Can scale up a single node by adding a single disk to an existing
system (assuming the ram and cpu is light)
4) OPP would be "easier" to balance out hot spots (maybe not on this
one in not an OPP)

What does everyone thing? Does it ever make sense to run this way?