cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Jeske <>
Subject cassandra vs hbase summary (was facebook messaging)
Date Mon, 22 Nov 2010 18:01:59 GMT
I havn't used either Cassandra or hbase, so please don't take any part of
this message as me attempting to state facts about either system. However,
I'm very familiar with data-storage design details, and I've worked
extensively optimizing applications running on MySQL, Oracle, berkeledb
(including distributed txn berkeleydb), and Google Bigtable.

The recent discussion triggered by Facebook messaging using HBase helped
surface many interesting design differences in the two systems. I'm writing
this message both to summarize what I've read in a few different places
about that topic, and to check my facts.

As far as I can descern, this is a decent summary of the consistency and
performance differences between hbase and cassandra (N3/R2/W2 or N3/R1/W3)
for an hbase acceptable workload.. (Please correct the fact if they appear

*1) Cassandra can't replicate the consistency situation of HBase.* Namely,
that when a write requiring a quorum fails it will never appear. Deriving
from this explanation:

[In Cassandra]Provided at least one node receives the write, it will
eventually be written to all replicas. A failure to meet the requested
ConsistencyLevel is just that; not a failure to write the data itself. Once
the write is received by a node, it will eventually reach all replicas,
there is no roll back. - Nick Telford

[In Hbase] The DFSClient call returns when all datanodes in the pipeline
have flushed (to the OS buffer) and ack'ed. That code comes from HDFS-200 in
the 0.20-append branch and HDFS-265 for all branches after 0.20, meaning
that it's in 0.21.0 - Jean-Daniel Cryans

in HBase, if a write is accepted by only 1 of 3 HDFS replicas; and the
region master never receives a response from the other two replicas; and it
fails the client write, that write should never appear. Even if the region
master then fails, when a new region master is elected, and it restarts and
recovers, it should read HDFS blocks and accept the consensus 2/3 opinion
that the log does not contain the write -- dropping the write. The write
will never be seen.

In Cassandra, if a write (requesting 2 or 3 copies) is accepted by only one
node, that write will fail to the client. Future reads R=1 will see that
write or not depending on whether they contact the one server that accepted
or not, until the data is propagated, at which time they will see the write.
Reads R=2 will not see the write until it is propagated until at least two
servers. There is no mechanism to assure that a write is either accepted by
the requested number of servers or aborted.

*2) Cassandra has a less efficient memory footprint data pinned in memory
(or cached).* With 3 replicas on Cassandra, each element of data pinned
in-memory is kept in memory on 3 servers, wheras in hbase only region
masters keep the data in memory, so there is only one-copy of each data

*3) Cassandra (N3/W2/R2) has slower reads of cached or pinned-in-memory
data.* HBase can answer a read-only query that is in memory from the single
region-master, while Cassandra (N3/W2/R2) must read from multiple servers.
 (note, N3/W2/R2 still doesn't produce the same consistency situation as
hbase, see #1)

*4) Cassandra (N3/W3/R1) takes longer to allow data to become writable again
in the face of a node-failure than HBase/HDFS.* Cassandra must repair the
keyrange to bring N from 2 to 3 to resume allowing writes with W=3. HDFS can
still acheive a 2 node quorum in the face of a node failure. (note, using
N3/W2 requires R2, see #3) (note, this still doesn't produce the same
consistency situation as hbase, see #1.)

*5) HBase can't match the row-availability situation of Cassandra(N3/W2/R2
or N3/W3/R1).* In the face of a single machine failure, if it is a region
master, those keys are offline in HBase until a new region master is elected
and brought online. In Cassandra, no single node failure causes the data to
become unavailable.

Is that summary correct? Am I missing any points? Did I get any facts wrong?

Note, I'm NOT attempting to advocate the following changes, but simply
understand the design differences....

>From my uninformed view, it seems that #1 causes the biggest cascade of
differences, affecting both #3 and #4. If Cassandra were allowed to do what
HBase/HDFS does, namely to specify a repair-consistency requirement, then
Cassandra (N3/W2/R2/Repair2) should be the same consistency guarantee as
HBase. Further, if Cassandra were allowed to elect one of the copies of data
as 'master'. then it could require the master participate in all quorum
writes, allowing reads to be consistent when conducted only through the
master. This could be a road to address 2/3/4. To my eyes, this would make
Cassandra capable of operating in a mode which is essentially equivilent to
HBase/HDFS. Do those facts seem correct?  --- AGAIN, I'm not advocating
Cassandra make these changes, I'm simply trying to understand the
differences, by considering what changes it would take to make them the

Thanks for all your great work on Cassandra!

View raw message