incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Frank Cooper <coop...@yahoo-inc.com>
Subject RE: Anybody experience one Cassandra server locking up?
Date Thu, 20 Aug 2009 01:51:48 GMT
Thanks for the detailed response. It is really helpful to understand what is going on behind
the covers. 

We are using "RandomPartitioner." However, I have noticed that some of the boxes have significantly
more data (in /var/cassandra/data and /var/cassandra/commitlog) than others (like 30 X more).
I have 50 client processes doing the read/write workload (and had 50 doing the load) but had
them round-robined between servers. E.g. the cassandra server to connect to was clientid %
6.

I have been reading and writing a single column family, so perhaps that's part of the issue.


Incidentally, the system is quite fun to play with, and the startup is very easy (just start
the nodes and they all find each other.) Writing the client (e.g. dealing with thrift) was
much harder. I wonder whether a lot of users had tried to write C++ clients; this was a little
non-trivial as the documentation I could find favors the java and PHP cases. 

(I didn't realize the FB folks had disconnected from the OSS project.)

brian
________________________________________
From: Jonathan Ellis [jbellis@gmail.com]
Sent: Wednesday, August 19, 2009 5:32 PM
To: cassandra-user@incubator.apache.org
Subject: Re: Anybody experience one Cassandra server locking up?

On Wed, Aug 19, 2009 at 5:19 PM, Brian Frank
Cooper<cooperb@yahoo-inc.com> wrote:
> We are trying to learn what we can about the performance of Cassandra. I hope to have
some results to share publicly in the next couple of weeks.
>
> The 0.4 version seems to have handled the insert load better, but is having trouble with
a 50/50 read/write workload. One server again has a busy core with the other 7 cores (and
the other servers) idle or near idle. Any ideas?

Writes are serialized per columnfamily.  There are some ways we can
improve that but right now you may need multiple CFs to max write
throughput.  (Reads are not serialized like that though so I am a
little surprised that the idleness difference is so complete.)

If only one server is getting all the load something is wrong.  Is
that the server all your clients are connecting to?  It's designed to
have the clients spread around the cluster.

Or, are you using OrderPreservingPartitioner?  Load balancing won't be
in until 0.5 so unless you manually pick your tokens carefully and/or
do writes in a non-sequential manner one server will get all the keys.
 Or just use RandomPartitioner (which of course means giving up range
queries).

> (Incidentally, we are reading and writing 10 KB records; does the large data size have
any impact?)

Since unlike a traditional K/V store you can update and retrieve
individual columns separately, most column sizes are for < 1KB but
10KB isn't totally unreasonable.

> I was hoping to chat with some of you Cassandra folks when we visited FB last week...perhaps
we can grab coffee sometime and chat about these issues...

The FB guys haven't been involved with the OSS project for some time,
unfortunately.

-Jonathan

Mime
View raw message