incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Understanding index builds
Date Thu, 10 Mar 2011 01:40:30 GMT
https://issues.apache.org/jira/browse/CASSANDRA-2294
https://issues.apache.org/jira/browse/CASSANDRA-2295

On Wed, Mar 9, 2011 at 5:47 PM, Matt Kennedy <stinkymatt@gmail.com> wrote:
> I'm trying to gain some insight into what happens with a cluster when
> indexes are being built, or when CFs with indexed columns are being written
> to.
>
> Over the past couple of days we've been doing some loads into a CF with 29
> indexed columns.  Eventually, the nodes just got overwhelmed and the client
> (Hector) started getting timeouts.  We were using using a MapReduce job to
> load an HDFS file into Cassandra, though we had limited the load job to one
> task per node.  My confusion comes from how difficult it was to know that
> the nodes were becoming overwhelmed.  The ring consistently reported that
> all nodes were up and it did not appear that there were pending operations
> under tpstats.  I also monitor this cluster with Ganglia, and at no point
> did any of the machine loads appear very high at all, yet our job kept
> failing with Hector reporting timeouts.
>
> Today we decided to leave index creation until the end, and just load the
> data using the same Hector code.  We bumped up the hadoop concurrency to two
> concurrent tasks per node, and everything went fine, as expected, we've done
> much larger loads than this using Hadoop and as long as you don't shoot for
> too much concurrency, Cassandra can deal with it.  So now we have the data
> in the column family and I updated the column family metadata in the CLI to
> enable the 29 indexes.  As soon as I do that, the ring starts reporting that
> nodes are down intermittently, and HintedHandoffs are starting to accumulate
> under tpstats. Ganglia is reporting very low overall load, so I'm wondering
> why it's taking so long for cli and nodetool commands to return.
>
> I'm just trying to get a better handle on what kind of actions have a
> serious impact on cluster availability and to know the right places to look
> to try to get ahead of those conditions.
>
> Thanks for any insight you can provide,
> Matt
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message