cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <>
Subject Re: Easy way to overload a single node on purpose?
Date Fri, 17 Jun 2011 07:20:17 GMT
The short answer to the problem you saw is monitor the disk space. Also monitor client side
logs for errors. Running out of commit log space does not stop the node from doing reads,
so it can still be considered up. 

One nodes view of it's own UP'ness is not as important as the other nodes (or clients) view
of it.  For example...

A node will appear UP in the ring view of another node if it is participating in gossip messages
and it's application state is normal. But a node will appear UP in it's own view of the ring
most of time (assuming not bootstrap, leaving etc and it has joined the ring). This applies
even if it's gossip service has been disabled.

To a client a node will appear down if it is not responding to RPC requests. But it could
still be part of the cluster, appear UP to other nodes and be responding to read and/or write.

So to monitor that a node is running in some form you can...

- you should be monitoring the TP stats anyway, so you know the node is in some running state

- check that you can connect as a client to each node and do some simple call. Either read/write
or describe_ring() which will exec locallay or describe_schema_versions() which will call
all live nodes. A read/write will only verify that the node can act as a coordinator, not
that it can read/write it's self. 
- monitor the other nodes view of each node using nodetool ring. 

Now that i've written that I'm not 100% sold on it, but it will do for now :)

Aaron Morton
Freelance Cassandra Developer

On 17 Jun 2011, at 10:25, Suan Aik Yeo wrote:

> > Having a ping column can work if every key is replicated to every node. It would
tell you the cluster is working, sort of. Once the number of nodes is greater than the RF,
it tells you a subset of the nodes works.
> The way our check works is that each node checks itself, so in this context we're not
concerned about whether the cluster is "up", but that each individual node is "up".
> So the symptoms I saw, the node actually going "down" etc, were probably due to many
different events happening at the time, and will be very hard to recreate?
> On Thu, Jun 16, 2011 at 6:16 AM, aaron morton <> wrote:
> >     DEBUG 14:36:55,546 ... timed out
> Is logged when the coordinator times out waiting for the replicas to respond, the timeout
setting is rpc_timeout in the yaml file. This results in the client getting a TimedOutException.
> AFAIK There is no global everything is good / bad flags to check. e.g. AFAIK I node will
not mark its self down if it runs out of disk space.  So you need to monitor the free disk
space and alert on that.
> Having a ping column can work if every key is replicated to every node. It would tell
you the cluster is working, sort of. Once the number of nodes is greater than the RF, it tells
you a subset of the nodes works.
> If you google around you'll find discussions about monitoring with munin, ganglia, cloud
kick and Ops Centre.
> If you install mx4j you can access the JMX metrics via HTTP,
> Cheers
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> On 16 Jun 2011, at 10:38, Suan Aik Yeo wrote:
> > Here's a weird one... what's the best way to get a Cassandra node into a "half-crashed"
> >
> > We have a 3-node cluster running 0.7.5. A few days ago this happened organically
to node1 - the partition the commitlog was on was 100% full and there was a "No space left
on device" error, and after a while, although the cluster and node1 was still up, to the other
nodes it was down, and messages like:
> >     DEBUG 14:36:55,546 ... timed out
> > started to show up in its debug logs.
> >
> > We have a tool to indicate to the load balancer that a Cassandra node is down, but
it didn't detect it that time. Now I'm having trouble purposefully getting the node back to
that state, so that I can try other monitoring methods. I've tried to fill up the commitlog
partition with other files, and although I get the "No space left on device" error, the node
still doesn't go down and show the other symptoms it showed before.
> >
> > Also, if anyone could recommend a good way for a node itself to detect that its
in such a state I'd be interested in that too. Currently what we're doing is making a "describe_cluster_name()"
thrift call, but that still worked when the node was "down". I'm thinking of something like
reading/writing to a fixed value in a keyspace as a check... Unfortunately Java-based solutions
are out of the question.
> >
> >
> > Thanks,
> > Suan

View raw message