cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Sargisson <>
Subject Node forgets about most of its column families
Date Thu, 23 Aug 2012 18:47:38 GMT
Hi all,
I was wondering if anybody had seen the following behaviour before and 
how we might detect it and keep the application running.

We have a 6 node cluster. It seems that one of these nodes forgot about 
all but one of the application column families - possibly after a 
restart. Then, when our application connects using Hector, it can't find 
any data so gives back an exception.

I'm currently running nodetool repair on one of the *other* nodes which 
is taking a very long time to complete. (35mins and counting a load of <9MB)

The  logs from the failing node say:
  INFO [MemoryMeter:1] 2012-08-23 14:59:14,807 (line 213) 
CFS(Keyspace='system', ColumnFamily='HintsColumnFamily') liveRatio is
1.1219167666485013 (just-counted was 1.0).  calculation took 28ms for 
1252 columns
  INFO [main] 2012-08-23 14:59:14,949 (line 272) 
Finished reading /var/lib/cassandra/commitlog/CommitLog-22654969122258
  INFO [main] 2012-08-23 14:59:14,950 (line 103) 
Skipped 8216 mutations from unknown (probably removed) CF with id 1016
  INFO [main] 2012-08-23 14:59:14,950 (line 103) 
Skipped 3013 mutations from unknown (probably removed) CF with id 1017

... and so on.

Hector is saying:
InvalidRequestException(why:unconfigured columnfamily user_conversations)

Thanks for any comments or advice,


Edward Sargisson

senior java developer
Global Relay <>

New York | Chicago | Vancouver | London (+44.0800.032.9829) | Singapore 

Global Relay Archive supports email, instant messaging, BlackBerry, 
Bloomberg, Thomson Reuters, Pivot, YellowJacket, LinkedIn, Twitter, 
Facebook and more.

Ask about *Global Relay Message* 
<>*--- *The Future of 
Collaboration in the Financial Services World

*All email sent to or from this address will be retained by Global 
Relay's email archiving system. This message is intended only for the 
use of the individual or entity to which it is addressed, and may 
contain information that is privileged, confidential, and exempt from 
disclosure under applicable law.  Global Relay will not be liable for 
any compliance or technical information provided herein. All trademarks 
are the property of their respective owners.

View raw message