incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Julie <julie.su...@nextcentury.com>
Subject Re: Cassandra disk space utilization WAY higher than I would expect
Date Fri, 20 Aug 2010 20:45:19 GMT
Robert Coli <rcoli <at> digg.com> writes:

> Check the size of the Hinted Handoff CF? If your nodes are flapping
> under sustained write, they could be storing a non-trivial number of
> hinted handoff rows? Probably not 5x usage though..
> 
> http://wiki.apache.org/cassandra/Operations
> "
> The reason why you run nodetool cleanup on all live nodes [after
> replacing a node] is to remove old Hinted Handoff writes stored for
> the dead node. 
> =Rob


Please see below and let me know if you think hinted handoff is to blame.  

I do see some down/up activity according to the gossiping on the nodes.  
Interestingly, I see mass "deaths" being detected on servers 1, 2, 4, 5, and 
6.  Each server detects the "mass death" at a unique time, making it look as 
though it's the server that's detecting the mass death that is really the 
culprit.  Server3, the bloated node, is not having this problem.

As far as nodes being reported as going down and coming back up (always 
quickly) --  being down/up is being reported for each server as follows:
server1: 16 times
server2: 17 times
server3: 12 times
server4: 18 times
server5: 24 times
server6: 20 times
server7: 13 times
server8: 12 times

Again, server3 is looking like a very healthy node, so you wouldn't think it
would have a backlog of hinted handoffs coming its way when the writes complete.
Server5 and server6 seem to be the least healthy.

Here's the streaming that took place on all 8 nodes, following the 800,000 row
write (writes completed at 13:47):

server1-system.log: INFO [Thread-25] 2010-08-16 12:59:49,858
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-172-Data.db
server1-system.log: INFO [Thread-31] 2010-08-16 18:12:01,829 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-475-Data.db

server3-system.log: INFO [Thread-25] 2010-08-16 14:34:06,827 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-467-Data.db
server3-system.log: INFO [Thread-28] 2010-08-16 15:38:34,685 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-475-Data.db 
server3-system.log: INFO [Thread-34] 2010-08-16 17:24:57,584 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-477-Data.db
server3-system.log: INFO [Thread-31] 2010-08-16 17:26:37,281 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-476-Data.db

server4-system.log: INFO [Thread-25] 2010-08-16 18:30:19,313 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-472-Data.db
server4-system.log: INFO [Thread-28] 2010-08-16 18:33:07,141 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-473-Data.db

server6-system.log: INFO [Thread-25] 2010-08-16 16:53:43,108 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-470-Data.db
server6-system.log: INFO [Thread-28] 2010-08-16 17:58:15,031 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-471-Data.db

server7-system.log: INFO [Thread-25] 2010-08-16 12:39:54,342 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-110-Data.db
server7-system.log: INFO [Thread-28] 2010-08-16 12:46:16,067 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-144-Data.db
server7-system.log: INFO [Thread-31] 2010-08-16 14:51:24,585 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-469-Data.db
server7-system.log: INFO [Thread-34] 2010-08-16 15:17:22,168  
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-472-Data.db

server8-system.log: INFO [Thread-25] 2010-08-16 12:46:39,462 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-133-Data.db
server8-system.log: INFO [Thread-28] 2010-08-16 15:56:38,124 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-473-Data.db
server8-system.log: INFO [Thread-31] 2010-08-16 17:05:24,805 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-475-Data.db
server8-system.log: INFO [Thread-34] 2010-08-16 18:52:03,416 
 StreamCompletionHandler.java (line 64) 
 Streaming added /var/lib/cassandra/data/Keyspace1/Standard1-476-Data.db

Do you see any red flags?

Thanks for your help!



Mime
View raw message