incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Igor <i...@4friends.od.ua>
Subject repair waiting for something
Date Sat, 17 Dec 2011 12:21:15 GMT
Hi,

10 nodes cassandra 1.0.3, several DC. weekly nodetool repair stuck for 
unusual long time for node 10.254.237.2.

output log on this node:
  INFO 11:19:42,045 Starting repair command #1, repairing 5 ranges.
  INFO 11:19:42,053 [repair #040aae00-28a1-11e1-0000-e378018944ff] new 
session: will sync _*localhost/10.254.237.2, /10.254.221.2, /10.253.2.2, 
/10.254.217.2, /10.254.94.2*_ on range 
(85070591730234615865843651857942052864,85070591730234615865843651857942052865] 
for meter.[eventschema, schema, ids, transaction]
  INFO 11:19:42,055 [repair #040aae00-28a1-11e1-0000-e378018944ff] 
requests for merkle tree sent for eventschema (to [/10.253.2.2, 
/10.254.221.2, localhost/10.254.237.2, /10.254.217.2, /10.254.94.2])
  INFO 11:19:42,063 Enqueuing flush of 
Memtable-eventschema@1509399856(18748/23435 serialized/live bytes, 4 ops)
  INFO 11:19:42,063 Writing Memtable-eventschema@1509399856(18748/23435 
serialized/live bytes, 4 ops)
  INFO 11:19:42,072 Completed flushing 
/spool1/cassandra/data/meter/eventschema-hb-40-Data.db (4745 bytes)
  INFO 11:19:42,073 Discarding obsolete commit 
log:CommitLogSegment(/var/lib/cassandra/commitlog/CommitLog-1324019623060.log)
  INFO 11:19:42,076 [repair #040aae00-28a1-11e1-0000-e378018944ff] 
Received merkle tree for eventschema from localhost/10.254.237.2
  INFO 11:19:42,102 [repair #040aae00-28a1-11e1-0000-e378018944ff] 
Received merkle tree for eventschema from /10.254.221.2
  INFO 11:19:42,128 [repair #040aae00-28a1-11e1-0000-e378018944ff] 
Received merkle tree for eventschema from /10.254.217.2
  INFO 11:19:42,228 [repair #040aae00-28a1-11e1-0000-e378018944ff] 
Received merkle tree for eventschema from /10.253.2.2

And nothing after that for long time. So node sent request for trees to 
other nodes and received all but from the 10.254.94.2_*

*_On that 10.254.94.2 node:
INFO 11:19:42,083 [repair #040aae00-28a1-11e1-0000-e378018944ff] Sending 
completed merkle tree to /10.254.237.2 for (meter,eventschema)

So merkle tree were lost somewhere. Will this waiting break somehow or I 
need to restart node?

Mime
View raw message