cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Modha, Digant" <>
Subject cassandra repair error
Date Thu, 16 Jul 2015 12:45:52 GMT

We're using Cassandra 2.0.10 ( 2 DC, 3 Nodes each RF=3 for each DC).  During one of the weekly
repairs, we received the following error:

ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 (line 242) Failed creating
a merkle tree for [repair #d2178ba0-2902-11e5-bd95-f14c61d86b85 on dmds/curve_dates, (-1942303675502999131,-1890400428284965630]],
/  (see log for details)
ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 (line 199) Exception
in thread Thread[ValidationExecutor:1280,1,main]
FSWriteError in /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85
          at org.apache.cassandra.db.Directories.clearSnapshot(
          at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(
          at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(
          at org.apache.cassandra.db.compaction.CompactionManager.access$600(
          at org.apache.cassandra.db.compaction.CompactionManager$
          at java.util.concurrent.ThreadPoolExecutor.runWorker(
          at java.util.concurrent.ThreadPoolExecutor$
Caused by: java.nio.file.DirectoryNotEmptyException: /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85
          at sun.nio.fs.UnixFileSystemProvider.implDelete(
          at sun.nio.fs.AbstractFileSystemProvider.delete(
          at java.nio.file.Files.delete(
          ... 10 more
ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,993 (line 364) Stopping
WARN [ValidationExecutor:1280] 2015-07-12 22:18:10,993 (line 278) Stopping
gossip by operator request
INFO [ValidationExecutor:1280] 2015-07-12 22:18:10,993 (line 1279) Announcing
Has anybody seen this error?  The drives are local. Once this happened, the other node performing
 repair maxed out the CPU and cluster became unresponsive.


TD Securities disclaims any liability or losses either direct or consequential caused by the
use of this information. This communication is for informational purposes only and is not
intended as an offer or solicitation for the purchase or sale of any financial instrument
or as an official confirmation of any transaction. TD Securities is neither making any investment
recommendation nor providing any professional or advisory services relating to the activities
described herein. All market prices, data and other information are not warranted as to completeness
or accuracy and are subject to change without notice Any products described herein are (i)
not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured
depository institution and (iii) subject to investment risks, including possible loss of the
principal amount invested. The information shall not be further distributed or duplicated
in whole or in part by any means without the prior written consent of TD Securities. TD Securities
is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain
investment banking activities of The Toronto-Dominion Bank and its subsidiaries.

View raw message