cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Modha, Digant" <Digant.Mo...@tdsecurities.com>
Subject RE: cassandra repair error
Date Thu, 16 Jul 2015 13:10:12 GMT
The hard drives have plenty of space left and working properly as far as I know.  We’ve been
running Cassandra for almost a year now. It’s the first time we’re seeing this problem.
 I see a JIRA for v2.1 but that is with NFS..  We’re not using NFS.

From: Carlos Rolo [mailto:rolo@pythian.com]
Sent: Thursday, July 16, 2015 8:59 AM
To: user@cassandra.apache.org
Subject: Re: cassandra repair error

Hello,
It seems that is a problem with Cassandra trying to delete a directory that is not empty.
First time seeing this error.
Are your harddrives with enough space and working correctly?

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: linkedin.com/in/carlosjuzarterolo<http://linkedin.com/in/carlosjuzarterolo>
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com<http://www.pythian.com/>

On Thu, Jul 16, 2015 at 2:45 PM, Modha, Digant <Digant.Modha@tdsecurities.com<mailto:Digant.Modha@tdsecurities.com>>
wrote:
Hi,

We’re using Cassandra 2.0.10 ( 2 DC, 3 Nodes each RF=3 for each DC).  During one of the
weekly repairs, we received the following error:

ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 Validator.java (line 242) Failed creating
a merkle tree for [repair #d2178ba0-2902-11e5-bd95-f14c61d86b85 on dmds/curve_dates, (-1942303675502999131,-1890400428284965630]],
/  (see log for details)
ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 CassandraDaemon.java (line 199) Exception
in thread Thread[ValidationExecutor:1280,1,main]
FSWriteError in /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85
          at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:122)
          at org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:384)
          at org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:488)
          at org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1877)
          at org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:811)
          at org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
          at org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:398)
          at java.util.concurrent.FutureTask.run(FutureTask.java:266)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.DirectoryNotEmptyException: /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85
          at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
          at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
          at java.nio.file.Files.delete(Files.java:1126)
          at org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:118)
          ... 10 more
ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,993 StorageService.java (line 364) Stopping
gossiper
WARN [ValidationExecutor:1280] 2015-07-12 22:18:10,993 StorageService.java (line 278) Stopping
gossip by operator request
INFO [ValidationExecutor:1280] 2015-07-12 22:18:10,993 Gossiper.java (line 1279) Announcing
shutdown
Has anybody seen this error?  The drives are local. Once this happened, the other node performing
 repair maxed out the CPU and cluster became unresponsive.


Thanks,
dm



TD Securities disclaims any liability or losses either direct or consequential caused by the
use of this information. This communication is for informational purposes only and is not
intended as an offer or solicitation for the purchase or sale of any financial instrument
or as an official confirmation of any transaction. TD Securities is neither making any investment
recommendation nor providing any professional or advisory services relating to the activities
described herein. All market prices, data and other information are not warranted as to completeness
or accuracy and are subject to change without notice Any products described herein are (i)
not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured
depository institution and (iii) subject to investment risks, including possible loss of the
principal amount invested. The information shall not be further distributed or duplicated
in whole or in part by any means without the prior written consent of TD Securities. TD Securities
is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain
investment banking activities of The Toronto-Dominion Bank and its subsidiaries.



--



TD Securities disclaims any liability or losses either direct or consequential caused by the
use of this information. This communication is for informational purposes only and is not
intended as an offer or solicitation for the purchase or sale of any financial instrument
or as an official confirmation of any transaction. TD Securities is neither making any investment
recommendation nor providing any professional or advisory services relating to the activities
described herein. All market prices, data and other information are not warranted as to completeness
or accuracy and are subject to change without notice Any products described herein are (i)
not insured by the FDIC, (ii) not a deposit or other obligation of, or guaranteed by, an insured
depository institution and (iii) subject to investment risks, including possible loss of the
principal amount invested. The information shall not be further distributed or duplicated
in whole or in part by any means without the prior written consent of TD Securities. TD Securities
is a trademark of The Toronto-Dominion Bank and represents TD Securities (USA) LLC and certain
investment banking activities of The Toronto-Dominion Bank and its subsidiaries.
Mime
View raw message