cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Carlos Rolo <r...@pythian.com>
Subject Re: cassandra repair error
Date Thu, 16 Jul 2015 12:59:23 GMT
Hello,

It seems that is a problem with Cassandra trying to delete a directory that
is not empty. First time seeing this error.
Are your harddrives with enough space and working correctly?

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
<http://linkedin.com/in/carlosjuzarterolo>*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Thu, Jul 16, 2015 at 2:45 PM, Modha, Digant <
Digant.Modha@tdsecurities.com> wrote:

> Hi,
>
>
>
> We’re using Cassandra 2.0.10 ( 2 DC, 3 Nodes each RF=3 for each DC).
> During one of the weekly repairs, we received the following error:
>
>
>
> ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992 Validator.java
> (line 242) Failed creating a merkle tree for [repair
> #d2178ba0-2902-11e5-bd95-f14c61d86b85 on dmds/curve_dates,
> (-1942303675502999131,-1890400428284965630]], /  (see log for details)
>
> ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,992
> CassandraDaemon.java (line 199) Exception in thread
> Thread[ValidationExecutor:1280,1,main]
>
> FSWriteError in
> /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85
>
>           at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:122)
>
>           at
> org.apache.cassandra.io.util.FileUtils.deleteRecursive(FileUtils.java:384)
>
>           at
> org.apache.cassandra.db.Directories.clearSnapshot(Directories.java:488)
>
>           at
> org.apache.cassandra.db.ColumnFamilyStore.clearSnapshot(ColumnFamilyStore.java:1877)
>
>           at
> org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:811)
>
>           at
> org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:63)
>
>           at
> org.apache.cassandra.db.compaction.CompactionManager$8.call(CompactionManager.java:398)
>
>           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
>           at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>           at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>
>           at java.lang.Thread.run(Thread.java:745)
>
> Caused by: java.nio.file.DirectoryNotEmptyException:
> /apps/data/cassandra/dmds/data/dmds/curve_dates/snapshots/d2178ba0-2902-11e5-bd95-f14c61d86b85
>
>           at
> sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
>
>           at
> sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103)
>
>           at java.nio.file.Files.delete(Files.java:1126)
>
>           at
> org.apache.cassandra.io.util.FileUtils.deleteWithConfirm(FileUtils.java:118)
>
>           ... 10 more
>
> ERROR [ValidationExecutor:1280] 2015-07-12 22:18:10,993
> StorageService.java (line 364) Stopping gossiper
>
> WARN [ValidationExecutor:1280] 2015-07-12 22:18:10,993 StorageService.java
> (line 278) Stopping gossip by operator request
>
> INFO [ValidationExecutor:1280] 2015-07-12 22:18:10,993 Gossiper.java (line
> 1279) Announcing shutdown
>
> Has anybody seen this error?  The drives are local. Once this happened,
> the other node performing  repair maxed out the CPU and cluster became
> unresponsive.
>
>
>
>
>
> Thanks,
>
> dm
>
>
>
>
> TD Securities disclaims any liability or losses either direct or
> consequential caused by the use of this information. This communication is
> for informational purposes only and is not intended as an offer or
> solicitation for the purchase or sale of any financial instrument or as an
> official confirmation of any transaction. TD Securities is neither making
> any investment recommendation nor providing any professional or advisory
> services relating to the activities described herein. All market prices,
> data and other information are not warranted as to completeness or accuracy
> and are subject to change without notice Any products described herein are
> (i) not insured by the FDIC, (ii) not a deposit or other obligation of, or
> guaranteed by, an insured depository institution and (iii) subject to
> investment risks, including possible loss of the principal amount invested.
> The information shall not be further distributed or duplicated in whole or
> in part by any means without the prior written consent of TD Securities. TD
> Securities is a trademark of The Toronto-Dominion Bank and represents TD
> Securities (USA) LLC and certain investment banking activities of The
> Toronto-Dominion Bank and its subsidiaries.
>

-- 


--




Mime
View raw message