cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benedict (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10697) Leak detected while running offline scrub
Date Mon, 25 Jan 2016 11:32:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15115066#comment-15115066
] 

Benedict commented on CASSANDRA-10697:
--------------------------------------

I've pushed a fix for this [here|https://github.com/belliottsmith/cassandra/tree/10697-2.1]

It would be helpful if test engineering could have a look at the dtest that is meant to check
this functionality, to try and get it to exercise this problem. I'm afraid my python is very
rusty and I'm only here for a few more days.

In the process of merging this patch upwards, I found that it looks very much like scrub is
not working at all in 2.2 upwards, as the transaction we are using doesn't get committed.
 So the newly scrubbed sstables should be being deleted, and the old ones restored.  Could
test engineering also check why the dtest isn't catching this problem?

> Leak detected while running offline scrub
> -----------------------------------------
>
>                 Key: CASSANDRA-10697
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10697
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Tools
>         Environment: C* 2.1.9 on Debian Wheezy
>            Reporter: mlowicki
>            Assignee: Benedict
>            Priority: Critical
>
> I got couple of those:
> {code}
> ERROR 05:09:15 LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@3b60e162)
to class org.apache.cassandra.io.sstable.SSTableReader$InstanceTidier@1433208674:/var/lib/cassandra/data/sync/entity2-e24b5040199b11e5a30f75bb514ae072/sync-entity2-ka-405434
was not released before the reference was garbage collected
> {code}
> and then:
> {code}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:99)
>         at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:81)
>         at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:353)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:444)
>         at java.io.RandomAccessFile.readFully(RandomAccessFile.java:424)
>         at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:378)
>         at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:348)
>         at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:327)
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:397)
>         at org.apache.cassandra.db.composites.AbstractCType$Serializer.deserialize(AbstractCType.java:381)
>         at org.apache.cassandra.db.OnDiskAtom$Serializer.deserializeFromSSTable(OnDiskAtom.java:75)
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:52)
>         at org.apache.cassandra.db.AbstractCell$1.computeNext(AbstractCell.java:46)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.io.sstable.SSTableIdentityIterator.hasNext(SSTableIdentityIterator.java:120)
>         at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:202)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645)
>         at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
>         at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
>         at org.apache.cassandra.db.ColumnIndex$Builder.buildForCompaction(ColumnIndex.java:165)
>         at org.apache.cassandra.db.compaction.LazilyCompactedRow.write(LazilyCompactedRow.java:121)
>         at org.apache.cassandra.io.sstable.SSTableWriter.append(SSTableWriter.java:192)
>         at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:127)
>         at org.apache.cassandra.io.sstable.SSTableRewriter.tryAppend(SSTableRewriter.java:158)
>         at org.apache.cassandra.db.compaction.Scrubber.scrub(Scrubber.java:220)
>         at org.apache.cassandra.tools.StandaloneScrubber.main(StandaloneScrubber.java:116)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message