Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 457B4200D28 for ; Mon, 9 Oct 2017 05:36:53 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 43DC8160BDE; Mon, 9 Oct 2017 03:36:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 13C051609E6 for ; Mon, 9 Oct 2017 05:36:51 +0200 (CEST) Received: (qmail 98576 invoked by uid 500); 9 Oct 2017 03:36:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 98561 invoked by uid 99); 9 Oct 2017 03:36:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 09 Oct 2017 03:36:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 4051718B97A for ; Mon, 9 Oct 2017 03:36:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.129 X-Spam-Level: *** X-Spam-Status: No, score=3.129 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, KAM_NUMSUBJECT=0.5, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id UmNtx2YRrir8 for ; Mon, 9 Oct 2017 03:36:46 +0000 (UTC) Received: from mail-oi0-f46.google.com (mail-oi0-f46.google.com [209.85.218.46]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A112E5FBED for ; Mon, 9 Oct 2017 03:36:46 +0000 (UTC) Received: by mail-oi0-f46.google.com with SMTP id f3so37802151oia.2 for ; Sun, 08 Oct 2017 20:36:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=IeOwnetknhiV8hFx2LWSsEQfis8AVMBkaNwZpvcD7Vo=; b=XzkXOCsLzjYqrXczWzSbBHKq7wDOPoKfCkSIJ+kHurTPNaLzKM7x0E/WJ9V7pPvRu7 m6MUdVebsy5Cr2r261+2IIpKypF06Fs/nSuItRvI9Z5yPSnpJ+0ssNtdbqXw+uEbPKVm snccatfrBf6JWLXn9pUbxNO8a7GUR+86FE50hQmTwHc8YViVCth4VmTewKZPCfLFJWyR 6R+p/hK/kRBtIR0jK+npqz78ULtGAm2iYbv6ANVrm4s3pJoHQZJz1PwM3koZ3nAWPR/j r8UjU+IWOZEb4bHJsxxR7H1b53WFdK8nBJPKog4xDPOLcW3/OYZSAeU4Rg5hEgWZkXfz +/kA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=IeOwnetknhiV8hFx2LWSsEQfis8AVMBkaNwZpvcD7Vo=; b=nWSUi3eJFsdfEGvgRzIcoc2X76HNsXYqo5zI0sQSx5RH820qzVOlDws63n4Mw8u4iq NTRsUMtbofRWFQHEogcQD3gQbMfJQaeAaZOMCTMTYR1tDzLbp9htk+iZ8yHDLAmFsUyI Qshm/pKfHiLui9rbr+6848mzQl2wpOBoZOaKAR4ybSzPMwG/kZCQWub9zXITFyJepv7L +uJjbAFxXu5A6R3sQSixXf94BpyhgaypU8AblrpItOCupWE7Gv80k1uBYCqvIkDD643Q L8xB+MDrRPOw7vQK7YTWcQZ3FB06FX2pI3DPmF6Wu9RPtgOY+FIzhtesO2kYyh9AQj7k G4Xw== X-Gm-Message-State: AMCzsaW5o/an843iy0xV5leDnXCoVqnXGu8IAnShSrOkqYi0kO+a8MiD hlw64fK0wHbPyJr7nMPnwgshwk6QnWq+7tjU2C1CL/rr X-Google-Smtp-Source: AOwi7QCKwkRm50nFKVnHGZZUInoapjiqHZsP6IJ8aT5/hLxzy4piVoVabhx3lYZp+zk4HPwyqh1sSiWMzRP6Kngd/iQ= X-Received: by 10.157.80.163 with SMTP id b35mr5551500oth.284.1507520205855; Sun, 08 Oct 2017 20:36:45 -0700 (PDT) MIME-Version: 1.0 Received: by 10.74.178.72 with HTTP; Sun, 8 Oct 2017 20:36:45 -0700 (PDT) In-Reply-To: <1CEA977C-FD04-473E-9FB2-6757B5B4DFE9@gmail.com> References: <1CEA977C-FD04-473E-9FB2-6757B5B4DFE9@gmail.com> From: Ridley Submission Date: Mon, 9 Oct 2017 14:36:45 +1100 Message-ID: Subject: Re: How to interpret uninterpreted `nodetool compact` java stack trace in 3.11.0 To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="f4030435bb745ca4e6055b14e697" archived-at: Mon, 09 Oct 2017 03:36:53 -0000 --f4030435bb745ca4e6055b14e697 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ok thanks for your reply. I do believe this one is already in JIRA: https://issues.apache.org/jira/browse/CASSANDRA-13897?jql=3Dtext%20~%20%22n= odetool%20flush%20compact%22 Regards, Ridley NB: The debug.log provides only minimal extra information above what is in that JIRA ticket. The name of the table this error occurs on changes occasionally. I've not yet come across any practical impact in the running of our system. I don't know if it is strange, but the only thing that seems strange to me is that the commit logs on a node appear to grow to be 20 times the size of the actual data sstables (even after compaction): DEBUG [PerDiskMemtableFlushWriter_0:967] 2017-10-09 14:31:44,659 Memtable.java:490 - Completed flushing /home/cass/apache-cassandra-3.11.0/data/data/newsreader/article-82769690885= f11e7ad7a7dd51e316d41/mc-307-big-Data.db (184.877KiB) for commitlog position CommitLogPosition(segmentId=3D1506151484359, position=3D28204113) ERROR [MemtablePostFlush:540] 2017-10-09 14:31:44,688 CassandraDaemon.java:228 - Exception in thread Thread[MemtablePostFlush:540,5,RMI Runtime] java.lang.AssertionError: null at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.(ChunkCache.j= ava:222) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.util.FileHandle$Builder.maybeCached(FileHandle.java= :412) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:38= 1) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:33= 1) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.openFinal(BigTabl= eWriter.java:333) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTab= leWriter.java:53) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.big.BigTableWriter$TransactionalProx= y.doPrepare(BigTableWriter.java:374) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.p= repareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTabl= eWriter.java:281) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.prepareToCommit(Si= mpleSSTableMultiWriter.java:101) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyS= tore.java:1153) ~[apache-cassandra-3.11.0.jar:3.11.0] at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:= 1086) ~[apache-cassandra-3.11.0.jar:3.11.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1= 149) ~[na:1.8.0_144] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:= 624) [na:1.8.0_144] at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallo= cator$0(NamedThreadFactory.java:81) [apache-cassandra-3.11.0.jar:3.11.0] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] ERROR [Reference-Reaper:1] 2017-10-09 14:31:45,247 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@eefaa4) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@20886337 :[Memory@[0..30), Memory@[0..1e0)] was not released before the reference was garbage collected ERROR [Reference-Reaper:1] 2017-10-09 14:31:46,725 Ref.java:224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Ref$State@1082ad6) to class org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@19319642 :[Memory@[0..90), Memory@[0..2d0)] was not released before the reference was garbage collected On Mon, Oct 9, 2017 at 2:22 PM, Jeff Jirsa wrote: > Likely failed but you=E2=80=99d need to check the full logs to be positiv= e > > Assertion errors typically should be reported on the JIRA - you=E2=80=99v= e > encountered a situation the developer didn=E2=80=99t think was possible s= o it=E2=80=99s > aborted, but we need to know that you encountered it so we can fix it > > Please open the jira here with as much info as you can safely provide > (anonymize logs as needed): > https://issues.apache.org/jira/projects/CASSANDRA/summary > > > -- > Jeff Jirsa > > > On Oct 8, 2017, at 8:19 PM, Ridley Submissions < > ridley.submission2015@gmail.com> wrote: > > Hi Everyone, > > I wonder how do we interpret this error message? Does it mean that the > compaction failed? Is this an error I should be concerned about? > > Thanks, > Ridley > > > cassandra@server:$ nodetool compact > error: null > -- StackTrace -- > java.lang.AssertionError > at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.< > init>(ChunkCache.java:222) > at org.apache.cassandra.cache.ChunkCache.wrap(ChunkCache.java:175) > at org.apache.cassandra.io.util.FileHandle$Builder. > maybeCached(FileHandle.java:412) > at org.apache.cassandra.io.util.FileHandle$Builder.complete( > FileHandle.java:381) > at org.apache.cassandra.io.util.FileHandle$Builder.complete( > FileHandle.java:331) > > --f4030435bb745ca4e6055b14e697 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Ok thanks for your reply.

I do be= lieve this one is already in JIRA:

https://issues.apache.org/jira/browse/CASSANDRA-13897?jql=3Dtext%20= ~%20%22nodetool%20flush%20compact%22

Regards,
Ridley

NB: The debug.log provides only min= imal extra information above what is in that JIRA ticket. The name of the t= able this error occurs on changes occasionally. I've not yet come acros= s any practical impact in the running of our system. I don't know if it= is strange, but the only thing that seems strange to me is that the commit= logs on a node appear to grow to be 20 times the size of the actual data s= stables (even after compaction):

DEBUG [PerDiskMemtableFlushWri= ter_0:967] 2017-10-09 14:31:44,659 Memtable.java:490 - Completed flushing /= home/cass/apache-cassandra-3.11.0/data/data/newsreader/article-82769690885f= 11e7ad7a7dd51e316d41/mc-307-big-Data.db (184.877KiB) for commitlog position= CommitLogPosition(segmentId=3D1506151484359, position=3D28204113)
ERROR [MemtablePostFlush:540] 2017-10-09 14:31:44,688 CassandraDaemon.jav= a:228 - Exception in thread Thread[MemtablePostFlush:540,5,RMI Runtime]
java.lang.AssertionError: null
at org.apache.cassandra.cache.ChunkCache$CachingRebufferer.<= ;init>(ChunkCache.java:222) ~[apache-cassandra-3.11.0.jar:3.11.0]
<= div> at org.apache.cassandra.cache.C= hunkCache.wrap(ChunkCache.java:175) ~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.io= .util.FileHandle$Builder.maybeCached(FileHandle.java:412) ~[apache-cassandr= a-3.11.0.jar:3.11.0]
at o= rg.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:381= ) ~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.io.util.FileHandle$Builder.complete(= FileHandle.java:331) ~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.io.sstable.format= .big.BigTableWriter.openFinal(BigTableWriter.java:333) ~[apache-cassandra-3= .11.0.jar:3.11.0]
at org.= apache.cassandra.io.sstable.format.big.BigTableWriter.access$600(BigTableWr= iter.java:53) ~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.io.sstable.format.big.= BigTableWriter$TransactionalProxy.doPrepare(BigTableWriter.java:374) ~[apac= he-cassandra-3.11.0.jar:3.11.0]
= at org.apache.cassandra.utils.concurrent.Transactional$AbstractTrans= actional.prepareToCommit(Transactional.java:173) ~[apache-cassandra-3.11.0.= jar:3.11.0]
at org.apache= .cassandra.io.sstable.format.SSTableWriter.prepareToCommit(SSTableWriter.ja= va:281) ~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.io.sstable.SimpleSSTableMultiW= riter.prepareToCommit(SimpleSSTableMultiWriter.java:101) ~[apache-cassandra= -3.11.0.jar:3.11.0]
at or= g.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilySto= re.java:1153) ~[apache-cassandra-3.11.0.jar:3.11.0]
at org.apache.cassandra.db.ColumnFamilyStore$F= lush.run(ColumnFamilyStore.java:1086) ~[apache-cassandra-3.11.0.jar:3.11.0]=
at java.util.concurrent.= ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_144]<= /div>
at java.util.concurrent.T= hreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_144]
at org.apache.cassandra.con= current.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFacto= ry.java:81) [apache-cassandra-3.11.0.jar:3.11.0]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0= _144]
ERROR [Reference-Reaper:1] 2017-10-09 14:31:45,247 Ref.java= :224 - LEAK DETECTED: a reference (org.apache.cassandra.utils.concurrent.Re= f$State@eefaa4) to class org.apache.cassandra.utils.concurrent.WrappedShare= dCloseable$Tidy@20886337:[Memory@[0..30), Memory@[0..1e0)] was not released= before the reference was garbage collected
ERROR [Reference-Reap= er:1] 2017-10-09 14:31:46,725 Ref.java:224 - LEAK DETECTED: a reference (or= g.apache.cassandra.utils.concurrent.Ref$State@1082ad6) to class org.apache.= cassandra.utils.concurrent.WrappedSharedCloseable$Tidy@19319642:[Memory@[0.= .90), Memory@[0..2d0)] was not released before the reference was garbage co= llected


On Mon, Oct 9, 2017 at 2:22 PM, Jeff Jirsa <jjirsa@g= mail.com> wrote:
Likely failed but you=E2=80=99d need to check the full logs to be= positive

Assertion errors typically should be reported = on the JIRA - you=E2=80=99ve encountered a situation the developer didn=E2= =80=99t think was possible so it=E2=80=99s aborted, but we need to know tha= t you encountered it so we can fix it=C2=A0

Please= open the jira here with as much info as you can safely provide (anonymize = logs as needed):=C2=A0
https://issues.apache.org/jira/= projects/CASSANDRA/summary


= --=C2=A0
Jeff Jirsa


On Oct 8, 2017, at 8:19 PM, Ridley Submissions <ridley.su= bmission2015@gmail.com> wrote:

Hi Everyone,

I wonder how d= o we interpret this error message? Does it mean that the compaction failed?= =C2=A0 Is this an error I should be concerned about?

Thanks,
Ridley


cassandr= a@server:$ nodetool compact
error: null
-- StackTrace -= -
java.lang.AssertionError
at org.apache.cassandra.cache.ChunkCache$CachingRebuf= ferer.<init>(ChunkCache.java:222)
at org.apache.cassandra.cache.ChunkCache.wrap= (ChunkCache.java:175)
= at org.apache.cassandra.io.util.FileHandle$Builder.maybeCa= ched(FileHandle.java:412)
at org.apache.cassandra.io.util.FileHandle$Builder.complete= (FileHandle.java:381)
= at org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:331)


--f4030435bb745ca4e6055b14e697--