From user-return-59907-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Feb 21 08:07:04 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9400718061A for ; Wed, 21 Feb 2018 08:07:03 +0100 (CET) Received: (qmail 79215 invoked by uid 500); 21 Feb 2018 07:07:01 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 79202 invoked by uid 99); 21 Feb 2018 07:07:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 21 Feb 2018 07:07:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id A7A96C001A for ; Wed, 21 Feb 2018 07:07:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.899 X-Spam-Level: X-Spam-Status: No, score=0.899 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_REPLY=1, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id tij90BHs1_e9 for ; Wed, 21 Feb 2018 07:06:59 +0000 (UTC) Received: from mail-io0-f174.google.com (mail-io0-f174.google.com [209.85.223.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 63B0E5F3CD for ; Wed, 21 Feb 2018 07:06:59 +0000 (UTC) Received: by mail-io0-f174.google.com with SMTP id t22so1065609ioa.7 for ; Tue, 20 Feb 2018 23:06:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:date:subject:message-id :references:in-reply-to:to; bh=JfXDqHDxgJSfUCcsO+BkpCGdbaKicILH0o55ptXRgSc=; b=TgBZIeC+Jpc2MKTNGAIfuGhZDKwedAyiD1cJj9ip8EIsJJLIyc71K9QZiruqrZuHud meWQmOYnbL+GauIPQcBwhoNRRLmvgPXNFeRpnaaFdM/K5JfU7WL0NnwnTavjLX/fNvoT y6++eM0CZlEFxWKHBn+J0HSXaBoZvO+6AnNb+mwiCQRuTFjkhd0yHAOi3z8SPNM4IFDz RMvxShZBaqsJ+24WyQCdOkfyIJPALSmp+qb0y4Wtyv6TQ2QI3L8XvwSaRDxDGlqc3h3b 303+BPZInLU7YEPPm6nVwLwfgfkf7gBW8hrKe5tiduXAlHlehdhFQmW7uOehy4m3Z4OJ KhCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version:date :subject:message-id:references:in-reply-to:to; bh=JfXDqHDxgJSfUCcsO+BkpCGdbaKicILH0o55ptXRgSc=; b=e8mnWx3+cDeb9ZhvLXGfQWBqMobNyAI35c2c7H0lHkzQw169XsGvwCWE6KeZafLl92 1OS0eR+IAhbyUvitQ7MW59pVsVE56e0nedcL9/pnw/kQTfZ6eRzvW7kau+0idaKoYG30 o8UK54FeXTuSH2kGLNUd+Rs2SF1bqjyo0PpopfibuR9MryHW9ETXrvmWuyxQg/yjpS/p Ou5IxSCI1Y59uaoEuOCXGssQ7IDXV1cSpD5CbBsOdX21zddhYvZIXEh5kODQUZwRU6tR ZGiC2qEPoWSGzsuanMrfuuRTQIBpHmCxIaXkW/oHikelm4SzhWssGPDyDVccHgJ3CLvY y7jQ== X-Gm-Message-State: APf1xPAJt8hNHsAsLNyp+CL21GXwobZBqZo934nwZdS+HGO912OJz/c6 2TBZHlmL+nPKg8FJWGkf6HoGCQlm X-Google-Smtp-Source: AG47ELvlvyot7yy7AkhZ3zZvV5GOLt4xWmPqjHjzF8d6Gy0WQhAFcC0VoMxzUVZD2LWckVUIsQc9eA== X-Received: by 10.107.150.196 with SMTP id y187mr2913644iod.294.1519196818514; Tue, 20 Feb 2018 23:06:58 -0800 (PST) Received: from [192.168.0.49] (97-113-127-41.tukw.qwest.net. [97.113.127.41]) by smtp.gmail.com with ESMTPSA id z131sm23261555itb.4.2018.02.20.23.06.57 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 20 Feb 2018 23:06:57 -0800 (PST) From: Jeff Jirsa Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Date: Tue, 20 Feb 2018 23:06:56 -0800 Subject: Re: LEAK DETECTED while minor compaction Message-Id: References: In-Reply-To: To: user@cassandra.apache.org X-Mailer: iPhone Mail (15D60) Your bloom filter settings look broken. Did you set the FP ratio to 0? If so= that=E2=80=99s a bad idea and we should have stopped you from doing it. --=20 Jeff Jirsa > On Feb 20, 2018, at 11:01 PM, =D0=94=D0=B0=D1=80=D1=8C=D1=8F =D0=9C=D0=B5=D0= =BB=D0=B5=D0=BD=D1=86=D0=BE=D0=B2=D0=B0 wrote: >=20 > Hello. >=20 > Could you help me with LEAK DETECTED error while minor compaction process?= >=20 > There is a table with a lot of small record 6.6*10^9 (mapping > (eventId, boxId) -> cellId)). > Minor compaction starts and then fails on 99% done with an error: >=20 > Stacktrace > ERROR [Reference-Reaper:1] 2018-02-05 10:06:17,032 Ref.java:207 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@1ca1bf87) to class > org.apache.cassandra.io.util.MmappedSegmentedFile$Cleanup@308695651:/stora= ge1/cassandra_events/data/EventsKeyspace/PerBoxEventSeriesEvents-41847c3049a= 211e6af50b9221207cca8/tmplink-lb-102593-big-Index.db > was not released before the reference was garbage collected > ERROR [Reference-Reaper:1] 2018-02-05 10:06:17,033 Ref.java:207 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@1659d4f7) to class > org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$1@1398495320:= [Memory@[0..dc), > Memory@[0..898)] was not released before the reference was garbage > collected > ERROR [Reference-Reaper:1] 2018-02-05 10:06:17,033 Ref.java:207 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@42978833) to class > org.apache.cassandra.utils.concurrent.WrappedSharedCloseable$1@1648504648:= [[OffHeapBitSet]] > was not released before the reference was garbage collected > ERROR [Reference-Reaper:1] 2018-02-05 10:06:17,033 Ref.java:207 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@3a64a19b) to class > org.apache.cassandra.io.sstable.format.SSTableReader$DescriptorTypeTidy@86= 3282967:/storage1/cassandra_events/data/EventsKeyspace/PerBoxEventSeriesEven= ts-41847c3049a211e6af50b9221207cca8/tmplink-lb-102593-big > was not released before the reference was garbage collected > ERROR [Reference-Reaper:1] 2018-02-05 10:06:17,033 Ref.java:207 - LEAK > DETECTED: a reference > (org.apache.cassandra.utils.concurrent.Ref$State@4ddc775a) to class > org.apache.cassandra.io.util.CompressedPoolingSegmentedFile$Cleanup@104170= 9510:/storage1/cassandra_events/data/EventsKeyspace/PerBoxEventSeriesEvents-= 41847c3049a211e6af50b9221207cca8/tmplink-lb-102593-big-Data.db > was not released before the reference was garbage collected >=20 > I have tried increase max heap size (8GB -> 16GB), but got the same error.= > How can I resolve the issue? >=20 >=20 > Cassandra parameters and the problem table > Cassandra v 2.2.9 > MAX_HEAP_SIZE=3D"16G" > java version "1.8.0_121" >=20 > compaction =3D {'min_threshold': '4', 'enabled': 'True', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > compression =3D {'sstable_compression': > 'org.apache.cassandra.io.compress.SnappyCompressor'} >=20 > nodetoole tablestats > Read Count: 1454605 > Read Latency: 2.0174777647540054 ms. > Write Count: 12034909 > Write Latency: 0.044917336558174224 ms. > Pending Flushes: 0 > Table: PerBoxEventSeriesEventIds > SSTable count: 20 > Space used (live): 885969527458 > Space used (total): 885981801994 > Space used by snapshots (total): 0 > Off heap memory used (total): 19706226232 > SSTable Compression Ratio: 0.5722091068132875 > Number of keys (estimate): 6614724684 > Memtable cell count: 375796 > Memtable data size: 31073510 > Memtable off heap memory used: 0 > Memtable switch count: 30 > Local read count: 1454605 > Local read latency: NaN ms > Local write count: 12034909 > Local write latency: NaN ms > Pending flushes: 0 > Bloom filter false positives: 0 > Bloom filter false ratio: 0.00000 > Bloom filter space used: -4075791744 > Bloom filter off heap memory used: 17399044576 > Index summary off heap memory used: 2091833184 > Compression metadata off heap memory used: 215348472 > Compacted partition minimum bytes: 104 > Compacted partition maximum bytes: 149 > Compacted partition mean bytes: 149 > Average live cells per slice (last five minutes): NaN > Maximum live cells per slice (last five minutes): 0 > Average tombstones per slice (last five minutes): NaN > Maximum tombstones per slice (last five minutes): 0 >=20 > Thank You > Darya Melentsova >=20 > email: ifireice@gmail.com >=20 > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org > For additional commands, e-mail: user-help@cassandra.apache.org >=20 --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For additional commands, e-mail: user-help@cassandra.apache.org