Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 66747997F for ; Tue, 13 Mar 2012 09:10:57 +0000 (UTC) Received: (qmail 39231 invoked by uid 500); 13 Mar 2012 09:10:55 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 39195 invoked by uid 500); 13 Mar 2012 09:10:54 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 39164 invoked by uid 99); 13 Mar 2012 09:10:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Mar 2012 09:10:54 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a42.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 13 Mar 2012 09:10:28 +0000 Received: from homiemail-a42.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a42.g.dreamhost.com (Postfix) with ESMTP id 883B868C05D for ; Tue, 13 Mar 2012 02:10:05 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; q=dns; s=thelastpickle.com; b=Uk0Vhs8eES HNIsSmNJZvtVRPTcZdHoWy3/G0kykJuY1spVmieRtKnLWzYXBIhqigJC72FzeWs7 B24Eo9gWdMQiB5KP+VPwJ9126i5rrFrWpfOGVqrR8mX8n8uR2015Cq/vRUmRTy7L bbGYLGcvHnz4TMWvre8qKW4oe0Z83VQwU= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=thelastpickle.com; h=from :mime-version:content-type:subject:date:in-reply-to:to :references:message-id; s=thelastpickle.com; bh=azoaic2Lqy0UereL cYsRr6vIoIY=; b=jQNpNFkvDeLf+qLUUb0oRKa7I/Utp6/aAY3zFGX39diO1lA9 MBR4HRABcc5opOJ+4ZWcyrMBuSD6ar5nUoxG6hj33FsGDk2tG9Qt9MAiP0CzoXwq y4REpo5o0NWnahp4yF9RKsPrU4UXkpP8XUb8/wehSqq4Gi7gsoLjyJaCCWM= Received: from [172.16.1.3] (125-236-193-159.adsl.xtra.co.nz [125.236.193.159]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a42.g.dreamhost.com (Postfix) with ESMTPSA id 0549768C058 for ; Tue, 13 Mar 2012 02:10:04 -0700 (PDT) From: aaron morton Mime-Version: 1.0 (Apple Message framework v1257) Content-Type: multipart/alternative; boundary="Apple-Mail=_04524812-3635-40AB-95E5-F1AA9849B1C8" Subject: Re: OOM opening bloom filter Date: Tue, 13 Mar 2012 22:10:02 +1300 In-Reply-To: <1331580247.7031.545.camel@localhost> To: user@cassandra.apache.org References: <1331495276.7031.16.camel@localhost> <1331508147.7031.29.camel@localhost> <1331509440.7031.35.camel@localhost> <06F26BDB-37FD-4527-9DA2-C3F60270C487@thelastpickle.com> <1331580247.7031.545.camel@localhost> Message-Id: X-Mailer: Apple Mail (2.1257) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_04524812-3635-40AB-95E5-F1AA9849B1C8 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Thanks for the update. How much smaller did the BF get to ? A ----------------- Aaron Morton Freelance Developer @aaronmorton http://www.thelastpickle.com On 13/03/2012, at 8:24 AM, Mick Semb Wever wrote: > >>>>> It's my understanding then for this use case that bloom filters are of >>>>> little importance and that i can > > > Ok. To summarise our actions to get us out of this situation, in hope > that it may help others one day, we did the following actions: > > 1) upgrade to 1.0.7 > 2) set fp_ratio=0.99 > 3) set index_interval=4096 > 4) restarted the node with Xmx30G > 5) run `nodetool scrub` > and monitor total size of bf files > using `du -hc *-Filter.db | grep total` > 6) restart node with original Xmx setting once total bf size is under > (scrub was running for >12hrs) > (remaining bloom filters can be rebuilt later from normal compact) > > Hopefully it will also eventuate that this cluster can run with a more > normal Xmx4G rather than the previous Xmx12G. > > (2) and (3) are very much dependent on our set up using hadoop where all > reads are get_range_slice with 16k rows per request. Both could be tuned > correctly but they're the numbers that worked first up. > > ~mck > > -- > "When there is no enemy within, the enemies outside can't hurt you." > African proverb > > | http://github.com/finn-no | http://tech.finn.no | --Apple-Mail=_04524812-3635-40AB-95E5-F1AA9849B1C8 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
http://www.thelastpickle.com
On 13/03/2012, at 8:24 AM, Mick Semb Wever wrote:


It's = my understanding then for this use case that bloom filters are = of
little importance and that i = can


Ok. = To summarise our actions to get us out of this situation, in = hope
that it may help others one day, we did the following = actions:

1) upgrade to 1.0.7
2) set fp_ratio=3D0.99
3) = set index_interval=3D4096
4) restarted the node with Xmx30G
5) = run `nodetool scrub`
     and monitor = total size of bf files
     using `du -hc = *-Filter.db | grep total`
6) restart node with original Xmx setting = once total bf size is under
     (scrub was = running for >12hrs)
     (remaining = bloom filters can be rebuilt later from normal compact)

Hopefully = it will also eventuate that this cluster can run with a more
normal = Xmx4G rather than the previous Xmx12G.

(2) and (3) are very much = dependent on our set up using hadoop where all
reads are = get_range_slice with 16k rows per request. Both could be = tuned
correctly but they're the numbers that worked first = up.

~mck

--
"When there is no enemy within, the = enemies outside can't hurt you."
African proverb

| http://github.com/finn-no | http://tech.finn.no = |

= --Apple-Mail=_04524812-3635-40AB-95E5-F1AA9849B1C8--