From user-return-33242-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Fri Apr 5 16:57:04 2013 Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D0B07F130 for ; Fri, 5 Apr 2013 16:57:04 +0000 (UTC) Received: (qmail 97815 invoked by uid 500); 5 Apr 2013 16:57:02 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 97780 invoked by uid 500); 5 Apr 2013 16:57:02 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 97771 invoked by uid 99); 5 Apr 2013 16:57:02 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Apr 2013 16:57:02 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: local policy) Received: from [208.113.200.5] (HELO homiemail-a80.g.dreamhost.com) (208.113.200.5) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Apr 2013 16:56:57 +0000 Received: from homiemail-a80.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a80.g.dreamhost.com (Postfix) with ESMTP id AC45C37A07E for ; Fri, 5 Apr 2013 09:56:36 -0700 (PDT) Received: from [10.65.15.205] (unknown [59.164.97.108]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: aaron@thelastpickle.com) by homiemail-a80.g.dreamhost.com (Postfix) with ESMTPSA id 2FF1837A065 for ; Fri, 5 Apr 2013 09:56:30 -0700 (PDT) From: aaron morton Content-Type: multipart/alternative; boundary="Apple-Mail=_2A92882E-5A52-436A-A0A7-9E504F5F2BEB" Message-Id: <6901CDF8-B7A0-4829-AF7A-DBA46CB35451@thelastpickle.com> Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: Problem loading Saved Key-Cache on 1.2.3 after upgrading from 1.1.10 Date: Fri, 5 Apr 2013 22:26:23 +0530 References: To: user@cassandra.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1499) X-Virus-Checked: Checked by ClamAV on apache.org --Apple-Mail=_2A92882E-5A52-436A-A0A7-9E504F5F2BEB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii > skipping sstable due to bloom filter debug messages What were these messages? Do you have the logs from the start up ?=20 Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 4/04/2013, at 6:11 AM, Arya Goudarzi wrote: > Hi, >=20 > I have upgraded 2 nodes out of a 12 mode test cluster from 1.1.10 to = 1.2.3. During startup while tailing C*'s system.log, I observed a series = of SSTable batch load messages and skipping sstable due to bloom filter = debug messages which is normal for startup, but when it reached loading = saved key caches, it gets stuck forever. The I/O wait stays high in the = CPU graph and I/O ops are sent to disk, but C* never passes that step of = loading the key cache file successfully. The saved key cache file was = about 75MB on one node and 125MB on the other node and they were for = different CFs.=20 >=20 > >=20 > The CPU I/O wait constantly stayed at 40%~ while system.log was stuck = at loading one saved key cache file. I have marked that on the graph = above. The workaround was to delete the saved cache files and things = loaded fine (See marked Normal Startup).=20 >=20 > These machines are m1.xlarge EC2 instances. And this issue happened on = both nodes upgraded. This did not happen during exercise of upgrade from = 1.1.6 to 1.2.2 using the same snapshot.=20 >=20 > Should I raise a JIRA?=20 >=20 > -Arya --Apple-Mail=_2A92882E-5A52-436A-A0A7-9E504F5F2BEB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii
skipping sstable due to = bloom filter debug messages
What were these = messages?

Do you have the logs from the start up = ? 

Cheers

http://www.thelastpickle.com

On 4/04/2013, at 6:11 AM, Arya Goudarzi <goudarzi@gmail.com> = wrote:

Hi,

I have upgraded 2 = nodes out of a 12 mode test cluster from 1.1.10 to 1.2.3. During startup = while tailing C*'s system.log, I observed a series of SSTable batch load = messages and skipping sstable due to bloom filter debug messages which = is normal for startup, but when it reached loading saved key caches, it = gets stuck forever. The I/O wait stays high in the CPU graph and I/O ops = are sent to disk, but C* never passes that step of loading the key cache = file successfully. The saved key cache file was about 75MB on one node = and 125MB on the other node and they were for different CFs. 
=

<image.jpeg>

The CPU I/O wait constantly stayed at 40%~ while = system.log was stuck at loading one saved key cache file. I have marked = that on the graph above. The workaround was to delete the saved cache = files and things loaded fine (See marked Normal Startup). 

These machines are m1.xlarge = EC2 instances. And this issue happened on both nodes upgraded. This did = not happen during exercise of upgrade from 1.1.6 to 1.2.2 = using the same snapshot. 

Should I raise a = JIRA? 

-Arya

= --Apple-Mail=_2A92882E-5A52-436A-A0A7-9E504F5F2BEB--