incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Problem loading Saved Key-Cache on 1.2.3 after upgrading from 1.1.10
Date Fri, 05 Apr 2013 16:56:23 GMT
> skipping sstable due to bloom filter debug messages
What were these messages?

Do you have the logs from the start up ? 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 4/04/2013, at 6:11 AM, Arya Goudarzi <goudarzi@gmail.com> wrote:

> Hi,
> 
> I have upgraded 2 nodes out of a 12 mode test cluster from 1.1.10 to 1.2.3. During startup
while tailing C*'s system.log, I observed a series of SSTable batch load messages and skipping
sstable due to bloom filter debug messages which is normal for startup, but when it reached
loading saved key caches, it gets stuck forever. The I/O wait stays high in the CPU graph
and I/O ops are sent to disk, but C* never passes that step of loading the key cache file
successfully. The saved key cache file was about 75MB on one node and 125MB on the other node
and they were for different CFs. 
> 
> <image.jpeg>
> 
> The CPU I/O wait constantly stayed at 40%~ while system.log was stuck at loading one
saved key cache file. I have marked that on the graph above. The workaround was to delete
the saved cache files and things loaded fine (See marked Normal Startup). 
> 
> These machines are m1.xlarge EC2 instances. And this issue happened on both nodes upgraded.
This did not happen during exercise of upgrade from 1.1.6 to 1.2.2 using the same snapshot.

> 
> Should I raise a JIRA? 
> 
> -Arya


Mime
View raw message