incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Janne Jalkanen <Janne.Jalka...@ecyrd.com>
Subject Re: HintedHandoff IOError?
Date Tue, 12 Mar 2013 06:19:03 GMT

Oops, forgot to mention that, did I… Cass 1.1.10. 

What is the sanctioned way of removing hints? rm -f HintsColumnFamily*? Truncate from CLI?

This is ls -l of my /system/HintsColumnFamily/ btw - the only one with zero size are the -tmp-
files.  It seems odd…

-rw-rw-r--  1 ubuntu ubuntu 86373144 Jan 26 21:39 system-HintsColumnFamily-hf-11-Data.db
-rw-rw-r--  1 ubuntu ubuntu       80 Jan 26 21:39 system-HintsColumnFamily-hf-11-Digest.sha1
-rw-rw-r--  1 ubuntu ubuntu      976 Jan 26 21:39 system-HintsColumnFamily-hf-11-Filter.db
-rw-rw-r--  1 ubuntu ubuntu       11 Jan 26 21:39 system-HintsColumnFamily-hf-11-Index.db
-rw-rw-r--  1 ubuntu ubuntu     4348 Jan 26 21:39 system-HintsColumnFamily-hf-11-Statistics.db
-rw-rw-r--  1 ubuntu ubuntu      569 Feb 27 08:33 system-HintsColumnFamily-hf-23-Data.db
-rw-rw-r--  1 ubuntu ubuntu       80 Feb 27 08:33 system-HintsColumnFamily-hf-23-Digest.sha1
-rw-rw-r--  1 ubuntu ubuntu     1936 Feb 27 08:33 system-HintsColumnFamily-hf-23-Filter.db
-rw-rw-r--  1 ubuntu ubuntu       11 Feb 27 08:33 system-HintsColumnFamily-hf-23-Index.db
-rw-rw-r--  1 ubuntu ubuntu     4356 Feb 27 08:33 system-HintsColumnFamily-hf-23-Statistics.db
-rw-rw-r--  1 ubuntu ubuntu  5500155 Feb 27 08:57 system-HintsColumnFamily-hf-24-Data.db
-rw-rw-r--  1 ubuntu ubuntu       80 Feb 27 08:57 system-HintsColumnFamily-hf-24-Digest.sha1
-rw-rw-r--  1 ubuntu ubuntu       16 Feb 27 08:57 system-HintsColumnFamily-hf-24-Filter.db
-rw-rw-r--  1 ubuntu ubuntu       26 Feb 27 08:57 system-HintsColumnFamily-hf-24-Index.db
-rw-rw-r--  1 ubuntu ubuntu     4340 Feb 27 08:57 system-HintsColumnFamily-hf-24-Statistics.db
-rw-rw-r--  1 ubuntu ubuntu        0 Feb 27 08:57 system-HintsColumnFamily-tmp-hf-25-Data.db
-rw-rw-r--  1 ubuntu ubuntu        0 Feb 27 08:57 system-HintsColumnFamily-tmp-hf-25-Index.db


/Janne

On Mar 12, 2013, at 08:07 , aaron morton <aaron@thelastpickle.com> wrote:

> What version of cassandra are you using?
> I would stop each node and delete the hints. If it happens again I could either indicate
a failing disk or a bug. 
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 11/03/2013, at 2:13 PM, Robert Coli <robert.d.a.coli@gmail.com> wrote:
> 
>> On Mon, Mar 11, 2013 at 7:05 AM, Janne Jalkanen
>> <janne.jalkanen@ecyrd.com> wrote:
>>> I keep seeing these in my log.  Three-node cluster, one node is working fine,
but two other nodes have increased latencies and these in the error logs (might of course
be unrelated). No obvious GC pressure, no disk errors that I can see.  Ubuntu 12.04 on EC2,
Java 7. Repair is run regularly.
>>> 
>>> My two questions: 1) should I worry, and 2) what might be going on, and 3) is
there any way to get rid of these? Can I just blow my HintedHandoff table to smithereens?
>> 
>> http://svn.apache.org/repos/asf/cassandra/trunk/src/java/org/apache/cassandra/io/sstable/IndexHelper.java
>> "
>> public static Filter defreezeBloomFilter(FileDataInput file, long
>> maxSize, boolean useOldBuffer) throws IOException
>>    {
>>        int size = file.readInt();
>>        if (size > maxSize || size <= 0)
>>            throw new EOFException("bloom filter claims to be " + size
>> + " bytes, longer than entire row size " + maxSize);
>>        ByteBuffer bytes = file.readBytes(size);
>> "
>> 
>> Based on the above, I would suspect either a zero byte -Filter.db file
>> or a corrupt one. Probably worry a little bit, but only a little bit
>> unless your cluster is RF=1.
>> 
>> =Rob
> 


Mime
View raw message