cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Héctor Izquierdo Seliva <izquie...@strands.com>
Subject Re: Problems recovering a dead node
Date Wed, 04 May 2011 09:25:47 GMT
I'm sorry but I can't provide more detailed info as I have restarted the
node. After that the number of pending tasks started at 40, and rapidly
went down as compactions finished. After that, the ring looks ok, with
all the nodes having about the same amount of data. There were no errors
in the node log, only info messages. 

Should I run repair again just in case? The next time I have to recover
a node, is there a safer/faster way of doing it?

My guess about the number of pending tasks and sstables is that during
repair, it seemed to ask for ranges of the same column family a lot of
times, thus yielding a lot of tiny sstables. This caused minor
compactions to pile up. I have read about never ending repairs, or
repairs done a lot of times over small amounts of data on the mailing
lists. Could this be what happened?


Thanks!


El mié, 04-05-2011 a las 21:02 +1200, aaron morton escribió:
> Certainly sounds a bit sick. 
> 
> The first error looks like it happens when the index file points to the wrong place in
the data file for the SSTable. The second one happens when the index file is corrupted. The
should be problems nodetool scrub can fix.
> 
> The disk space may be dead space to cassandra compaction or some other streaming failure.
You can check how much it considers to be live (in use) space using nodetool cfstats. This
will also tell you how many sstables are live. Having a lot of dead SSTables is not necessarily
a bad thing. 
> 
> What are the pending tasks ? what is nodetool tpstats showing ? And what does nodetool
ring show from one of the other nodes ? 
> 
> I'm assuming there are no errors in the logs on the node. What are the most recent INFO
messages?
> 
> Hope that helps. 
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 4 May 2011, at 17:54, Héctor Izquierdo Seliva wrote:
> 
> > 
> > Hi Aaron
> > 
> > It has no data files whatsoever. The upgrade path is 0.7.4 -> 0.7.5. It
> > turns out the initial problem was the sw raid failing silently because
> > of another faulty disk.
> > 
> > Now that the storage is working, I brought up the node again, same IP,
> > same token and tried doing nodetool repair. 
> > 
> > All adjacent nodes have finished the streaming session, and now the node
> > has a total of 248 GB of data. Is this normal when the load per node is
> > about 18GB? 
> > 
> > Also there are 1245 pending tasks. It's been compacting or rebuilding
> > sstables for the last 8 hours non stop. There are 2057 sstables in the
> > data folder.
> > 
> > Should I have done thing differently or is this the normal behaviour?
> > 
> > Thanks!
> > 
> > El mié, 04-05-2011 a las 07:54 +1200, aaron morton escribió:
> >> When you say "it's clean" does that mean the node has no data files ?
> >> 
> >> After you replaced the disk what process did you use to recover  ?
> >> 
> >> Also what version are you running and what's the recent upgrade history ?
> >> 
> >> Cheers
> >> Aaron
> >> 
> >> On 3 May 2011, at 23:09, Héctor Izquierdo Seliva wrote:
> >> 
> >>> Hi everyone. One of the nodes in my 6 node cluster died with disk
> >>> failures. I have replaced the disks, and it's clean. It has the same
> >>> configuration (same ip, same token).
> >>> 
> >>> When I try to restart the node it starts to throw mmap underflow
> >>> exceptions till it closes again.
> >>> 
> >>> I tried setting io to standard, but it still fails. It gives errors
> >>> about two decorated keys being different, and the EOFException.
> >>> 
> >>> Here is an excerpt of the log
> >>> 
> >>> http://pastebin.com/ZXW1wY6T
> >>> 
> >>> I can provide more info if needed. I'm at a loss here so any help is
> >>> appreciated.
> >>> 
> >>> Thanks all for your time
> >>> 
> >>> Héctor Izquierdo
> >>> 
> >> 
> > 
> > 
> 



Mime
View raw message