cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis Lauthier <alx7539-cassan...@yahoo.com>
Subject Re : Compressed families not created on new node
Date Tue, 17 Jan 2012 23:37:24 GMT
That was worth a try. Unfortunately it failed in exactly the same way as before.

I had no trouble decommissioning the bad node. Only the good node remained. When I bootstrapped
an empty new node, the schema migration failed again with UnserializableColumnFamilyException.

So it looks like the I/O errors are not he direct cause of the schema migration failure. I
guess the cluster got corrupted at some point.

Since I can't risk remaining in the current situation for too long, I'm going to consider
the current cluster is dead. I will install a completely separate cluster on the new node,
create an empty schema, then stop the old node and copy its sstable data to the new node.
My site will be down for less than two hours if everything goes smoothly.

Thank you for your help, Aaron. If someone is interested in investigating this "cluster corruption"
problem, I'll be happy to provide more info.


Alexis Lauthier



________________________________
 De : aaron morton <aaron@thelastpickle.com>
À : user@cassandra.apache.org 
Envoyé le : Mardi 17 Janvier 2012 1h49
Objet : Re: Compressed families not created on new node
 

eeek, HW errors. 

I would guess (thats all it is) that an IO error may have stopped the schema from migrating. 

Stop cassandra on that node and copy the files off as best you can. 

I would then try a node replacement

First remove the failed new node with nodetool decomission or removetoken. 

You are now down to one server. 

Copy the yaml file from the old machine (with IO errors) to a new one. To make things potentially
less complicated bump the initial token slightly (e.g. add 1 to it) so the new node will not
be exactly replacing the old one. 

Now start the new node. The other node will notice the schema is out of date and send it across. 

Once all the CF's are added and the schema's match stop the new node, copy the SSTable data
from the old node to the new one and restart it. There are other ways to do this, this is
the simplest though. 

With the new node in place the ring should now show the IO error node as down, the new node
with a token very close to the IO error node and the one other node. 

You can now remove the IO error node with decomission or removetoken. 

Now run a repair from the new node.

At any stage the rollback plan is to simply turn the IO error node back on. 

Hope that helps. 

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com 

On 16/01/2012, at 10:41 PM, Alexis Lauthier wrote:

The I/O errors are caused by disk failure. Syslog contains some of those things:
>
>
>
>Jan 16 09:53:24 --- kernel: [7065781.460804] sd 4:0:0:0: [sda]  Add. Sense: Unrecovered
read error
>Jan 16 09:53:24 --- kernel: [7065781.460810] sd 4:0:0:0: [sda] CDB: Read(10): 28 00 11
cf 60 70 00 00 08 00
>Jan 16 09:53:24 --- kernel: [7065781.460820] end_request: I/O error, dev sda, sector 298803312
>
>
>
>
>Scrub failed:
>
>
><a lot of lines saying it is scrubbing happily up to this point>
>
> INFO [CompactionExecutor:5818] 2012-01-16 09:45:20,650 CompactionManager.java (line
477) Scrubbing SSTableReader(path='/home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db')
>ERROR
 [CompactionExecutor:5818] 2012-01-16 09:47:51,531 PrecompactedRow.java (line 119) Skipping
row DecoratedKey(Token(bytes[01f9332e566a3a8d5a1cc17e530ae46e]), 01f9332e566a3a8d5a1cc17e530ae46e)
in /home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db
>java.io.IOException: (/home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db) failed to read
13705 bytes from offset 3193541.
>    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:87)
>    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:75)
>    at org.apache.cassandra.io.util.RandomAccessReader.read(RandomAccessReader.java:302)
>    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:397)
>    at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
>    at
 org.apache.cassandra.utils.BytesReadTracker.readFully(BytesReadTracker.java:95)
>    at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:392)
>    at org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:354)
>    at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:120)
>    at org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:37)
>    at org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:147)
>    at org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:232)
>    at org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:115)
>    at
 org.apache.cassandra.db.compaction.PrecompactedRow.<init>(PrecompactedRow.java:102)
>    at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:133)
>    at org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:139)
>    at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:565)
>    at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:472)
>    at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:63)
>    at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:224)
>    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>    at
 java.util.concurrent.FutureTask.run(FutureTask.java:138)
>    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>    at java.lang.Thread.run(Thread.java:662)
> WARN [CompactionExecutor:5818] 2012-01-16 09:47:51,531 CompactionManager.java (line
581) Non-fatal error reading row (stacktrace follows)
>java.lang.NullPointerException
> WARN [CompactionExecutor:5818] 2012-01-16 09:47:51,532 CompactionManager.java (line
623) Row at 14740167 is unreadable; skipping to next
>ERROR [CompactionExecutor:5818] 2012-01-16 09:53:24,395 AbstractCassandraDaemon.java (line
133) Fatal exception in thread Thread[CompactionExecutor:5818,1,RMI Runtime]
>java.io.IOException: (/home/cassprod/data/ptprod/UrlInfo-hb-1326-Data.db) failed to read
13705 bytes from offset
 3193541.
>    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.decompressChunk(CompressedRandomAccessReader.java:87)
>    at org.apache.cassandra.io.compress.CompressedRandomAccessReader.reBuffer(CompressedRandomAccessReader.java:75)
>    at org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:259)
>    at org.apache.cassandra.db.compaction.CompactionManager.scrubOne(CompactionManager.java:625)
>    at org.apache.cassandra.db.compaction.CompactionManager.doScrub(CompactionManager.java:472)
>    at org.apache.cassandra.db.compaction.CompactionManager.access$300(CompactionManager.java:63)
>    at org.apache.cassandra.db.compaction.CompactionManager$3.call(CompactionManager.java:224)
>    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
>    at
 java.util.concurrent.FutureTask.run(FutureTask.java:138)
>    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>    at java.lang.Thread.run(Thread.java:662)
>
>
>
>
>The same kind of "failed to read" IOExceptions have been routinely logged for 13 days
now.
>
>
>The best idea I can come up with now is: decommission the failing node, then add the new
node. And hope the schema will fully replicate. This will leave me with only one node for
a time, and I'm not sure it will play nice with replication_factor=2.
>This feels a lot like jumping out of a plane with an untested parachute. So any other
ideas ?
>
>
>Thanks,
>Alexis Lauthier
>
>
>
>
>
>
>
>
>________________________________
> De : aaron morton <aaron@thelastpickle.com>
>À : user@cassandra.apache.org 
>Envoyé le : Lundi 16 Janvier 2012 1h05
>Objet : Re: Compressed families not created on new node
> 
>
>Without knowing what the IOErrors are I would do the following:
>
>
>
>
>nodetool scrub to fix any on disk errors, this will also take a snapshot you can use for
rollback.  
>
>nodetool repair to ensure data is consistent. 
>
>
>Hope that helps. 
>
>
>-----------------
>Aaron Morton
>Freelance Developer
>@aaronmorton
>http://www.thelastpickle.com 
>
>On 16/01/2012, at 7:53 AM, Alexis Lauthier wrote:
>
>I see only one line "JOINING: sleeping 30000 ms for pending range setup".
>>
>>
>>Before that, I have a lot of migration INFO messages, only for the uncompressed families.
>>
>>
>>I have currently killed the new node, so a describe cluster shows only one schema
version on the first two nodes.
>>
>>
>>
>>[default@unknown] describe cluster;
>>Cluster Information:
>>   Snitch: org.apache.cassandra.locator.SimpleSnitch
>>   Partitioner: org.apache.cassandra.dht.ByteOrderedPartitioner
>>   Schema versions: 
>>    b42595d0-2247-11e1-0000-a0e9ff9ab7bf: [---.36,
 ---.35]
>>
>>    UNREACHABLE: [---.56]
>>
>>
>>
>>
>>Also, on one of the old nodes, I have a lot of I/O errors on the data files for some
(but not all) of the compressed families. It began a few days ago. All "nodetool repair" calls
have been blocking since then.
>>
>>
>>Any ideas on how I can get the data on the new node, before the old one dies?
>>
>>
>>
>>
>>
>>Thanks,
>>Alexis Lauthier
>>
>>
>>
>>
>>
>>
>>________________________________
>> De : aaron morton <aaron@>
>>À : user@cassandra.apache.org 
>>Envoyé le : Dimanche 15 Janvier 2012 19h17
>>Objet : Re: Compressed families not created on new node
>> 
>>
>>Sounds like the schema has not fully migrated to the new node. It is applied to the
joining node one change at a time. A quick scan of the changes file does not find anything
fixed after 1.0.3
>>
>>
>>You can check schema versions in the CLI using the describe cluster command. 
>>
>>
>>Check for errors in the logs with Migration in the text.  
>>
>>
>>Are you seeing this line a lot in the log ? 
>> INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) JOINING: sleeping
30000 ms for pending range setup
>>>
>>
>>cheers
>>
>>
>>-----------------
>>Aaron Morton
>>Freelance Developer
>>@aaronmorton
>>http://www.thelastpickle.com 
>>
>>On 14/01/2012, at 4:20 AM, Alexis Lauthier wrote:
>>
>>I'm using Cassandra 1.0.3 on a 2 nodes cluster. My schema (with replication_factor=2)
contains both compressed (with sstable_compression=DeflateCompressor) and uncompressed column
families.
>>>
>>>
>>>
>>>When bootstrapping a third node, the uncompressed families are created on the
new node as expected, but the compressed families are not. Only the uncompressed families
appear in a "show schema", and the new node data size is small, which is consistent with the
big compressed data not being there.
>>>
>>>
>>>I'm seeing frequent exceptions in the log :
>>>
>>>
>>> INFO [main] 2012-01-13 14:55:00,493 StorageService.java (line 616) JOINING:
sleeping 30000 ms for pending range setup
>>>ERROR [MutationStage:1] 2012-01-13 14:55:01,511 RowMutationVerbHandler.java (line
65) Error in row
 mutation
>>>org.apache.cassandra.db.UnserializableColumnFamilyException: Couldn't find cfId=1008
>>>    at org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:129)
>>>    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:401)
>>>    at org.apache.cassandra.db.RowMutation$RowMutationSerializer.deserialize(RowMutation.java:409)
>>>    at org.apache.cassandra.db.RowMutation.fromBytes(RowMutation.java:357)
>>>    at org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:46)
>>>    at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>    at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>    at java.lang.Thread.run(Thread.java:722)
>>>
>>>
>>>
>>>
>>>
>>>After a few minutes, The column family names are shown instead of their ids ("UrlText"
is one of the compressed families) :
>>>
>>>
>>>
>>>ERROR [ReadStage:46] 2012-01-13 14:59:33,924 AbstractCassandraDaemon.java (line
133) Fatal exception in thread Thread[ReadStage:46,5,main]
>>>java.lang.IllegalArgumentException: Unknown ColumnFamily UrlText in keyspace ptprod
>>>    at org.apache.cassandra.config.Schema.getComparator(Schema.java:226)
>>>    at org.apache.cassandra.db.ColumnFamily.getComparatorFor(ColumnFamily.java:300)
>>>    at org.apache.cassandra.db.ReadCommand.getComparator(ReadCommand.java:92)
>>>    at
 org.apache.cassandra.db.SliceByNamesReadCommand.<init>(SliceByNamesReadCommand.java:44)
>>>    at org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:106)
>>>    at org.apache.cassandra.db.SliceByNamesReadCommandSerializer.deserialize(SliceByNamesReadCommand.java:74)
>>>    at org.apache.cassandra.db.ReadCommandSerializer.deserialize(ReadCommand.java:132)
>>>    at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:51)
>>>    at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
>>>    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
>>>    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>    at
 java.lang.Thread.run(Thread.java:722)
>>>
>>>
>>>
>>>
>>>
>>>How can I get the compressed families on the new node ?
>>>
>>>
>>>Thanks,
>>>Alexis Lauthier
>>>
>>
>>
>>
>
>
>
Mime
View raw message