cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sasha Dolgy <sdo...@gmail.com>
Subject Re: odd logs after repair
Date Mon, 13 Jun 2011 13:34:01 GMT
Hi Aaron,

The error is being reported on all 4 nodes. I have confirmed (for my
own sanity) that each node is running:  ReleaseVersion: 0.8.0

I can reproduce the error on any node by trailing
cassandra/logs/system.log and running "nodetool repair"

 INFO [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13
21:28:39,877 AntiEntropyService.java (line 177) Excluding
/10.128.34.18 from repair because it is on version 0.7 or sooner. You
should consider updating this node before running repair again.
ERROR [manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec] 2011-06-13
21:28:39,877 AbstractCassandraDaemon.java (line 113) Fatal exception
in thread Thread[manual-repair-1c6b33bc-ef14-4ec8-94f6-f1464ec8bdec,5,RMI
Runtime]
java.util.ConcurrentModificationException
        at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
        at java.util.HashMap$KeyIterator.next(HashMap.java:828)
        at org.apache.cassandra.service.AntiEntropyService.getNeighbors(AntiEntropyService.java:173)
        at org.apache.cassandra.service.AntiEntropyService$RepairSession.run(AntiEntropyService.java:776)

When I run nodetool ring, the ring looks balanced and nothing out of sorts.

I also have this set up with RF=3 on 4 nodes ... but repair was
working fine prior to the 0.8.0 upgrade.

Are there any special commands I need to run?  I've tried scrub,
cleanup, flush too ... still, repair gives the same issues.

-- I have stopped one of the nodes and started it.  Issue still
persists.  I stop another node that is reported in the logs (like .18
above) and start it ... run repair again ... issue is persisted to the
log file still.

-sd



On Mon, Jun 13, 2011 at 3:02 PM, aaron morton <aaron@thelastpickle.com> wrote:
> You can double check with node tool e.g.
>
> $ ./bin/nodetool -h localhost version
> ReleaseVersion: 0.8.0-SNAPSHOT
>
> This error is about the internode wire protocol one node thinks another is using. Not
sure how it could get confused, does it go away if you restart the node that logged the error
?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 13 Jun 2011, at 06:19, Sasha Dolgy wrote:
>
>> Hi Everyone,
>>
>> Last week, upgraded all 4 nodes to apache-cassandra-0.8.0 .. no
>> issues.  Trolling the logs today, I find messages like this on all
>> four nodes:
>>
>> INFO [manual-repair-0b61c9e2-3593-4633-a80f-b6ca52cfe948] 2011-06-13
>> 02:16:45,978 AntiEntropyService.java (line 177) Excluding
>> /10.128.34.18 from repair because it is on version 0.7 or sooner. You
>> should consider updating this node before running repair again.
>>
>> Maybe it would be nice to have the version of all nodes print in
>> nodetool ring ?  I don't think I'm crazy though ... have manually
>> checked all are on 0.8.0
>>
>>
>> --
>> Sasha Dolgy
>> sasha.dolgy@gmail.com
>
>



-- 
Sasha Dolgy
sasha.dolgy@gmail.com

Mime
View raw message