0.7.4/ 3 node cluster/ RF -3 /Quorum read/write

After I re-introduced a corrupted node, followed the process as (thanks to folks on the mailing list for helping me) listed on the operations wiki to handle failures.
Still doing a cleanup on one node at this point. But I noticed that I am seeing this same exception appear 10/12 times in a minute, on an existing node (not the new one). I think it started around the removetoken.

How do I solve this, should I just restart this node? Any other cleanups/resets I need to do?

Thanks


On Thu, Apr 28, 2011 at 2:26 AM, aaron morton <aaron@thelastpickle.com> wrote:
I *think* that code is used when one node tells others via gossip it is removing a token that is not it's own. The ode that receives information in gossip does some work and then replies to the first node with a REPLICATION_FINISHED message, which is the node I assume the error is happening on.

Have you been doing any moves / removes or additions or tokens/nodes?

Thanks
Aaron

On 28 Apr 2011, at 08:39, Alexis L-Quc wrote:

> Hi,
>
> I've been getting the following lately, every few seconds.
>
> 2011-04-27T20:21:18.299885+00:00 10.202.61.193 [MiscStage: 97] Error
> in ThreadPoolExecutor
> 2011-04-27T20:21:18.299885+00:00 10.202.61.193 java.lang.AssertionError
> 2011-04-27T20:21:18.300038+00:00 10.202.61.193 10.202.61.193 at
> org.apache.cassandra.service.StorageService.confirmReplication(StorageService.java:1872)
> 2011-04-27T20:21:18.300038+00:00 10.202.61.193 10.202.61.193 at
> org.apache.cassandra.streaming.ReplicationFinishedVerbHandler.doVerb(ReplicationFinishedVerbHandler.java:38)
> 2011-04-27T20:21:18.300047+00:00 10.202.61.193 10.202.61.193 at
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:72)
> 2011-04-27T20:21:18.300047+00:00 10.202.61.193 10.202.61.193 at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> 2011-04-27T20:21:18.300055+00:00 10.202.61.193 10.202.61.193 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> 2011-04-27T20:21:18.300055+00:00 10.202.61.193 10.202.61.193 at
> java.lang.Thread.run(Thread.java:636)
> 2011-04-27T20:21:18.300555+00:00 10.202.61.193 [MiscStage: 97] Fatal
> exception in thread Thread[MiscStage:97,5,main]
>
> I see it coming from
> 32 public class ReplicationFinishedVerbHandler implements IVerbHandler
> 33 {
> 34 private static Logger logger =
> LoggerFactory.getLogger(ReplicationFinishedVerbHandler.class);
> 35
> 36 public void doVerb(Message msg, String id)
> 37 {
> 38 StorageService.instance.confirmReplication(msg.getFrom());
> 39 Message response =
> msg.getInternalReply(ArrayUtils.EMPTY_BYTE_ARRAY);
> 40 if (logger.isDebugEnabled())
> 41 logger.debug("Replying to " + id + "@" + msg.getFrom());
> 42 MessagingService.instance().sendReply(response, id, msg.getFrom());
> 43 }
> 44 }
>
> Before I dig deeper in the code, has anybody dealt with this before?
>
> Thanks,
>
> --
> Alexis L-Quc