cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eduardo Cusa <eduardo.c...@gmail.com>
Subject Re: [cassandra 2.1.3] Missing host ID
Date Thu, 08 Oct 2015 18:58:40 GMT
Hello Paulo, this issue started today and always happened in the same node.

Run the following command seems to solve the problem :

$ nodetool truncatehints


Now the node is up.

Regards

Eduardo




2015-10-08 15:41 GMT-03:00 Paulo Motta <pauloricardomg@gmail.com>:

> Hello Eduardo,
>
> Your node is trying to write a hint to another node (after a timed out
> write), but because of some race condition it does not have its token table
> updated soon after startup, so it cannot locate the node with that ID. You
> should not be worried, as the only consequence is that one hint was lost
> and data consistency can be fixed with a simple repair (or during read
> repairs).
>
> Some other people have reported a similar condition so I opened a JIRA
> ticket: https://issues.apache.org/jira/browse/CASSANDRA-10485
>
> Some questions to help troubleshooting:
>
> - Does it happen always with the same node or any node that you restart?
> - Was that node ever replaced or upgraded?
> - With what frequency does it happen?
>
> Thanks,
>
> Paulo
>
> 2015-10-08 10:45 GMT-07:00 Eduardo Cusa <eduardo.cusa@gmail.com>:
>
>> Hi Guys, I have a custer with 12 nodes.
>>
>> when I restart one of them I receive the error "Missing host ID":
>>
>>
>>
>> WARN  [SharedPool-Worker-1] 2015-10-08 13:15:33,882
>> AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread
>> Thread[SharedPool-Worker-1,5,main]: {}
>> java.lang.AssertionError: Missing host ID for 63.251.156.141
>>         at
>> org.apache.cassandra.service.StorageProxy.writeHintForMutation(StorageProxy.java:978)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>         at
>> org.apache.cassandra.service.StorageProxy$6.runMayThrow(StorageProxy.java:950)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>         at
>> org.apache.cassandra.service.StorageProxy$HintRunnable.run(StorageProxy.java:2235)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>         at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_60]
>>         at
>> org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
>> ~[apache-cassandra-2.1.3.jar:2.1.3]
>>         at
>> org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
>> [apache-cassandra-2.1.3.jar:2.1.3]
>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>>
>>
>>
>> If I made nodetool status, the problematic node has ID:
>>
>> UN  10.10.10.12  1.3 TB     1       ?
>> 4d5c8fd2-a909-4f09-a23c-4cd6040f338a  rack3
>>
>>
>>
>>
>> Any idea what could be happening?
>>
>>
>> Regards
>> Eduardo
>>
>>
>>
>

Mime
View raw message