cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jean Carlo <jean.jeancar...@gmail.com>
Subject Re: Sync failed between in AntiEntropySessions - Repair
Date Fri, 05 Aug 2016 18:23:09 GMT
Thank you very much Paulo

On Aug 5, 2016 17:31, "Paulo Motta" <pauloricardomg@gmail.com> wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-11840
>
> increase streaming_socket_timeout to 86400000 or upgrade to
> cassandra-2.1.15.
>
> 2016-08-05 12:28 GMT-03:00 Jean Carlo <jean.jeancarl48@gmail.com>:
>
>>
>> Hello Paulo,
>>
>> Thx for your fast replay.
>>
>> You are right about that node, I did not see it that fast. In this node
>> we have errors of .SocketTimeoutException: null
>>
>>
>>
>> ERROR [STREAM-IN-/192.168.0.146] 2016-08-04 19:10:59,456
>> StreamSession.java:505 - [Stream #06c02460-5a5e-11e6-8e9a-a5bf51981ad8]
>> Streaming error occurred
>> java.net.SocketTimeoutException: null
>>         at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:211)
>> ~[na:1.8.0_60]
>>         at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>> ~[na:1.8.0_60]
>>         at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> ~[na:1.8.0_60]
>>         at org.apache.cassandra.streaming.messages.StreamMessage.
>> deserialize(StreamMessage.java:51) ~[apache-cassandra-2.1.14.jar:2.1.14]
>>         at org.apache.cassandra.streaming.ConnectionHandler$IncomingMes
>> sageHandler.run(ConnectionHandler.java:257)
>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>
>> We have this error with at least 5 more nodes in the same log. But the
>> error doesn't say that much
>>
>>
>> Saludos
>>
>> Jean Carlo
>>
>> "The best way to predict the future is to invent it" Alan Kay
>>
>> On Fri, Aug 5, 2016 at 5:16 PM, Paulo Motta <pauloricardomg@gmail.com>
>> wrote:
>>
>>> you need to check 192.168.0.36/10.234.86.36 for streaming ERRORS
>>>
>>> 2016-08-05 12:08 GMT-03:00 Jean Carlo <jean.jeancarl48@gmail.com>:
>>>
>>>> Hi Paulo
>>>>
>>>> I found the lines, we got an exception "Outgoing stream handler has
>>>> been closed" these are
>>>>
>>>> ERROR [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>>>> StreamSession.java:621 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Remote peer 192.168.0.36 failed stream session.
>>>> INFO  [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,772
>>>> StreamResultFuture.java:180 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Session with /192.168.0.36 is complete
>>>> WARN  [STREAM-IN-/10.234.86.36] 2016-08-04 16:55:53,773
>>>> StreamResultFuture.java:207 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Stream failed
>>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>>>> StreamReceiveTask.java:183 - Error applying streamed data:
>>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>>>>         at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>>>> nRunnable.run(StreamReceiveTask.java:179)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> [na:1.8.0_60]
>>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> [na:1.8.0_60]
>>>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>>
>>>> ERROR [StreamReceiveTask:107995] 2016-08-04 16:55:53,782
>>>> StreamSession.java:505 - [Stream #c4e79260-5a46-11e6-9993-e11d93fd5b40]
>>>> Streaming error occurred
>>>> java.lang.RuntimeException: Outgoing stream handler has been closed
>>>>         at org.apache.cassandra.streaming.ConnectionHandler.sendMessage(ConnectionHandler.java:138)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:697)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:653)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletio
>>>> nRunnable.run(StreamReceiveTask.java:179)
>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>>> [na:1.8.0_60]
>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> [na:1.8.0_60]
>>>>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>>> [na:1.8.0_60]
>>>>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>>> [na:1.8.0_60]
>>>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>>
>>>> I al looking through the changes files to see if it is a bug fixed in
>>>> the 2.1.15
>>>>
>>>>
>>>>
>>>> Saludos
>>>>
>>>> Jean Carlo
>>>>
>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>
>>>> On Fri, Aug 5, 2016 at 4:24 PM, Paulo Motta <pauloricardomg@gmail.com>
>>>> wrote:
>>>>
>>>>> It seems you have a streaming error, look for ERROR statement in the
>>>>> streaming classes before that which may give you a more specific root
>>>>> cause. In any case, I'd suggest you to upgrade to 2.1.15 as there were
a
>>>>> couple of streaming fixes on this version that might help.
>>>>>
>>>>> 2016-08-05 11:15 GMT-03:00 Jean Carlo <jean.jeancarl48@gmail.com>:
>>>>>
>>>>>>
>>>>>> Hi guys, Doing a repair I got this error for 2 tokenranges.
>>>>>>
>>>>>> ERROR [Thread-2499244] 2016-08-04 20:05:24,288
>>>>>> StorageService.java:3068 - Repair session 41e4bab0-5a63-11e6-9993-e11d93fd5b40
>>>>>> for range (487410372471205090,492009442088088379] failed with error
>>>>>> org.apache.cassandra.exceptions.RepairExcep
>>>>>> tion: [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 192.168.0.144 and /192.168.0.37
>>>>>> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
>>>>>> org.apache.cassandra.exceptions.RepairException: [repair
>>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 192.168.0.144 and /192.168.0.37
>>>>>>         at java.util.concurrent.FutureTask.report(FutureTask.java:122)
>>>>>> [na:1.8.0_60]
>>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:192)
>>>>>> [na:1.8.0_60]
>>>>>>         at org.apache.cassandra.service.S
>>>>>> torageService$4.runMayThrow(StorageService.java:3059)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>>         at org.apache.cassandra.utils.Wra
>>>>>> ppedRunnable.run(WrappedRunnable.java:28)
>>>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>>>         at java.util.concurrent.Executors
>>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
>>>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> [na:1.8.0_60]
>>>>>>         at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60]
>>>>>> Caused by: java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
>>>>>> [repair #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 10.234.72.144 and
>>>>>>  /10.234.86.37
>>>>>>         at com.google.common.base.Throwab
>>>>>> les.propagate(Throwables.java:160) ~[guava-16.0.jar:na]
>>>>>>         at org.apache.cassandra.utils.Wra
>>>>>> ppedRunnable.run(WrappedRunnable.java:32)
>>>>>> [apache-cassandra-2.1.14.jar:2.1.14]
>>>>>>         at java.util.concurrent.Executors
>>>>>> $RunnableAdapter.call(Executors.java:511) [na:1.8.0_60]
>>>>>>         at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>>>> [na:1.8.0_60]
>>>>>>         at java.util.concurrent.ThreadPoo
>>>>>> lExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_60]
>>>>>>         at java.util.concurrent.ThreadPoo
>>>>>> lExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_60]
>>>>>>         ... 1 common frames omitted
>>>>>> Caused by: org.apache.cassandra.exceptions.RepairException: [repair
>>>>>> #41e4bab0-5a63-11e6-9993-e11d93fd5b40 on ks/cf_adv,
>>>>>> (487410372471205090,492009442088088379]] Sync failed between /
>>>>>> 192.168.0.144 and /192.168.0.37
>>>>>>         at org.apache.cassandra.repair.Re
>>>>>> pairSession.syncComplete(RepairSession.java:223)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>>         at org.apache.cassandra.service.A
>>>>>> ctiveRepairService.handleMessage(ActiveRepairService.java:422)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>>         at org.apache.cassandra.repair.Re
>>>>>> pairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>>         at org.apache.cassandra.net.Messa
>>>>>> geDeliveryTask.run(MessageDeliveryTask.java:64)
>>>>>> ~[apache-cassandra-2.1.14.jar:2.1.14]
>>>>>>         ... 3 common frames omitted
>>>>>>
>>>>>> Is it a network problem or we are having a bug ? There is nothing
>>>>>> else in the log of cassandra more than
>>>>>> Sync failed between.
>>>>>>
>>>>>> We are having cassandra 2.1.14
>>>>>>
>>>>>> Best regards
>>>>>>
>>>>>> Jean Carlo
>>>>>>
>>>>>> "The best way to predict the future is to invent it" Alan Kay
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message