cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alain RODRIGUEZ <arodr...@gmail.com>
Subject Re: Experiencing Timeouts on one node
Date Thu, 02 Jul 2015 16:11:26 GMT
Hi,

I am not sure about what is happening (I have never seen this error
before). Yet from
https://github.com/apache/cassandra/blob/cassandra-1.2/CHANGES.txt  it
looks like some bugs were fixed in late revision of 1.2.x.

I would advice you upgrading to last 1.2.19 (It is an old and stable
version, I see no reason not doing it).

"The problem goes away if we reboot the node, and then reappears after
seven days"
--> Have you TTLs on any table ? (Set to 7 days ?)
--> Do you see any GC warn / heap pressures ?

C*heers,

Alain



2015-07-02 16:20 GMT+02:00 Shashi Yachavaram <shashi007@gmail.com>:

> Jason,
>
> The load was evenly distributed. And regarding network connectivity, our
> applications were successfully able to connect to the node, but the read
> and write operations were timing out. Also we were able to ssh to this
> node.
>
> I just pasted  "/bin/nodetool -h node version" and "java -version".
>
> Thanks
> shashi
>
> On Thu, Jul 2, 2015 at 8:42 AM, Jason Wee <peichieh@gmail.com> wrote:
>
>> you should check the network connectivity for this node and also its
>> system average load. is that typo or literary what it is, cassandra
>> 1.2.15.*1* and java 6 update *85* ?
>>
>>
>>
>> On Thu, Jul 2, 2015 at 12:59 AM, Shashi Yachavaram <shashi007@gmail.com>
>> wrote:
>>
>>> We have a 28 node cluster, out of which only one node is experiencing
>>> timeouts.
>>> We thought it was the raid, but there are two other nodes on the same
>>> raid without
>>> any problem. Also The problem goes away if we reboot the node, and then
>>> reappears
>>> after seven  days. The following hinted hand-off timeouts are seen on
>>> the node
>>> experiencing the timeouts. Also we did not notice any gossip errors.
>>>
>>> I was wondering if anyone has seen this issue and how they resolved it.
>>>
>>> Cassandra Version: 1.2.15.1
>>> OS: Linux cm 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST
>>> 2014 x86_64 x86_64 x86_64 GNU/Linux
>>> java version "1.6.0_85"
>>>
>>>
>>> ------------------------------------------------------------------------------------------------------------------------------------
>>> INFO [HintedHandoff:2] 2015-06-17 22:52:08,130 HintedHandOffManager.java
>>> (line 296) Started hinted handoff for host:
>>> 4fe86051-6bca-4c28-b09c-1b0f073c1588 with IP: /192.168.1.122
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:08,131
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> bbf0878b-b405-4518-b649-f6cf7c9a6550 with IP: /192.168.1.119
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,634
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.122; aborting (0 delivered)
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:17,635
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> f7b7ab10-4d42-4f0c-af92-2934a075bee3 with IP: /192.168.1.108
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.119; aborting (0 delivered)
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:17,643
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> ddb79f35-3e2b-4be8-84d8-7942086e2b73 with IP: /192.168.1.104
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,143
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.108; aborting (0 delivered)
>>>  INFO [HintedHandoff:2] 2015-06-17 22:52:27,144
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> 6a2fa431-4a51-44cb-af19-1991c960e075 with IP: /192.168.1.117
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,153
>>> HintedHandOffManager.java (line 422) Timed out replaying hints to /
>>> 192.168.1.104; aborting (0 delivered)
>>>  INFO [HintedHandoff:1] 2015-06-17 22:52:27,154
>>> HintedHandOffManager.java (line 296) Started hinted handoff for host:
>>> cf03174a-533c-44d6-a679-e70090ad2bc5 with IP: /192.168.1.107
>>>
>>> ------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> Thanks
>>> -shashi..
>>>
>>
>>
>

Mime
View raw message