cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kyrylo Lebediev <Kyrylo_Lebed...@epam.com>
Subject Re: 3.11.2 io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)
Date Mon, 07 May 2018 07:52:10 GMT
Hi!

You are talking about  messages like below, right?


INFO  [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 - Unexpected exception
during request; channel = [id: 0xa4879fdd, L:/10.175.20.112:9042 - R:/10.175.20.73:2508]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset
by peer

        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

As far as I understand, these messages mean that some C* client (10.175.20.73 in your case)
closed connection abnormally and the issue is on client side. So, this should be investigated
on client side. I'd say if your app works fine it's up to you to investigate the issue or
not ("due diligence").


At the same time there are some messages in the log related to hinted-handoff, which could
mean your cluster suffers from write timeouts (unless these messages are caused by temporary
unavailability of  10.175.20.114 because of node reboot etc). This definitely worth investigation.
It can be caused by: non-optimal GC setting (btw messages like "ParNew GC in 309ms"  may imply
this, delay of 309 ms is too long), hw performance issues, networking issues etc.


Regards,

Kyrill


________________________________
From: Xiangfei Ni <xiangfei.ni@cm-dt.com>
Sent: Monday, May 7, 2018 5:44:21 AM
To: user@cassandra.apache.org
Subject: 3.11.2 io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source)


Hi Expert,

  I have this error in the Cassandra logs,the nodes are ok and behave as normal,

INFO  [epollEventLoopGroup-2-8] 2018-05-05 16:57:45,537 Message.java:623 - Unexpected exception
during request; channel = [id: 0xa4879fdd, L:/10.175.20.112:9042 - R:/10.175.20.73:2508]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset
by peer

        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [epollEventLoopGroup-2-9] 2018-05-05 16:57:45,537 Message.java:623 - Unexpected exception
during request; channel = [id: 0x86e1a3d9, L:/10.175.20.112:9042 - R:/10.175.20.37:44006]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset
by peer

        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

INFO  [HintsDispatcher:43] 2018-05-05 16:57:49,991 HintsStore.java:126 - Deleted hint file
5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints

INFO  [HintsDispatcher:43] 2018-05-05 16:57:49,991 HintsDispatchExecutor.java:282 - Finished
hinted handoff of file 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb-1525510648347-1.hints to endpoint
/10.175.20.114: 5696cbef-edd1-42a6-b4f1-3c2f0881b8cb

INFO  [Service Thread] 2018-05-05 16:58:01,159 GCInspector.java:284 - ParNew GC in 309ms.
 CMS Old Gen: 1258291920 -> 1258482424; Par Eden Space: 503316480 -> 0; Par Survivor
Space: 700080 -> 782256

WARN  [GossipTasks:1] 2018-05-05 16:58:23,273 Gossiper.java:791 - Gossip stage has 2 pending
tasks; skipping status check (no nodes will be marked down)

INFO  [HintsDispatcher:44] 2018-05-05 16:58:33,913 HintsStore.java:126 - Deleted hint file
21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints

INFO  [HintsDispatcher:44] 2018-05-05 16:58:33,913 HintsDispatchExecutor.java:282 - Finished
hinted handoff of file 21033fe2-dca9-42a5-8e68-496dd3517f53-1525510690633-1.hints to endpoint
/10.175.20.113: 21033fe2-dca9-42a5-8e68-496dd3517f53

INFO  [epollEventLoopGroup-2-10] 2018-05-05 16:58:41,750 Message.java:623 - Unexpected exception
during request; channel = [id: 0xc640c7a2, L:/10.175.20.112:9042 - R:/10.175.20.37:56784]

io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection reset
by peer

        at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]

the limit setting in CentOS 6.8 is setting as below :

* hard memlock unlimited

* hard nofile 100000

* soft nofile 100000

* soft memlock unlimited

* hard nproc 32768

* hard as unlimited

* soft nproc 32768

* soft as unlimited

root soft nofile 32768

root hard nofile 32768



  can I ignore this error or do you have any suggestions about this error?



Best Regards,



倪项菲/ David Ni

中移德电网络科技有限公司

Virtue Intelligent Network Ltd, co.

Add: 2003,20F No.35 Luojia creative city,Luoyu Road,Wuhan,HuBei

Mob: +86 13797007811|Tel: + 86 27 5024 2516


Mime
View raw message