cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Brown (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-13649) Uncaught exceptions in Netty pipeline
Date Fri, 30 Jun 2017 19:35:00 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070620#comment-16070620
] 

Jason Brown commented on CASSANDRA-13649:
-----------------------------------------

It's a netty common practice to include an exception handler at the end of a netty pipeline
to handle cases like this. However, I'm reticent to add yet another handler to the pipeline
as some of my testing for CASSANDRA-8457 (admittedly, very early-stage testing) showed that
we spend extra time in the pipeline just by all the mechanics around invoking another handler
(checking the promise, state of the channel, and so on).

That being said, I can probably find some time to reinvestigate as part of finalizing all
the netty-related things for 4.0. [~spodxx@gmail.com] feel free to assign to me if you like,
but I probably can't get to it for about a month.

> Uncaught exceptions in Netty pipeline
> -------------------------------------
>
>                 Key: CASSANDRA-13649
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13649
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Stefan Podkowinski
>         Attachments: test_stdout.txt
>
>
> I've noticed some netty related errors in trunk in [some of the dtest results|https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-devbranch-dtest/106/#showFailuresLink].
Just want to make sure that we don't have to change anything related to the exception handling
in our pipeline and that this isn't a netty issue. Actually if this causes flakiness but is
otherwise harmless, we should do something about it, even if it's just on the dtest side.
> {noformat}
> WARN  [epollEventLoopGroup-2-9] 2017-06-28 17:23:49,699 Slf4JLogger.java:151 - An exceptionCaught()
event was fired, and it reached at the tail of the pipeline. It usually means the last handler
in the pipeline did not handle the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection
reset by peer
> 	at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> And again in another test:
> {noformat}
> WARN  [epollEventLoopGroup-2-8] 2017-06-29 02:27:31,300 Slf4JLogger.java:151 - An exceptionCaught()
event was fired, and it reached at the tail of the pipeline. It usually means the last handler
in the pipeline did not handle the exception.
> io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: Connection
reset by peer
> 	at io.netty.channel.unix.FileDescriptor.readAddress(...)(Unknown Source) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> {noformat}
> This one looks also odd and makes upgrade_internal_auth_test.TestAuthUpgrade.upgrade_to_30_test
fail:
> {noformat}
> WARN  [epollEventLoopGroup-2-9] 2017-06-29 02:41:37,125 Slf4JLogger.java:151 - An exceptionCaught()
event was fired, and it reached at the tail of the pipeline. It usually means the last handler
in the pipeline did not handle the exception.
> io.netty.handler.codec.DecoderException: org.apache.cassandra.transport.ProtocolException:
Invalid or unsupported protocol version: 4
> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:375)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:342)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:325)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:220)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1289)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:241)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:227)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:893)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:691) ~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:399)
[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:307) [netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
> Caused by: org.apache.cassandra.transport.ProtocolException: Invalid or unsupported protocol
version: 4
> 	at org.apache.cassandra.transport.Frame$Decoder.decode(Frame.java:186) ~[main/:na]
> 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
> 	... 16 common frames omitted
> {noformat}
> /cc [~jasobrown], [~norman]
> Edit:
> The {{io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed}} error
also causes tests to fail for 3.0 and 3.11. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message