cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexey Burylov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-10206) Incorrect handling of end-of stream leading to infinite loop in streaming session
Date Thu, 27 Aug 2015 15:03:46 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-10206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alexey Burylov updated CASSANDRA-10206:
---------------------------------------
    Attachment: threaddump-1440686737505.tdump
                threaddump-1440602926719.tdump
                cassandra-2.1-10206.txt

I'm upload more dump, from different server.

Also i'm write litle JMX unity what monitor thread and capture stack-traces, when found new
one. May be, it output more useful? 

Output for thread 17633 from threaddump-1440686737505.tdump

{code}
	at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
	at java.lang.Thread.run(Thread.java:745)

	at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:384)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
	at java.lang.Thread.run(Thread.java:745)

	at java.lang.System$2.blockedOn(System.java:1195)
	at java.nio.channels.spi.AbstractInterruptibleChannel.blockedOn(AbstractInterruptibleChannel.java:211)
	at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:198)
	at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:387)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
	at java.lang.Thread.run(Thread.java:745)

	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
	at java.lang.Thread.run(Thread.java:745)

	at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
	at java.nio.ByteBuffer.allocate(ByteBuffer.java:331)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:50)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
	at java.lang.Thread.run(Thread.java:745)

	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:258)
	at java.lang.Thread.run(Thread.java:745)

	at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:375)
	at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:51)
	at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:250)
	at java.lang.Thread.run(Thread.java:745)
{code}

Unity code:
{code}
		try (JMXConnector connector = JMXConnectorFactory.connect(
				new JMXServiceURL(url),
				ENVIRONMENT)) {
			MBeanServerConnection mbsc = connector.getMBeanServerConnection();
			ObjectName threading = new ObjectName("java.lang:type=Threading");
			ThreadMXBean treading = ManagementFactory.newPlatformMXBeanProxy(mbsc, "java.lang:type=Threading",
ThreadMXBean.class);


			Set<String> set = new HashSet<>();
			for (int i = 0; i < 1000000; i++) {
				ThreadInfo info = treading.getThreadInfo(17633, Integer.MAX_VALUE);
				StackTraceElement[] stackTrace = info.getStackTrace();
				String st = stackTrace[0].toString();
				if (!set.contains(st)) {
					set.add(st);
					System.out.println();
					for (StackTraceElement stackTraceElement : stackTrace) {
						System.out.println("\tat " + stackTraceElement);
					}
				}
			}
}
{code}

> Incorrect handling of end-of stream leading to infinite loop in streaming session
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-10206
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10206
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: cassndra 2.1.8 12 nodes cluster in two DC, with enabled ssl encryption.
>            Reporter: Alexey Burylov
>             Fix For: 2.1.x
>
>         Attachments: cassandra-2.1-10206.txt, threaddump-1440601628223.tdump, threaddump-1440602926719.tdump,
threaddump-1440672821181.tdump, threaddump-1440686737505.tdump
>
>
> StreamMessage.deserialize (StreamMessage.java:63) function return null when receive end
of stream for ReadableByteChannel.read () function. IncomingMessageHandler.run (ConnectionHandler.java:257)
interpreting null - as unsupported message and ignore it. It resulting infinite loop when
IncomingMessageHandler try to read message from closed steram, and deserialize always return
null - to indicate close stream.
> It not happen on normal socket, because when normal socket closed it throw IOException.
But happen from time to time when iner-node ssl encryption enabled.
> I'm attach two tread dump from one node with two infinity lopped thread with id 614 and
583



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message