tomcat-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mitch Claborn <mitch...@claborn.net>
Subject Re: Error receiving all sessions in cluster
Date Tue, 18 Sep 2018 13:51:35 GMT
I continue to have problems of this nature, although not with the exact 
symptom shown below. I've been able to identify some possible problems 
in my code that might have been causing some objects in the session to 
deserialize improperly.  I've fixed those and have not seen those 
specific errors crop up again. I have another error that I'm working on 
now: some of my own objects serialize to/from JSON and the JSON string 
in the serialized session object has obviously been corrupted.  I can't 
see how that could have happened, but I'm still working on it.  So.... 2 
big questions/ideas:

1) Any time the DeltaManager runs into an error deserializing the "all 
sessions" message, all sessions from the one with the error and later 
are lost. I realize it must currently be this way because once there is 
an error in the serialized stream there is no realistic way to find 
where the next session object begins in the stream. I'd like to be able 
to configure the Delta Manager to send the "all sessions" message in 
chunks of N sessions per message, where N can be configured in the XML. 
Doing this would limit the impact of a problematic session to only those 
in the same chunk. The final chunk would need some kind of marker so 
that the receiver knows that all sessions have been received. Any 
thoughts on this? I don't think it is possible in the current code. How 
hard would it be to make this change.  If I were to code it, would it be 
considered for acceptance into the base code?

2) Is it possible that a session object could be accessed and used while 
it is being deserialized in another thread for the "all sessions" 
message to another host? This might explain the corruption I'm seeing.

PS - here's how I'm discovering the session objects that are in error. I 
have a channel interceptor that watches for the receipt of the "all 
sessions" message and writes it to disk then stops watching for that 
message. I copy that disk file to my development machine where I run a 
quick-and-dirty program that I wrote that emulates the processing of 
Delta Manager with some debugging in place so that I can see exactly 
which session had the problem and which attribute within that session. 
I'd also like to propose some updates to Delta Manager that would 
capture that information and put it into the error logging.


Mitch

On 09/07/2018 04:44 PM, Mitch Claborn wrote:
> I'm using a cluster with the DeltaManager between two servers on Tomcat 
> 9.0.11.  When I restart my "A" server, I sometimes see the error below. 
> Peaking at the code in DeltaManager it appears that there is a problem 
> in the serialized data stream when receiving all of the session data 
> from the "B" server. Any ideas what could be causing this? It seems like 
> that if there was something in a Session object that was not 
> Serializable it would not even make it over the wire. Perhaps if 
> something in a session did not serialize properly?
> 
> (From the code, it looks like all the sessions in the message after the 
> error are lost, so this is kind of important.)
> 
> 07-Sep-2018 16:09:53.220 SEVERE 
> [Tribes-Task-Receiver[Catalina-Channel]-6] 
> org.apache.catalina.ha.session.DeltaManager.deserializeSessions 
> IOException while loading persisted sessions: 
> [java.io.StreamCorruptedException: invalid type code: 00]
>   java.io.StreamCorruptedException: invalid type code: 00
>      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1381)
>      at 
> java.io.ObjectInputStream.skipCustomData(ObjectInputStream.java:1984)
>      at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1946)
>      at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
>      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
>      at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
>      at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
>      at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
>      at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
>      at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
>      at 
> org.apache.catalina.ha.session.DeltaSession.doReadObject(DeltaSession.java:748) 
> 
>      at 
> org.apache.catalina.ha.session.DeltaSession.readObjectData(DeltaSession.java:569) 
> 
>      at 
> org.apache.catalina.ha.session.DeltaManager.deserializeSessions(DeltaManager.java:613)

> 
>      at 
> org.apache.catalina.ha.session.DeltaManager.handleALL_SESSION_DATA(DeltaManager.java:1323)

> 
>      at 
> org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1162) 
> 
>      at 
> org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:917)

> 
>      at 
> org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:77)

> 
>      at 
> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:788)

> 
>      at 
> org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:771)

> 
>      at 
> org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:324)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)

> 
>      at 
> org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor.messageReceived(TcpPingInterceptor.java:175)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)

> 
>      at 
> org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:116)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)

> 
>      at 
> org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor.messageReceived(StaticMembershipInterceptor.java:96)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)

> 
>      at 
> org.apache.catalina.tribes.group.interceptors.ThroughputInterceptor.messageReceived(ThroughputInterceptor.java:86)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:91)

> 
>      at 
> org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:274)

> 
>      at 
> org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:261)

> 
>      at 
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:213)

> 
>      at 
> org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:101)

> 
>      at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
> 
>      at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
> 
>      at java.lang.Thread.run(Thread.java:745)
> 
> 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org


Mime
View raw message