storm-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michal Koziorowski <Michal.Koziorow...@tomtom.com>
Subject Re: Kryo errors after upgrading to Storm 2.0
Date Thu, 20 Jun 2019 21:10:34 GMT
<https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/messaging/DeserializingConnectionCallback.java#L39>https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/messaging/DeserializingConnectionCallback.java#L39

Looks quite good for me. There should be one Kryo instance per thread.


But I'm less sure about other serialization / deserialization parts:

https://github.com/apache/storm/search?p=1&q=KryoValuesDeserializer&unscoped_q=KryoValuesDeserializer

<https://github.com/apache/storm/search?q=KryoValuesSerializer&unscoped_q=KryoValuesSerializer>https://github.com/apache/storm/search?q=KryoValuesSerializer&unscoped_q=KryoValuesSerializer


What do you think about creating wrapper around Kryo*Serializers and Kryo*Deserializers with
ThreadLocal to get rid of possible future kryo threading issues?


Best regards,

Michal

-------- Wiadomość oryginalna ----------
Od: Stig Rohde Døssing <stigdoessing@gmail.com<mailto:Stig%20Rohde%20%3d%3fISO-8859-1%3fQ%3fD%3dF8ssing%3f%3d%20%3cstigdoessing@gmail.com%3e>>
Odpowiedź do: user@storm.apache.org<mailto:user@storm.apache.org>
Do: user@storm.apache.org<mailto:user@storm.apache.org>
Temat: Re: Kryo errors after upgrading to Storm 2.0
Data: Thu, 20 Jun 2019 21:17:23 +0200

I don't think we've changed Kryo stuff much since 1.0.0. Last version upgrade was https://github.com/apache/storm/pull/1255.

Maybe it could be a threading issue in the Netty messaging implementation in https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apache/storm/messaging/netty.
I know a couple of issues of that sort were fixed in 2.0.0. 2.0.0 also upgraded Netty, so
maybe there's an issue in that code somewhere.

Den tor. 20. jun. 2019 kl. 19.10 skrev Michal Koziorowski <Michal.Koziorowski@tomtom.com<mailto:Michal.Koziorowski@tomtom.com>>:
Hi,

After upgrading to Storm 2.0 and running in production couple of days I saw following error:

server errors in handling the request com.esotericsoftware.kryo.KryoException: Unable to find
class: j, Serialization trace: value (org.apache.storm.metric.api.IMetricsConsumer$DataPoint)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?] at
com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?]
at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:45)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:66)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(Server.java:140)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.received(Server.java:269)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
[storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
Caused by: java.lang.ClassNotFoundException: j, at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_212] at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) ~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
~[?:1.8.0_212] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_212] at java.lang.Class.forName(Class.java:348)
~[?:1.8.0_212] at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:154)
~[kryo-3.0.3.jar:?] ... 40 more

and also messages similar to one below, but with different class IDs

server errors in handling the request com.esotericsoftware.kryo.KryoException: Encountered
unregistered class ID: 108 Serialization trace: value (org.apache.storm.metric.api.IMetricsConsumer$DataPoint)
at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:137)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?] at
com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:161) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]
at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?]
at org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:45)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:66)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(Server.java:140)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.received(Server.java:269)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
[storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]


And also:
server errors in handling the request com.esotericsoftware.kryo.KryoException: Buffer underflow.
at com.esotericsoftware.kryo.io.Input.require(Input.java:182) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.io.Input.readLong(Input.java:686)
~[kryo-3.0.3.jar:?] at org.apache.storm.tuple.MessageId.deserialize(MessageId.java:56) ~[storm-client-2.0.0.jar:2.0.0]
at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:44)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:66)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(Server.java:140)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.received(Server.java:269)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.java:51)
~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:323)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
[storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
[storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

After which storm process is dying.

I'm not 100% that this is related to upgrade to 2.0 version of storm or if anything related
to kryo was touched, but I haven't seen anything similar with previous version of storm (1.1.x)

Have you seen anything similar?

Best regards,
Michal

Mime
View raw message