From user-return-14474-archive-asf-public=cust-asf.ponee.io@storm.apache.org Sat Jun 22 10:01:15 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3FA2C180627 for ; Sat, 22 Jun 2019 12:01:15 +0200 (CEST) Received: (qmail 33821 invoked by uid 500); 22 Jun 2019 10:01:09 -0000 Mailing-List: contact user-help@storm.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@storm.apache.org Delivered-To: mailing list user@storm.apache.org Received: (qmail 33811 invoked by uid 99); 22 Jun 2019 10:01:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 22 Jun 2019 10:01:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8A46DC0143 for ; Sat, 22 Jun 2019 10:01:08 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.551 X-Spam-Level: ** X-Spam-Status: No, score=2.551 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, KAM_LOTSOFHASH=0.25, KAM_NUMSUBJECT=0.5, MANY_SPAN_IN_TEXT=0.001, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id D7gXo5pCZlCv for ; Sat, 22 Jun 2019 10:01:05 +0000 (UTC) Received: from mail-oi1-f195.google.com (mail-oi1-f195.google.com [209.85.167.195]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id AABB45FE50 for ; Sat, 22 Jun 2019 10:01:04 +0000 (UTC) Received: by mail-oi1-f195.google.com with SMTP id w196so6372766oie.7 for ; Sat, 22 Jun 2019 03:01:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=RrybvPcs0loqakmVY1qjmYiLmuNlcroXLoxvojMm1Tw=; b=a0Ebzbagsb6VBw+53POhlyMm1DzkSYFhwcX4IkaTjV0b0qk2UVnGdH8kEE+zigSTt3 RL8dhZxAl5Fa6GewfYK+odDVOKGHP3gRLx4ZBgeIPb8MDTqS9dVvk2RTf+RUfpxGEN1N r1J2aN5cBuAlk4zSAYDgz9jEc4RQ7cxJ6YQDIYl64JrIDRF5AfRY4lh2voRjXTywBJSt XOtVQyd4b8ND1l7G56pRa5ELvHdx6yd/OZGEZWsHo+et32QhHy2CPCeXcvZpdQddJ+jZ Wi5dVidvugPKkD0AgZgguShq7hg5BhWFHlKE4vN6Ha2ZcyVUcKwBrlGBYYyVcMi4wcdq fKOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=RrybvPcs0loqakmVY1qjmYiLmuNlcroXLoxvojMm1Tw=; b=g9wT+dkc+ji8oi2+ZryGVXtKf7PaeJxQcBikjKEdSv3mYXoR3qr+LChvedIIyk4jQF S6Paf3zX1OfHzmPIu6pgLRCQ3SAPnCcgxw66R0iiuWoa11enYG/zFl6D+oyNAJKQAXez 2At7LUg6zneRSUFYs2as+mpPlU+rDcWiQISK9gz/muHQI2+VhCTBOQr/YGGUHMO2zALC NQmVPBBC5BfMC1YG5zc5lCc4FwXpdJIg9Fd57t9sHpt4poi0poYa6Sgfn24S0JZW+Wfa OXdVXzMb8e1+3vS2N/7m6+57EsUSxteVBMaBtJ7wiMhSx2m/WBZGPXXOEWBTeryuQHM8 cr6g== X-Gm-Message-State: APjAAAVXf757suqkF+NG/dL8xj5WYcbbAAuVzljFk3VuaRSKPtcOWqzM /rm1+s7wBUjk+ZxWdgf0nI3Ayd8Lb29HxAusDp5RWEaZ X-Google-Smtp-Source: APXvYqzUlgyOW0UM6XO1uDeUq/4GJh9EZEaJcAcVY1cJuj+N2S86PfuDwCme2Pm/nEPXk125FmgtK8wW5QBCC2uinMM= X-Received: by 2002:aca:fd4e:: with SMTP id b75mr5420790oii.129.1561197663863; Sat, 22 Jun 2019 03:01:03 -0700 (PDT) MIME-Version: 1.0 References: <92eeb7cb93caea5556979b62ac4d5b2e9edc4728.camel@tomtom.com> <2bdbe1c2a784b9b8d1fb1fc5f8afdb500d75e2a5.camel@tomtom.com> In-Reply-To: <2bdbe1c2a784b9b8d1fb1fc5f8afdb500d75e2a5.camel@tomtom.com> From: =?UTF-8?Q?Stig_Rohde_D=C3=B8ssing?= Date: Sat, 22 Jun 2019 12:00:52 +0200 Message-ID: Subject: Re: Kryo errors after upgrading to Storm 2.0 To: user@storm.apache.org Content-Type: multipart/alternative; boundary="0000000000002dc8a9058be6a81c" --0000000000002dc8a9058be6a81c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable For KryoValuesDeserializer, I see instantiation in https://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d= 1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormClientPipelineF= actory.java#L40 https://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d= 1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormServerPipelineF= actory.java#L40 https://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d= 1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormClientHandler.j= ava#L35 https://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d= 1a /storm-client/src/jvm/org/apache/storm/serialization/KryoTupleDeserializer.= java#L30 The client and server pipeline factories are creating the deserializer as part of creating a Netty channel, which as I understand it means only one thread will be interacting with the deserializer. The client handler is only being used in the client pipeline factories, also as part of making a channel. We could probably get rid of this instance and just share the one used in the rest of the channel pipeline. The tuple deserializer is only used when instantiating values that go in ThreadLocals. For the serializer, I see instantiation in https://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d= 1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormServerPipelineF= actory.java#L43 https://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d= 1a/storm-client/src/jvm/org/apache/storm/serialization/KryoTupleSerializer.= java https://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d= 1a/storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java#L79 The pipeline factory should be okay for the same reason as above. The tuple serializer is used in a ThreadLocal, and also by ExecutorTransfer. The serializer used by Server is only used in a synchronized block, synchronizing on the serializer. The uses look mostly safe to me. I don't think we want to use ThreadLocals in the pipeline factories, it shouldn't be necessary. My main worry is the serializer in ExecutorTransfer. I think this can get used both by the main executor thread, and by any user threads interacting with the OutputCollector. For example, if you ack tuples in a different thread from the main thread in a bolt, you will likely be using this serializer from multiple threads. We could probably do the following: * Replace the Server serializer synchronization with a ThreadLocal. * Share the existing pipeline deserializer with the StormClientHandler instead of creating a new one * Make the ExecutorTransfer serializer ThreadLocal What do you think? Den tor. 20. jun. 2019 kl. 23.11 skrev Michal Koziorowski < Michal.Koziorowski@tomtom.com>: > https://githu= b.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/messag= ing/DeserializingConnectionCallback.java#L39 > > Looks quite good for me. There should be one Kryo instance per thread. > > > But I'm less sure about other serialization / deserialization parts: > > https://github.com/apache/storm/search?p=3D1&q=3DKryoValuesDeserializer&u= nscoped_q=3DKryoValuesDeserializer > > https://github.com/apache/storm/search?q=3DKryoV= aluesSerializer&unscoped_q=3DKryoValuesSerializer > > > What do you think about creating wrapper around Kryo*Serializers and Kryo= *Deserializers with ThreadLocal to get rid of possible future kryo threadin= g issues? > > > Best regards, > > Michal > > > -------- Wiadomo=C5=9B=C4=87 oryginalna ---------- > *Od*: Stig Rohde D=C3=B8ssing > > > *Odpowied=C5=BA do*: user@storm.apache.org > *Do*: user@storm.apache.org > *Temat*: Re: Kryo errors after upgrading to Storm 2.0 > *Data*: Thu, 20 Jun 2019 21:17:23 +0200 > > I don't think we've changed Kryo stuff much since 1.0.0. Last version > upgrade was https://github.com/apache/storm/pull/1255. > > Maybe it could be a threading issue in the Netty messaging implementation > in > https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apac= he/storm/messaging/netty. > I know a couple of issues of that sort were fixed in 2.0.0. 2.0.0 also > upgraded Netty, so maybe there's an issue in that code somewhere. > > Den tor. 20. jun. 2019 kl. 19.10 skrev Michal Koziorowski < > Michal.Koziorowski@tomtom.com>: > > Hi, > > After upgrading to Storm 2.0 and running in production couple of days I > saw following error: > > server errors in handling the request > com.esotericsoftware.kryo.KryoException: Unable to find class: j, > Serialization trace: value > (org.apache.storm.metric.api.IMetricsConsumer$DataPoint) at > com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClass= Resolver.java:156) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClas= sResolver.java:133) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:= ?] > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.ja= va:161) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.ja= va:39) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:1= 25) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerialize= r.java:551) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:134) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:40) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:134) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:40) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) > ~[kryo-3.0.3.jar:?] at > org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(Kry= oValuesDeserializer.java:31) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupl= eDeserializer.java:45) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.DeserializingConnectionCallback.recv(Deseriali= zingConnectionCallback.java:66) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.Server.enqueue(Server.java:140) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.Server.received(Server.java:269) > ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Storm= ServerHandler.channelRead(StormServerHandler.java:51) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fir= eChannelRead(AbstractChannelHandlerContext.java:340) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireCh= annelRead(ByteToMessageDecoder.java:323) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channe= lRead(ByteToMessageDecoder.java:297) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fir= eChannelRead(AbstractChannelHandlerContext.java:340) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContex= t.channelRead(DefaultChannelPipeline.java:1434) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChanne= lRead(DefaultChannelPipeline.java:965) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByt= eUnsafe.read(AbstractNioByteChannel.java:163) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= ey(NioEventLoop.java:644) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= eysOptimized(NioEventLoop.java:579) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= eys(NioEventLoop.java:496) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop= .java:458) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor= $5.run(SingleThreadEventExecutor.java:897) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] Caused by: > java.lang.ClassNotFoundException: j, at > java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_212] > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_212] a= t > sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > ~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ~[?:1.8.0_212] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_212] = at > java.lang.Class.forName(Class.java:348) ~[?:1.8.0_212] at > com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClass= Resolver.java:154) > ~[kryo-3.0.3.jar:?] ... 40 more > > and also messages similar to one below, but with different class IDs > > server errors in handling the request > com.esotericsoftware.kryo.KryoException: Encountered unregistered class I= D: > 108 Serialization trace: value > (org.apache.storm.metric.api.IMetricsConsumer$DataPoint) at > com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClas= sResolver.java:137) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670) ~[kryo-3.0.3.jar:= ?] > at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kry= o-3.0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.ja= va:161) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.ja= va:39) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar= :?] > at > com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:1= 25) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerialize= r.java:551) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3= .0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:134) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:40) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3= .0.3.jar:?] > at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:134) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collectio= nSerializer.java:40) > ~[kryo-3.0.3.jar:?] at > com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar= :?] > at > org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(Kry= oValuesDeserializer.java:31) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupl= eDeserializer.java:45) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.DeserializingConnectionCallback.recv(Deseriali= zingConnectionCallback.java:66) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.Server.enqueue(Server.java:140) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.Server.received(Server.java:269) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServ= erHandler.java:51) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fir= eChannelRead(AbstractChannelHandlerContext.java:340) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireCh= annelRead(ByteToMessageDecoder.java:323) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireCh= annelRead(ByteToMessageDecoder.java:310) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.callDe= code(ByteToMessageDecoder.java:426) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channe= lRead(ByteToMessageDecoder.java:278) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fir= eChannelRead(AbstractChannelHandlerContext.java:340) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContex= t.channelRead(DefaultChannelPipeline.java:1434) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChanne= lRead(DefaultChannelPipeline.java:965) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByt= eUnsafe.read(AbstractNioByteChannel.java:163) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= ey(NioEventLoop.java:644) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= eysOptimized(NioEventLoop.java:579) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= eys(NioEventLoop.java:496) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop= .java:458) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor= $5.run(SingleThreadEventExecutor.java:897) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] > > > And also: > server errors in handling the request > com.esotericsoftware.kryo.KryoException: Buffer underflow. at > com.esotericsoftware.kryo.io.Input.require(Input.java:182) ~[kryo-3.0.3.j= ar:?] > at com.esotericsoftware.kryo.io.Input.readLong(Input.java:686) ~[kryo-3.0= .3.jar:?] > at org.apache.storm.tuple.MessageId.deserialize(MessageId.java:56) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupl= eDeserializer.java:44) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.DeserializingConnectionCallback.recv(Deseriali= zingConnectionCallback.java:66) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.Server.enqueue(Server.java:140) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.Server.received(Server.java:269) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServ= erHandler.java:51) > ~[storm-client-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fir= eChannelRead(AbstractChannelHandlerContext.java:340) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.fireCh= annelRead(ByteToMessageDecoder.java:323) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.handler.codec.ByteToMessageDecoder.channe= lRead(ByteToMessageDecoder.java:297) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fir= eChannelRead(AbstractChannelHandlerContext.java:340) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContex= t.channelRead(DefaultChannelPipeline.java:1434) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:362) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.inv= okeChannelRead(AbstractChannelHandlerContext.java:348) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChanne= lRead(DefaultChannelPipeline.java:965) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.AbstractNioByteChannel$NioByt= eUnsafe.read(AbstractNioByteChannel.java:163) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= ey(NioEventLoop.java:644) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= eysOptimized(NioEventLoop.java:579) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedK= eys(NioEventLoop.java:496) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop= .java:458) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor= $5.run(SingleThreadEventExecutor.java:897) > [storm-shaded-deps-2.0.0.jar:2.0.0] at > java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] > > After which storm process is dying. > > I'm not 100% that this is related to upgrade to 2.0 version of storm or i= f > anything related to kryo was touched, but I haven't seen anything similar > with previous version of storm (1.1.x) > > Have you seen anything similar? > > Best regards, > Michal > > --0000000000002dc8a9058be6a81c Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
For KryoValuesDeserializer, I see instantiation in

https://github.com/apache/storm/= blob/dfae3c7e5a4128f821c9c8594156caf17f375d1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormClientPipelineF= actory.java#L40
https://github.com/apache/storm/bl= ob/dfae3c7e5a4128f821c9c8594156caf17f375d1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormServerPipelineF= actory.java#L40
https://github.com/apache/storm/bl= ob/dfae3c7e5a4128f821c9c8594156caf17f375d1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormClientHandler.j= ava#L35
https://github.com/apache/storm/blob/dfae3= c7e5a4128f821c9c8594156caf17f375d1a /storm-client/src/jvm/org/apache/storm/serialization/KryoTupleDeserializer.= java#L30

The client and server pipeline factories = are creating the deserializer as part of creating a Netty channel, which as= I understand it means only one thread will be interacting with the deseria= lizer.
The client handler is only being used in the client pipeli= ne factories, also as part of making a channel. We could probably get rid o= f this instance and just share the one used in the rest of the channel pipe= line.
The tuple deserializer is only used when instantiating valu= es that go in ThreadLocals.

For the serializer, I = see instantiation in

https://github= .com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d1a /storm-client/src/jvm/org/apache/storm/messaging/netty/StormServerPipelineF= actory.java#L43
htt= ps://github.com/apache/storm/blob/dfae3c7e5a4128f821c9c8594156caf17f375d1a/= storm-client/src/jvm/org/apache/storm/messaging/netty/Server.java#L79

The pipeline factory should be okay for the same re= ason as above.
The tuple serializer is used in a ThreadLocal,= and also by ExecutorTransfer.
The serializer used by Server= is only used in a synchronized block, synchronizing on the serializer.

The uses look mostly safe to me.=20 I don't think we want to use ThreadLocals in the pipeline factories, it= shouldn't be necessary.
My main worry is the serializer= in ExecutorTransfer. I think this can get used both by the main executor t= hread, and by any user threads interacting with the OutputCollector. For ex= ample, if you ack tuples in a different thread from the main thread in a bo= lt, you will likely be using this serializer from multiple threads.

We could probably do the following:
* Repla= ce the Server serializer synchronization with a ThreadLocal.
* Sh= are the existing pipeline deserializer with the StormClientHandler instead = of creating a new one
* Make the ExecutorTransfer serializer Thre= adLocal

What do you think?

=
Den tor. 2= 0. jun. 2019 kl. 23.11 skrev Michal Koziorowski <Michal.Koziorowski@tomtom.com>:
<= blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-l= eft:1px solid rgb(204,204,204);padding-left:1ex"> 
https://github.com/apache/storm/blob/mas=
ter/storm-client/src/jvm/org/apache/storm/messaging/DeserializingConnection=
Callback.java#L39
Looks quite good for me. There should be one Kryo instance per thread.=


But I'm less sure about other serialization / deserialization part=
s:
https://github.com/apache/storm/search?p=3D1&q=3DKryoValuesDeserialize=
r&unscoped_q=3DKryoValuesDeserializer
https://github.com/apache=
/storm/search?q=3DKryoValuesSerializer&unscoped_q=3DKryoValuesSerialize=
r

What do you think about creating wrapper around Kryo*Serializers and K=
ryo*Deserializers with ThreadLocal to get rid of possible future kryo threa=
ding issues?

Best regards,
Michal

-------- Wiadomo=C5=9B=C4=87 oryginalna ----------
Od: Stig Rohde D=C3=B8ssing <stigdoessing@gmail.com>
Odpowied=C5=BA do: user@storm.apache.org
Temat: Re: Kryo errors after upgrading to Storm 2.0
Data: Thu, 20 Jun 2019 21:17:23 +0200

I don't think we've changed Kryo stuff much since 1.0.0. Last = version upgrade was htt= ps://github.com/apache/storm/pull/1255.

Maybe it could be a threading issue in the Netty messaging implementat= ion in https://github.com/apache/storm/tree/master/storm-client/src/jvm/org/apache= /storm/messaging/netty. I know a couple of issues of that sort were fix= ed in 2.0.0. 2.0.0 also upgraded Netty, so maybe there's an issue in th= at code somewhere.

Den tor. 20. jun. 2019 kl. 19.10 skre= v Michal Koziorowski <Michal.Koziorowski@tomtom.com>:
Hi,

After upgrading to Storm 2.0 and running in production couple of days = I saw following error:

server errors in= handling the request com.esotericsoftware.kryo.KryoException: Unable to find class: j, Serialization trace: value (org.apache.storm.metr= ic.api.IMetricsConsumer$DataPoint) at com.esotericsoftware.kryo.util.Defaul= tClassResolver.readName(DefaultClassResolver.java:156) ~[kryo-3.0.3.jar:?] = at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultCla= ssResolver.java:133) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:= 670) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObje= ct(Kryo.java:781) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializ= ers.MapSerializer.read(MapSerializer.java:161) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.MapSerializer= .read(MapSerializer.java:39) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kr= yo.Kryo.readObject(Kryo.java:711) ~[kryo-3.0.3.jar:?] at com.esotericsoftwa= re.kryo.serializers.ObjectField.read(ObjectField.java:125) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.FieldSerializ= er.read(FieldSerializer.java:551) ~[kryo-3.0.3.jar:?] at com.esotericsoftwa= re.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?] at com.e= sotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerial= izer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSer= ializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esote= ricsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?]= at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collect= ionSerializer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSer= ializer.read(CollectionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esote= ricsoftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?] at org.= apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValue= sDeserializer.java:31) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupl= eDeserializer.deserialize(KryoTupleDeserializer.java:45) ~[storm-client-2.0= .0.jar:2.0.0] at org.apache.storm.messaging.DeserializingConnectionCallback= .recv(DeserializingConnectionCallback.java:66) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server= .enqueue(Server.java:140) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.sto= rm.messaging.netty.Server.received(Server.java:269) ~[storm-client-2.0.0.ja= r:2.0.0] at org.apache.storm.messaging.netty.StormServerHandler.channelRead(StormServerHandler.j= ava:51) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel= .AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerCont= ext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade= .io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractC= hannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerCo= ntext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.sha= de.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessag= eDecoder.java:323) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.han= dler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) = [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.chan= nel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerC= ontext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.s= hade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstrac= tChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.= java:1434) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io= .netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChan= nelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.s= hade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannel= Pipeline.java:965) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.j= ava:163) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.n= etty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [= storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.chann= el.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.0.0.j= ar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEv= entExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:74= 8) [?:1.8.0_212] Caused by: java.lang.ClassNotFoundException: j, at java.ne= t.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_212] at java.= lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_212] at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java= :349) ~[?:1.8.0_212] at java.lang.ClassLoader.loadClass(ClassLoader.java:35= 7) ~[?:1.8.0_212] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_212]= at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_212] at com.esotericsoftware.kryo.util.DefaultClassResolver.read= Name(DefaultClassResolver.java:154) ~[kryo-3.0.3.jar:?] ... 40 more<= /span>

and also messages similar to one below, but with different class IDs

server errors in handling the request com.esotericsoftware.kryo.KryoException: En= countered unregistered class ID: 108 Serialization trace: value (org.apache= .storm.metric.api.IMetricsConsumer$DataPoint) at com.esotericsoftware.kryo.= util.DefaultClassResolver.readClass(DefaultClassResolver.java:137) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.read= Class(Kryo.java:670) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.MapS= erializer.read(MapSerializer.java:161) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serialize= rs.MapSerializer.read(MapSerializer.java:39) ~[kryo-3.0.3.j= ar:?] at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) ~[kryo<= span style=3D"font-size:12px;font-variant-ligatures:normal;letter-spacing:-= 0.08px">-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.ObjectField.= read(ObjectField.java:125) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSeriali= zer.java:551) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.k= ryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serialize= rs.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collect= ionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esoterics= oftware.kryo.Kryo.readClassAndObject(Kryo.java:793) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serialize= rs.CollectionSerializer.read(CollectionSerializer.java:134) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collect= ionSerializer.java:40) ~[kryo-3.0.3.jar:?] at com.esoterics= oftware.kryo.Kryo.readObject(Kryo.java:689) ~[kryo-3.0.3.jar:?] at org.apache.storm.serialization.Kryo= ValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:31) ~[storm-= client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupleDeserial= izer.deserialize(KryoTupleDeserializer.java:45) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.Deserializin= gConnectionCallback.recv(DeserializingConnectionCallback.java:66) ~[storm-c= lient-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(S= erver.java:140) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.received(Server.java:269) ~[sto= rm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.StormServerH= andler.channelRead(StormServerHandler.java:51) ~[storm-client-2.0.0.jar:2.0= .0] at org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContex= t.invokeChannelRead(AbstractChannelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.s= hade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstrac= tChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.han= dler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:3= 23) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.= handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.jav= a:310) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.han= dler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:426) [= storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.handl= er.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.s= hade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstr= actChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerCo= ntext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.sha= de.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultC= hannelPipeline.java:1434) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.s= hade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Abstr= actChannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965= ) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.ch= annel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.= java:163) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shad= ed-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.Nio= EventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-sha= ded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel.nio.Ni= oEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.uti= l.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java= :897) [storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.ja= va:748) [?:1.8.0_212]


And also:
server errors in handling the request com.esotericsoftware.kryo.KryoException: Bu= ffer underflow. at com.esotericsoftware.kryo.io.Input.require(Input.java:18= 2) ~[kryo-3.0.3.jar:?] at com.esotericsoftware.kryo.io.Input.readLong(Input.java:686) ~[kr= yo-3.0.3.jar:?] at org.apache.storm.tuple.MessageId.deserialize(Me= ssageId.java:56) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTu= pleDeserializer.java:44) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.stor= m.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCal= lback.java:66) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.enqueue(Server.java:140) ~[stor= m-client-2.0.0.jar:2.0.0] at org.apache.storm.messaging.netty.Server.receiv= ed(Server.java:269) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.mes= saging.netty.StormServerHandler.channelRead(StormServerHandler.java:51) ~[storm-client-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.channel= .AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerCont= ext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade= .io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractC= hannelHandlerContext.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerCo= ntext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.sha= de.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessag= eDecoder.java:323) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.han= dler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:297) = [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.chan= nel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerC= ontext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.s= hade.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(Abstrac= tChannelHandlerContext.java:340) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.= java:1434) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io= .netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChan= nelHandlerContext.java:362) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandler= Context.java:348) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.s= hade.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannel= Pipeline.java:965) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.j= ava:163) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.n= etty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [= storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.chann= el.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [storm-shaded-deps-2.0.0.jar:2.0.0] at org.apache.storm.shade.io.netty.cha= nnel.nio.NioEventLoop.run(NioEventLoop.java:458) [storm-shaded-deps-2.0.0.j= ar:2.0.0] at org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEv= entExecutor$5.run(SingleThreadEventExecutor.java:897) [storm-shaded-deps-2.0.0.jar:2.0.0] at java.lang.Thread.run(Thread.java:74= 8) [?:1.8.0_212]

After which storm process is dying.

I'm not 100% that this is related to upgrade to 2.0 version of sto= rm or if anything related to kryo was touched, but I haven't seen anyth= ing similar with previous version of storm (1.1.x)

Have you seen anything similar?

Best regards,
Michal

--0000000000002dc8a9058be6a81c--