Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id D24652009F9 for ; Mon, 23 May 2016 09:32:33 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id D10F2160A0F; Mon, 23 May 2016 07:32:33 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id D853D1609A8 for ; Mon, 23 May 2016 09:32:31 +0200 (CEST) Received: (qmail 70484 invoked by uid 500); 23 May 2016 07:32:30 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 70474 invoked by uid 99); 23 May 2016 07:32:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 23 May 2016 07:32:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 69697180572 for ; Mon, 23 May 2016 07:32:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.28 X-Spam-Level: * X-Spam-Status: No, score=1.28 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=okkam-it.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id foDYYfMykbWa for ; Mon, 23 May 2016 07:32:22 +0000 (UTC) Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id CBCC95F244 for ; Mon, 23 May 2016 07:32:21 +0000 (UTC) Received: by mail-lf0-f47.google.com with SMTP id e130so34390119lfe.3 for ; Mon, 23 May 2016 00:32:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=okkam-it.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=NbbTRtttyCIDsFZU3Z0Jync4oxakEnjJkSy7CZAwApY=; b=TyN/+NvEcLzoajUs+6TvtaqaWuoXNUMcLmWlTJW2IyL/8gLLEIcJ3Bt/Q6Jz/Kesvw BUorWQ+X5HD4/IihygHv5RfdQRktP4u1TzlUnmk0gJphMqIimaP8Q9GfqVBHgVwCElg6 /EVo7UeY3Vs4n9Hi/4cvh4m5xV/YdwjX9lmpo6kQuIdfhqb4stMaaNE7Ij3MbzNU1cm9 MSEdjxu8R4JtgQSpY7qgWOXsuPxGtTlf4IhIsi9idUAB6kAlAo0K+TK2VvaqxN9yJX+D YGpyEoJ8qIFQeGn/HmSq3l+tHIJZpDzv2SUb+3aqnvIdbJe9FHCnLFbTR4vpdAUZwRKQ JliA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=NbbTRtttyCIDsFZU3Z0Jync4oxakEnjJkSy7CZAwApY=; b=al0K42w3nQ1SAU/Jd0GdN0drYq3YrulhCfO8xWwdKOexcQRaxD5E07puH+3y2KqpgP vGPTnpA/CQ1TlcFyN9pcgqZnVIj+JUyEKTrdPXjvpyYE8tHip6yYOOxFy6CaF2Zkbfvy 2rvsj8CSx7CmlH6mPyo6Oal+pJ6rDKs8+EfKv4dkM7Q2NndSp9DOvoLv/pDwFmh9WxgJ SrTNf1YiXxAWQ6xjRRqmpVOwaCjmNXAFfVpidXC2GRN4eEnJu5w0Efxm5QjIslb8rAV+ KWGZaIdDm4aRhwzjHARDRfNDC8hyqf3xXjLuejE0YM4Zc1+5cZUZZZhuo7S6rHcayXuF IYFw== X-Gm-Message-State: AOPr4FVlao6exVh3ZTJ+d1trS/jBeccV1GpHaR6lVvEktJK4hb9WJm5sXq1DKAYE6dyUPCzrb70TrVFMMALRfg== X-Received: by 10.25.149.66 with SMTP id x63mr4809090lfd.128.1463988734021; Mon, 23 May 2016 00:32:14 -0700 (PDT) MIME-Version: 1.0 Received: by 10.114.12.104 with HTTP; Mon, 23 May 2016 00:31:54 -0700 (PDT) X-Originating-IP: [213.203.177.29] In-Reply-To: References: From: Flavio Pompermaier Date: Mon, 23 May 2016 09:31:54 +0200 Message-ID: Subject: Re: Weird Kryo exception (Unable to find class: java.ttil.HashSet) To: user Content-Type: multipart/alternative; boundary=001a114034e47269ce05337d7073 archived-at: Mon, 23 May 2016 07:32:34 -0000 --001a114034e47269ce05337d7073 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I've slightly modified the program to shorten the length on the entire job and this time I had this Exception: 2016-05-23 09:26:51,438 ERROR org.apache.flink.runtime.io.disk.iomanager.IOManager - IO Thread 'IOManager writer thread #1' terminated due to an exception. Shutting down I/O Manager. java.lang.ClassCastException: java.nio.DirectByteBuffer$Deallocator cannot be cast to org.apache.flink.runtime.io.disk.iomanager.WriteRequest at org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(= IOManagerAsync.java:486) I don't know wheter this is related to the others or not.. On Sat, May 21, 2016 at 11:00 AM, Flavio Pompermaier wrote: > I think this bug comes from something in > SpillingAdaptiveSpanningRecordDeserializer..I've tried to find a common > point of failure in all those messages and I found that it contains also > this error message that I got once: > > private static final String BROKEN_SERIALIZATION_ERROR_MESSAGE =3D > "Serializer consumed more bytes than the record > had. " + > "This indicates broken serialization. If you are > using custom serialization types " + > "(Value or Writable), check their serialization > methods. If you are using a " + > "Kryo-serialized type, check the corresponding > Kryo serializer."; > > Any clue about how to find what is causing this? > > > > > On Sat, May 21, 2016 at 10:53 AM, Flavio Pompermaier > wrote: > >> I tried to move flink tmp dir from hdd disks to sdd ones (in order to >> exclude faulty disks) and I had another of those Exception: >> >> java.lang.IllegalArgumentException: The datetime zone id 'Europe/Romd' i= s >> not recognised >> at org.joda.time.DateTimeZone.forID(DateTimeZone.java:229) >> at >> de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.readTimeZo= ne(JodaDateTimeSerializer.java:94) >> at >> de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.read(JodaD= ateTimeSerializer.java:74) >> at >> de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.read(JodaD= ateTimeSerializer.java:59) >> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >> at >> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserial= ize(KryoSerializer.java:228) >> at >> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserial= ize(KryoSerializer.java:242) >> at >> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(P= ojoSerializer.java:501) >> at >> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(P= ojoSerializer.java:441) >> at >> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read= (NonReusingDeserializationDelegate.java:55) >> at >> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSp= anningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeseri= alizer.java:124) >> at >> org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getN= extRecord(AbstractRecordReader.java:65) >> at >> org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next(= MutableRecordReader.java:34) >> at >> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIterat= or.java:73) >> at >> org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.java:= 101) >> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480) >> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:34= 5) >> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) >> at java.lang.Thread.run(Thread.java:745) >> >> >> On Fri, May 20, 2016 at 8:34 PM, Flavio Pompermaier > > wrote: >> >>> Right now I'm using Flink 1.0.2...to which version should I downgrade? >>> The hardware seems to be ok..how could I detect a faulty hardware? >>> These errors appeared in every run of my job after I moved the temporar= y >>> directory from ssd to hdd and I extended my pipeline with a dataset tha= t >>> grows as the pipeline goes on,accumulating data from intermediate datas= ets.. >>> On 20 May 2016 18:31, "Fabian Hueske" wrote: >>> >>>> The problem seems to occur quite often. >>>> Did you update your Flink version recently? If so, could you try to >>>> downgrade and see if the problem disappears. >>>> >>>> Is it otherwise possible that it is cause by faulty hardware? >>>> >>>> 2016-05-20 18:05 GMT+02:00 Flavio Pompermaier : >>>> >>>>> This time (Europed instead of Europe): >>>>> >>>>> java.lang.Exception: The data preparation for task 'CHAIN GroupReduce= (GroupReduce at createResult(PassaggioWithComprAndVend.java:132)) -> Map (= Key Extractor)' , caused an error: Error obtaining the sorted input: Thread= 'SortMerger spilling thread' terminated due to an exception: The datetime = zone id 'Europd/Rome' is not recognised >>>>> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:4= 56) >>>>> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.jav= a:345) >>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> Caused by: java.lang.RuntimeException: Error obtaining the sorted inp= ut: Thread 'SortMerger spilling thread' terminated due to an exception: The= datetime zone id 'Europd/Rome' is not recognised >>>>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getI= terator(UnilateralSortMerger.java:619) >>>>> at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.j= ava:1079) >>>>> at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(Grou= pReduceDriver.java:94) >>>>> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:4= 50) >>>>> ... 3 more >>>>> Caused by: java.io.IOException: Thread 'SortMerger spilling thread' t= erminated due to an exception: The datetime zone id 'Europd/Rome' is not re= cognised >>>>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$Thre= adBase.run(UnilateralSortMerger.java:800) >>>>> Caused by: java.lang.IllegalArgumentException: The datetime zone id '= Europd/Rome' is not recognised >>>>> at org.joda.time.DateTimeZone.forID(DateTimeZone.java:229) >>>>> at de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.rea= dTimeZone(JodaDateTimeSerializer.java:94) >>>>> at de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.rea= d(JodaDateTimeSerializer.java:74) >>>>> at de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.rea= d(JodaDateTimeSerializer.java:59) >>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.d= eserialize(KryoSerializer.java:228) >>>>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.d= eserialize(KryoSerializer.java:242) >>>>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.c= opy(KryoSerializer.java:252) >>>>> at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(P= ojoSerializer.java:556) >>>>> at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.c= opy(TupleSerializerBase.java:75) >>>>> at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.write= ToOutput(NormalizedKeySorter.java:499) >>>>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$Spil= lingThread.go(UnilateralSortMerger.java:1344) >>>>> at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$Thre= adBase.run(UnilateralSortMerger.java:796) >>>>> >>>>> >>>>> >>>>> On Fri, May 20, 2016 at 4:25 PM, Flavio Pompermaier < >>>>> pompermaier@okkam.it> wrote: >>>>> >>>>>> This time another error (rerialization instead of serialization): >>>>>> >>>>>> com.esotericsoftware.kryo.KryoException: Unable to find class: it.ok= kam.flink.entitons.*rerialization*.pojo.EntitonQuadPojo >>>>>> at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(Def= aultClassResolver.java:138) >>>>>> at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(De= faultClassResolver.java:115) >>>>>> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641) >>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752) >>>>>> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(= CollectionSerializer.java:116) >>>>>> at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(= CollectionSerializer.java:22) >>>>>> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>>>> at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.= deserialize(KryoSerializer.java:228) >>>>>> at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deser= ialize(PojoSerializer.java:431) >>>>>> at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.dese= rialize(TupleSerializer.java:135) >>>>>> at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.dese= rialize(TupleSerializer.java:30) >>>>>> at org.apache.flink.runtime.plugable.NonReusingDeserializationDeleg= ate.read(NonReusingDeserializationDelegate.java:55) >>>>>> at org.apache.flink.runtime.io.network.api.serialization.SpillingAd= aptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningReco= rdDeserializer.java:124) >>>>>> at org.apache.flink.runtime.io.network.api.reader.AbstractRecordRea= der.getNextRecord(AbstractRecordReader.java:65) >>>>>> at org.apache.flink.runtime.io.network.api.reader.MutableRecordRead= er.next(MutableRecordReader.java:34) >>>>>> at org.apache.flink.runtime.operators.util.ReaderIterator.next(Read= erIterator.java:73) >>>>>> at org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriv= er.java:101) >>>>>> at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:= 480) >>>>>> at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.ja= va:345) >>>>>> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) >>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>> Caused by: java.lang.ClassNotFoundException: it.okkam.flink.entitons= .rerialization.pojo.EntitonQuadPojo >>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>> at java.lang.Class.forName0(Native Method) >>>>>> at java.lang.Class.forName(Class.java:348) >>>>>> at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(Def= aultClassResolver.java:136) >>>>>> ... 20 more >>>>>> >>>>>> >>>>>> >>>>>> On Fri, May 20, 2016 at 12:48 PM, Flavio Pompermaier < >>>>>> pompermaier@okkam.it> wrote: >>>>>> >>>>>>> Hi Ufuk, >>>>>>> my records could be quite large Pojos (I think some MB). >>>>>>> The only thing I do to configure Kryo is: >>>>>>> >>>>>>> env.registerTypeWithKryoSerializer(DateTime.class, >>>>>>> JodaDateTimeSerializer.class ); >>>>>>> >>>>>>> Best, >>>>>>> Flavio >>>>>>> >>>>>>> On Fri, May 20, 2016 at 12:38 PM, Ufuk Celebi >>>>>>> wrote: >>>>>>> >>>>>>>> @Stefano: the records are serialized anyway for batch jobs. The >>>>>>>> spilling deserializer is only relevant if single records are very >>>>>>>> large. How large are your records? In any case, I don't expect thi= s >>>>>>>> to >>>>>>>> be the problem. >>>>>>>> >>>>>>>> @Flavio: The class name "typo" errors (Vdhicle instead of Vehicle >>>>>>>> and >>>>>>>> ttil instead of util) look like some kind of data corruption and >>>>>>>> would >>>>>>>> need further investigation. The other failure you reported might b= e >>>>>>>> related to this. As a starting point, how do you configure the Kry= o >>>>>>>> serializer? >>>>>>>> >>>>>>>> On Fri, May 20, 2016 at 10:02 AM, Flavio Pompermaier >>>>>>>> wrote: >>>>>>>> > Today I've got this other strange error.. Obviously I don't have= a >>>>>>>> > VdhicleEvent class, but a VehicleEvent class :( >>>>>>>> > >>>>>>>> > java.lang.RuntimeException: Cannot instantiate class. >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deseria= lize(PojoSerializer.java:407) >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegat= e.read(NonReusingDeserializationDelegate.java:55) >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.runtime.io.network.api.serialization.SpillingAdap= tiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecord= Deserializer.java:124) >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.runtime.io.network.api.reader.AbstractRecordReade= r.getNextRecord(AbstractRecordReader.java:65) >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.runtime.io.network.api.reader.MutableRecordReader= .next(MutableRecordReader.java:34) >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.runtime.operators.util.ReaderIterator.next(Reader= Iterator.java:73) >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver= .java:101) >>>>>>>> > at >>>>>>>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:48= 0) >>>>>>>> > at >>>>>>>> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java= :345) >>>>>>>> > at >>>>>>>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) >>>>>>>> > at java.lang.Thread.run(Thread.java:745) >>>>>>>> > Caused by: java.lang.ClassNotFoundException: >>>>>>>> > it.okkam.flink.test.model.pojo.VdhicleEvent >>>>>>>> > at >>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>> > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>> > at java.lang.Class.forName0(Native Method) >>>>>>>> > at java.lang.Class.forName(Class.java:348) >>>>>>>> > at >>>>>>>> > >>>>>>>> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deseria= lize(PojoSerializer.java:405) >>>>>>>> > ... 10 more >>>>>>>> > >>>>>>>> > >>>>>>>> > Thanks in advance, >>>>>>>> > Flavio >>>>>>>> > >>>>>>>> > >>>>>>>> > On Mon, May 16, 2016 at 10:08 AM, Stefano Bortoli < >>>>>>>> s.bortoli@gmail.com> >>>>>>>> > wrote: >>>>>>>> >> >>>>>>>> >> Hi Flavio, Till, >>>>>>>> >> >>>>>>>> >> do you think this can be possibly related to the serialization >>>>>>>> problem >>>>>>>> >> caused by 'the management' of Kryo serializer buffer when >>>>>>>> spilling on disk? >>>>>>>> >> We are definitely going beyond what is managed in memory with >>>>>>>> this task. >>>>>>>> >> >>>>>>>> >> saluti, >>>>>>>> >> Stefano >>>>>>>> >> >>>>>>>> >> 2016-05-16 9:44 GMT+02:00 Flavio Pompermaier < >>>>>>>> pompermaier@okkam.it>: >>>>>>>> >>> >>>>>>>> >>> That exception showed just once, but the following happens >>>>>>>> randomly (if I >>>>>>>> >>> re-run the job after stopping and restartign the cluster it >>>>>>>> doesn't show up >>>>>>>> >>> usually): >>>>>>>> >>> >>>>>>>> >>> Caused by: java.io.IOException: Serializer consumed more bytes >>>>>>>> than the >>>>>>>> >>> record had. This indicates broken serialization. If you are >>>>>>>> using custom >>>>>>>> >>> serialization types (Value or Writable), check their >>>>>>>> serialization methods. >>>>>>>> >>> If you are using a Kryo-serialized type, check the >>>>>>>> corresponding Kryo >>>>>>>> >>> serializer. >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.io.network.api.serialization.SpillingAdap= tiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecord= Deserializer.java:142) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.io.network.api.reader.AbstractRecordReade= r.getNextRecord(AbstractRecordReader.java:65) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.io.network.api.reader.MutableRecordReader= .next(MutableRecordReader.java:34) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.operators.util.ReaderIterator.next(Reader= Iterator.java:73) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver= .java:101) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:48= 0) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java= :345) >>>>>>>> >>> at >>>>>>>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) >>>>>>>> >>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> >>> Caused by: java.lang.ArrayIndexOutOfBoundsException: -2 >>>>>>>> >>> at java.util.ArrayList.elementData(ArrayList.java:418) >>>>>>>> >>> at java.util.ArrayList.get(ArrayList.java:431) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(= MapReferenceResolver.java:42) >>>>>>>> >>> at >>>>>>>> com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:805) >>>>>>>> >>> at >>>>>>>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:759) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.de= serialize(KryoSerializer.java:228) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deseria= lize(PojoSerializer.java:431) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deseri= alize(TupleSerializer.java:135) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deseri= alize(TupleSerializer.java:30) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegat= e.read(NonReusingDeserializationDelegate.java:55) >>>>>>>> >>> at >>>>>>>> >>> >>>>>>>> org.apache.flink.runtime.io.network.api.serialization.SpillingAdap= tiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecord= Deserializer.java:124) >>>>>>>> >>> >>>>>>>> >>> >>>>>>>> >>> On Sun, May 15, 2016 at 10:58 AM, Flavio Pompermaier >>>>>>>> >>> wrote: >>>>>>>> >>>> >>>>>>>> >>>> Hi to all, >>>>>>>> >>>> in my last run of a job I received this weird Kryo Exception >>>>>>>> in one of >>>>>>>> >>>> the TaskManager...obviously this class in not mentioned >>>>>>>> anywhere, neither in >>>>>>>> >>>> my project nor in flink... >>>>>>>> >>>> Any help is appreciated! >>>>>>>> >>>> >>>>>>>> >>>> Best, >>>>>>>> >>>> Flavio >>>>>>>> >>>> >>>>>>>> >>>> INFO org.apache.flink.runtime.taskmanager.Task - CHAIN >>>>>>>> GroupReduce >>>>>>>> >>>> (GroupReduce at createResult(IndexMappingExecutor.java:42)) -= > >>>>>>>> Map (Map at >>>>>>>> >>>> main(Jsonizer.java:128)) (4/9) switched to FAILED with >>>>>>>> exception. >>>>>>>> >>>> java.lang.Exception: The data preparation for task 'CHAIN >>>>>>>> GroupReduce >>>>>>>> >>>> (GroupReduce at createResult(IndexMappingExecutor.java:42)) -= > >>>>>>>> Map (Map at >>>>>>>> >>>> main(Jsonizer.java:128))' , caused an error: Error obtaining >>>>>>>> the sorted >>>>>>>> >>>> input: Thread 'SortMerger spilling thread' terminated due to >>>>>>>> an exception: >>>>>>>> >>>> Unable to find class: java.ttil.HashSet >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:45= 6) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java= :345) >>>>>>>> >>>> at >>>>>>>> org.apache.flink.runtime.taskmanager.Task.run(Task.java:559) >>>>>>>> >>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> >>>> Caused by: java.lang.RuntimeException: Error obtaining the >>>>>>>> sorted input: >>>>>>>> >>>> Thread 'SortMerger spilling thread' terminated due to an >>>>>>>> exception: Unable >>>>>>>> >>>> to find class: java.ttil.HashSet >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIt= erator(UnilateralSortMerger.java:619) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.ja= va:1079) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.GroupReduceDriver.prepare(Group= ReduceDriver.java:94) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:45= 0) >>>>>>>> >>>> ... 3 more >>>>>>>> >>>> Caused by: java.io.IOException: Thread 'SortMerger spilling >>>>>>>> thread' >>>>>>>> >>>> terminated due to an exception: Unable to find class: >>>>>>>> java.ttil.HashSet >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$Threa= dBase.run(UnilateralSortMerger.java:800) >>>>>>>> >>>> Caused by: com.esotericsoftware.kryo.KryoException: Unable to >>>>>>>> find >>>>>>>> >>>> class: java.ttil.HashSet >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(Defau= ltClassResolver.java:138) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(Defa= ultClassResolver.java:115) >>>>>>>> >>>> at >>>>>>>> com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerial= izer.java:143) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerial= izer.java:21) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.de= serialize(KryoSerializer.java:228) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.de= serialize(KryoSerializer.java:242) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.co= py(KryoSerializer.java:252) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(Po= joSerializer.java:556) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.co= py(TupleSerializerBase.java:75) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeT= oOutput(NormalizedKeySorter.java:499) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$Spill= ingThread.go(UnilateralSortMerger.java:1344) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerger$Threa= dBase.run(UnilateralSortMerger.java:796) >>>>>>>> >>>> Caused by: java.lang.ClassNotFoundException: java.ttil.HashSe= t >>>>>>>> >>>> at >>>>>>>> java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>>>>>> >>>> at >>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>>>>>> >>>> at >>>>>>>> java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>>>>>> >>>> at java.lang.Class.forName0(Native Method) >>>>>>>> >>>> at java.lang.Class.forName(Class.java:348) >>>>>>>> >>>> at >>>>>>>> >>>> >>>>>>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(Defau= ltClassResolver.java:136) >>>>>>>> >>>> ... 14 more >>>>>>>> >>>> >>>>>>>> >>> >>>>>>>> >> >>>>>>>> > >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> >> >> > --001a114034e47269ce05337d7073 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
I've slightly modified the program to shorten the= length on the entire job and this time I had this Exception:

2016-0= 5-23 09:26:51,438 ERROR org.apache.flink.runtime.io.disk.iomanager.IOManage= r=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - IO Thread 'IO= Manager writer thread #1' terminated due to an exception. Shutting down= I/O Manager.
java.lang.ClassCastException: java.nio.DirectByteBuffer$De= allocator cannot be cast to org.apache.flink.runtime.io.disk.iomanager.Writ= eRequest
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 at org.apache.flink.= runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.ja= va:486)


I don't know wheter this is related to the oth= ers or not..

O= n Sat, May 21, 2016 at 11:00 AM, Flavio Pompermaier <<= a href=3D"mailto:pompermaier@okkam.it" target=3D"_blank">pompermaier@okkam.= it> wrote:
I think this bug comes from something in SpillingAdaptiveSpanningRecordDes= erializer..I've tried to find a common point of failure in all those me= ssages and I found that it=C2=A0contains also this error message that I got= once:

private static final String BROKEN_SERIALIZATION_ERROR_MESSAG= E =3D
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 "Serializer consumed more bytes than the record had.= " +
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0=C2=A0"This indicates broken serialization. If yo= u are using custom serialization types " +
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0=C2=A0"(Val= ue or Writable), check their serialization methods. If you are using a &quo= t; +
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0=C2=A0"Kryo-serialized type, check the corresponding = Kryo serializer.";

Any clue about how to find what = is causing this?




On Sat, May 21, 2016 at 10:53 AM,= Flavio Pompermaier <pompermaier@okkam.it> wrote:
I tried to move flink tmp dir from hdd disks = to sdd ones (in order to =C2=A0exclude faulty disks) and I had another of t= hose Exception:

java.lang.IllegalArgumentException:= The datetime zone id 'Europe/Romd' is not recognised
at org.joda.time.DateTimeZo= ne.forID(DateTimeZone.java:229)
at de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerialize= r.readTimeZone(JodaDateTimeSerializer.java:94)
at de.javakaffee.kryoserializers.jodatime.JodaDa= teTimeSerializer.read(JodaDateTimeSerializer.java:74)
at de.javakaffee.kryoserializers.jodatime= .JodaDateTimeSerializer.read(JodaDateTimeSerializer.java:59)
at com.esotericsoftware.kryo.Kryo.= readClassAndObject(Kryo.java:761)
at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSeria= lizer.deserialize(KryoSerializer.java:228)
at org.apache.flink.api.java.typeutils.runtime.kryo.= KryoSerializer.deserialize(KryoSerializer.java:242)
at org.apache.flink.api.java.typeuti= ls.runtime.PojoSerializer.deserialize(PojoSerializer.java:501)
at org.apache.flink.api.java.typ= eutils.runtime.PojoSerializer.deserialize(PojoSerializer.java:441)
at org.apache.flink.ru= ntime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializa= tionDelegate.java:55)
at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptive= SpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDese= rializer.java:124)
a= t org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.getNe= xtRecord(AbstractRecordReader.java:65)
at org.apache.flink.runtime.io.network.api.reader.Mutabl= eRecordReader.next(MutableRecordReader.java:34)
at org.apache.flink.runtime.operators.util.Read= erIterator.next(ReaderIterator.java:73)
at org.apache.flink.runtime.operators.FlatMapDriver.run= (FlatMapDriver.java:101)
at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480= )
at org.apache.flin= k.runtime.operators.BatchTask.invoke(BatchTask.java:345)
at org.apache.flink.runtime.taskmanage= r.Task.run(Task.java:559)
<= /span>at java.lang.Thread.run(Thread.java:745)
=


On= Fri, May 20, 2016 at 8:34 PM, Flavio Pompermaier <pompermaier@okkam.it= > wrote:

Right now I'm= using Flink 1.0.2...to which version should I downgrade?
The hardware seems to be ok..how could I detect a faulty hardware?
These errors appeared in every run of my job after I moved the temporary di= rectory from ssd to hdd and I extended my pipeline with a dataset that grow= s as the pipeline goes on,accumulating data from intermediate datasets..

On 20 May 2016 18:31, "Fabian Hueske" = <fhueske@gmail.co= m> wrote:
=
The problem seems to occur quite often.
Did you update your Flink = version recently? If so, could you try to downgrade and see if the problem = disappears.

Is it otherwise possible that it is cause by = faulty hardware?

2016-05-20 18:05 GMT+02:00 Flavio Pompermaier <pomperm= aier@okkam.it>:
This tim= e (Europed instead of Europe):

java.lang.Exception: The data pr=
eparation for task 'CHAIN GroupReduce (GroupReduce at createResult(Pass=
aggioWithComprAndVend.java:132)) -> Map (Key Extractor)' , caused an=
 error: Error obtaining the sorted input: Thread 'SortMerger spilling t=
hread' terminated due to an exception: The datetime zone id 'Europd=
/Rome' is not recognised
	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:456)
	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:345)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error obtaining the sorted input: Th=
read 'SortMerger spilling thread' terminated due to an exception: T=
he datetime zone id 'Europd/Rome' is not recognised
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterato=
r(UnilateralSortMerger.java:619)
	at org.apache.flink.runtime.operators.BatchTask.getInput(BatchTask.java:10=
79)
	at org.apache.flink.runtime.operators.GroupReduceDriver.prepare(GroupReduc=
eDriver.java:94)
	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:450)
	... 3 more
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'=
 terminated due to an exception: The datetime zone id 'Europd/Rome'=
 is not recognised
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase=
.run(UnilateralSortMerger.java:800)
Caused by: java.lang.IllegalArgumentException: The datetime zone id 'Eu=
ropd/Rome' is not recognised
	at org.joda.time.DateTimeZone.forID(DateTimeZone.java:229)
	at de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.readTimeZ=
one(JodaDateTimeSerializer.java:94)
	at de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.read(Joda=
DateTimeSerializer.java:74)
	at de.javakaffee.kryoserializers.jodatime.JodaDateTimeSerializer.read(Joda=
DateTimeSerializer.java:59)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deseria=
lize(KryoSerializer.java:228)
	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deseria=
lize(KryoSerializer.java:242)
	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.copy(Kr=
yoSerializer.java:252)
	at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(PojoSer=
ializer.java:556)
	at org.apache.flink.api.java.typeutils.runtime.TupleSerializerBase.copy(Tu=
pleSerializerBase.java:75)
	at org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutp=
ut(NormalizedKeySorter.java:499)
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingTh=
read.go(UnilateralSortMerger.java:1344)
	at org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase=
.run(UnilateralSortMerger.java:796)


On Fri, May 20, 2016 at 4:25 PM, Flav= io Pompermaier <pompermaier@okkam.it> wrote:
This time another error (rerialization instead of = serialization):

com.esotericsoftware.kryo.KryoException: Unable to find class: it.=
okkam.flink.entitons.rerialization.pojo.EntitonQuadPojo
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultCla=
ssResolver.java:138)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultCl=
assResolver.java:115)
	at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:641)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:752)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collect=
ionSerializer.java:116)
	at com.esotericsoftware.kryo.serializers.CollectionSerializer.read(Collect=
ionSerializer.java:22)
	at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
	at org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deseria=
lize(KryoSerializer.java:228)
	at org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize(=
PojoSerializer.java:431)
	at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize=
(TupleSerializer.java:135)
	at org.apache.flink.api.java.typeutils.runtime.TupleSerializer.deserialize=
(TupleSerializer.java:30)
	at org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.rea=
d(NonReusingDeserializationDelegate.java:55)
	at org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveS=
panningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeser=
ializer.java:124)
	at org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.get=
NextRecord(AbstractRecordReader.java:65)
	at org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.next=
(MutableRecordReader.java:34)
	at org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderItera=
tor.java:73)
	at org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.java=
:101)
	at org.apache.flink.runtime.operators.BatchTask.run(BatchTask.java:480)
	at org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.java:345)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:559)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: it.okkam.flink.entitons.rerial=
ization.pojo.EntitonQuadPojo
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultCla=
ssResolver.java:136)
	... 20 more


On Fri, May= 20, 2016 at 12:48 PM, Flavio Pompermaier <pompermaier@okkam.it>= wrote:
Hi Ufuk,
my records= could be quite large Pojos (I think some MB).
The only thing I d= o to configure Kryo is:

=C2=A0e= nv.registerTypeWithKryoSerializer(DateTime.cla= ss, JodaDateTimeSerializer.class );

Best,
= Flavio

On Fri, May 20, 2016 at 12:38 PM, Ufuk Celebi <uce@apache= .org> wrote:
@Stefano: the records are se= rialized anyway for batch jobs. The
spilling deserializer is only relevant if single records are very
large. How large are your records? In any case, I don't expect this to<= br> be the problem.

@Flavio: The class name "typo" errors (Vdhicle instead of Vehicle= and
ttil instead of util) look like some kind of data corruption and would
need further investigation. The other failure you reported might be
related to this. As a starting point, how do you configure the Kryo
serializer?

On Fri, May 20, 2016 at 10:02 AM, Flavio Pompermaier
<pom= permaier@okkam.it> wrote:
> Today I've got this other strange error.. Obviously I don't ha= ve a
> VdhicleEvent class, but a VehicleEvent class :(
>
> java.lang.RuntimeException: Cannot instantiate class.
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize= (PojoSerializer.java:407)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.re= ad(NonReusingDeserializationDelegate.java:55)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptive= SpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDese= rializer.java:124)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.runtime.io.network.api.reader.AbstractRecordReader.ge= tNextRecord(AbstractRecordReader.java:65)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.runtime.io.network.api.reader.MutableRecordReader.nex= t(MutableRecordReader.java:34)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.runtime.operators.util.ReaderIterator.next(ReaderIter= ator.java:73)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDriver.jav= a:101)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.flink.runtime.operators.BatchT= ask.run(BatchTask.java:480)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.flink.runtime.operators.BatchT= ask.invoke(BatchTask.java:345)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.flink.runtime.taskmanager.Task= .run(Task.java:559)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> it.okkam.flink.test.model.pojo.VdhicleEvent
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.URLClassLoader.findClass(URLClas= sLoader.java:381)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.ClassLoader.loadClass(ClassLoad= er.java:424)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.ClassLoader.loadClass(ClassLoad= er.java:357)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Class.forName0(Native Method) >=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Class.forName(Class.java:348) >=C2=A0 =C2=A0 =C2=A0 =C2=A0at
> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.deserialize= (PojoSerializer.java:405)
>=C2=A0 =C2=A0 =C2=A0 =C2=A0... 10 more
>
>
> Thanks in advance,
> Flavio
>
>
> On Mon, May 16, 2016 at 10:08 AM, Stefano Bortoli <s.bortoli@gmail.com>
> wrote:
>>
>> Hi Flavio, Till,
>>
>> do you think this can be possibly related to the serialization pro= blem
>> caused by 'the management' of Kryo serializer buffer when = spilling on disk?
>> We are definitely going beyond what is managed in memory with this= task.
>>
>> saluti,
>> Stefano
>>
>> 2016-05-16 9:44 GMT+02:00 Flavio Pompermaier <pompermaier@okkam.it>:
>>>
>>> That exception showed just once, but the following happens ran= domly (if I
>>> re-run the job after stopping and restartign the cluster it do= esn't show up
>>> usually):
>>>
>>> Caused by: java.io.IOException: Serializer consumed more bytes= than the
>>> record had. This indicates broken serialization. If you are us= ing custom
>>> serialization types (Value or Writable), check their serializa= tion methods.
>>> If you are using a Kryo-serialized type, check the correspondi= ng Kryo
>>> serializer.
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.io.network.api.serialization.Spilling= AdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRe= cordDeserializer.java:142)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.io.network.api.reader.AbstractRecordR= eader.getNextRecord(AbstractRecordReader.java:65)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.io.network.api.reader.MutableRecordRe= ader.next(MutableRecordReader.java:34)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.operators.util.ReaderIterator.next(Re= aderIterator.java:73)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.operators.FlatMapDriver.run(FlatMapDr= iver.java:101)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask.jav= a:480)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.operators.BatchTask.invoke(BatchTask.= java:345)
>>>=C2=A0 =C2=A0 =C2=A0at org.apache.flink.runtime.taskmanager.Tas= k.run(Task.java:559)
>>>=C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:745) >>> Caused by: java.lang.ArrayIndexOutOfBoundsException: -2
>>>=C2=A0 =C2=A0 =C2=A0at java.util.ArrayList.elementData(ArrayLis= t.java:418)
>>>=C2=A0 =C2=A0 =C2=A0at java.util.ArrayList.get(ArrayList.java:4= 31)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObj= ect(MapReferenceResolver.java:42)
>>>=C2=A0 =C2=A0 =C2=A0at com.esotericsoftware.kryo.Kryo.readRefer= enceOrNull(Kryo.java:805)
>>>=C2=A0 =C2=A0 =C2=A0at com.esotericsoftware.kryo.Kryo.readClass= AndObject(Kryo.java:759)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerialize= r.deserialize(KryoSerializer.java:228)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.api.java.typeutils.runtime.PojoSerializer.des= erialize(PojoSerializer.java:431)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.de= serialize(TupleSerializer.java:135)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.api.java.typeutils.runtime.TupleSerializer.de= serialize(TupleSerializer.java:30)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.plugable.NonReusingDeserializationDel= egate.read(NonReusingDeserializationDelegate.java:55)
>>>=C2=A0 =C2=A0 =C2=A0at
>>> org.apache.flink.runtime.io.network.api.serialization.Spilling= AdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRe= cordDeserializer.java:124)
>>>
>>>
>>> On Sun, May 15, 2016 at 10:58 AM, Flavio Pompermaier
>>> <= pompermaier@okkam.it> wrote:
>>>>
>>>> Hi to all,
>>>> in my last run of a job I received this weird Kryo Excepti= on in one of
>>>> the TaskManager...obviously this class in not mentioned an= ywhere, neither in
>>>> my project nor in flink...
>>>> Any help is appreciated!
>>>>
>>>> Best,
>>>> Flavio
>>>>
>>>> INFO=C2=A0 org.apache.flink.runtime.taskmanager.Task - CHA= IN GroupReduce
>>>> (GroupReduce at createResult(IndexMappingExecutor.java:42)= ) -> Map (Map at
>>>> main(Jsonizer.java:128)) (4/9) switched to FAILED with exc= eption.
>>>> java.lang.Exception: The data preparation for task 'CH= AIN GroupReduce
>>>> (GroupReduce at createResult(IndexMappingExecutor.java:42)= ) -> Map (Map at
>>>> main(Jsonizer.java:128))' , caused an error: Error obt= aining the sorted
>>>> input: Thread 'SortMerger spilling thread' termina= ted due to an exception:
>>>> Unable to find class: java.ttil.HashSet
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask= .java:456)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.BatchTask.invoke(BatchT= ask.java:345)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.flink.runti= me.taskmanager.Task.run(Task.java:559)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(T= hread.java:745)
>>>> Caused by: java.lang.RuntimeException: Error obtaining the= sorted input:
>>>> Thread 'SortMerger spilling thread' terminated due= to an exception: Unable
>>>> to find class: java.ttil.HashSet
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerg= er.getIterator(UnilateralSortMerger.java:619)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.BatchTask.getInput(Batc= hTask.java:1079)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.GroupReduceDriver.prepa= re(GroupReduceDriver.java:94)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.BatchTask.run(BatchTask= .java:450)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0... 3 more
>>>> Caused by: java.io.IOException: Thread 'SortMerger spi= lling thread'
>>>> terminated due to an exception: Unable to find class: java= .ttil.HashSet
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerg= er$ThreadBase.run(UnilateralSortMerger.java:800)
>>>> Caused by: com.esotericsoftware.kryo.KryoException: Unable= to find
>>>> class: java.ttil.HashSet
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readNa= me(DefaultClassResolver.java:138)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readCl= ass(DefaultClassResolver.java:115)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at com.esotericsoftware.k= ryo.Kryo.readClass(Kryo.java:641)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.jav= a:752)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> com.esotericsoftware.kryo.serializers.MapSerializer.read(M= apSerializer.java:143)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> com.esotericsoftware.kryo.serializers.MapSerializer.read(M= apSerializer.java:21)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.jav= a:761)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSeria= lizer.deserialize(KryoSerializer.java:228)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSeria= lizer.deserialize(KryoSerializer.java:242)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSeria= lizer.copy(KryoSerializer.java:252)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.api.java.typeutils.runtime.PojoSerializer= .copy(PojoSerializer.java:556)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.api.java.typeutils.runtime.TupleSerialize= rBase.copy(TupleSerializerBase.java:75)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.sort.NormalizedKeySorte= r.writeToOutput(NormalizedKeySorter.java:499)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerg= er$SpillingThread.go(UnilateralSortMerger.java:1344)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> org.apache.flink.runtime.operators.sort.UnilateralSortMerg= er$ThreadBase.run(UnilateralSortMerger.java:796)
>>>> Caused by: java.lang.ClassNotFoundException: java.ttil.Has= hSet
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.net.URLClassLoade= r.findClass(URLClassLoader.java:381)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.ClassLoader.= loadClass(ClassLoader.java:424)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.ClassLoader.= loadClass(ClassLoader.java:357)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Class.forNam= e0(Native Method)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Class.forNam= e(Class.java:348)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0at
>>>> com.esotericsoftware.kryo.util.DefaultClassResolver.readNa= me(DefaultClassResolver.java:136)
>>>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0... 14 more
>>>>
>>>
>>
>

<= /p>








=


--001a114034e47269ce05337d7073--