From user-return-27705-archive-asf-public=cust-asf.ponee.io@flink.apache.org Thu May 16 08:15:04 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 42D7618066B for ; Thu, 16 May 2019 10:15:04 +0200 (CEST) Received: (qmail 76628 invoked by uid 500); 16 May 2019 08:14:59 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 76618 invoked by uid 99); 16 May 2019 08:14:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 May 2019 08:14:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id E5A3DC2C80 for ; Thu, 16 May 2019 08:14:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.002 X-Spam-Level: ** X-Spam-Status: No, score=2.002 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=ververica-com.20150623.gappssmtp.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id M5Z8vREmjBtz for ; Thu, 16 May 2019 08:14:57 +0000 (UTC) Received: from mail-vs1-f47.google.com (mail-vs1-f47.google.com [209.85.217.47]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 530FA5F644 for ; Thu, 16 May 2019 08:14:56 +0000 (UTC) Received: by mail-vs1-f47.google.com with SMTP id j184so1717547vsd.11 for ; Thu, 16 May 2019 01:14:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ververica-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2qH2ZyQ8XscBq/MAjoesE1Mt1A+cscah+5pQckC/ANg=; b=h5YAXbypQrYIMbXqEG7VkVNRLLEjjGwoa+MLHhtrxPICoNb35H7W4pHr60AJcz82x+ sy/RDb2KsCCxT0jcDT6I03iTNRU8Flfbd213KSzq5LLSCeRNpWa3NzXbNeQYwrETj9ev 9uH406nSWimyk46BipbXAoLlIQtl9vIOkNzju/3yvG1iBIe7SKYIdQBG26X6IUskyK3j VUPpLKetbEw8Q9xgmV37h5ly0Hs0JBT7DfkxwOCrzK4WwPkAQn39XN2D0BjaQ9Pmoo8o +k+MJgvEJN7xGNZnLJmU/NHhbtn5kkxLT7+S3R/FjCR5r1CqwKPmEeOhh6j5f3z50Dv+ x01Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2qH2ZyQ8XscBq/MAjoesE1Mt1A+cscah+5pQckC/ANg=; b=LUEeNmA9eSVEDvYR4TWYtEym0L72jGXg12b6lK5QkT6hXhXD/PQvhKlZcCsVLqDK/x ddLBT/dTkz7k9kE1mpuXOXUuCp2IDbDsq67YsaC7pNTH63q8Vt862+Lx+flSzE6OknDx bqmjj908CezLOhhrARxsfNY2bPsxHAFumvmeHzdVKslUYWin5ToLisDdkhEkGDe0yC4C r/qSLkZDPxJQrmIBdfvNRtAi4NOxW7OvlRdBT3CNIvxksdJa5xbkYO1Vy4F7bY8N4Jak E5fQBt/YwIXZw1lyEIFhODZeLQpiuf6UYCJht+579SVSjkuPQf0A/3AxGACz5eFMEK9a my8A== X-Gm-Message-State: APjAAAXv9WQ9rj/+hapzJJQl2KdbmHC9EyWd7+/u0Gtuzbk61jU9koqy hsmxrbr0EGdF7n80jUTZKA/pDtjU+/nFvi775kTGAA== X-Google-Smtp-Source: APXvYqzzTZ72cRMdLSGX8AfsiUSAt8a2zZGZJSd4FRdR1TeeVvAqttK4qtoPHzN0CWxh/s8wgEx4UwFfh2g8I35k0sM= X-Received: by 2002:a67:ebd6:: with SMTP id y22mr7817362vso.87.1557994494712; Thu, 16 May 2019 01:14:54 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Andrey Zagrebin Date: Thu, 16 May 2019 10:14:43 +0200 Message-ID: Subject: Re: Getting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError when stopping/canceling job. To: John Smith Cc: user Content-Type: multipart/alternative; boundary="0000000000006b73f20588fcdcf5" --0000000000006b73f20588fcdcf5 Content-Type: text/plain; charset="UTF-8" Could you share the source code of your RichAsyncFunction? Looks like netty threads of vertx are still being shutdown after user code class loader has been shutdown. It probably means that RichAsyncFunction was not closed properly or not all resources have been fully freed there (logging your RichAsyncFunction.close could help). Do you mean cancelation by stopping the job? On Wed, May 15, 2019 at 10:02 PM John Smith wrote: > So these are the two exceptions I see in the logs... > > Exception in thread "vert.x-worker-thread-0" Exception in thread > "vert.x-internal-blocking-0" java.lang.NoClassDefFoundError: > io/netty/util/concurrent/FastThreadLocal > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:32) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > io.netty.util.concurrent.FastThreadLocal > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 2 more > java.lang.NoClassDefFoundError: io/netty/util/concurrent/FastThreadLocal > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:32) > at java.lang.Thread.run(Thread.java:748) > May 15, 2019 10:42:07 AM io.vertx.core.impl.ContextImpl > SEVERE: Unhandled exception > java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: > io/vertx/core/impl/VertxImpl$SharedWorkerPool > at > io.vertx.core.impl.VertxImpl.lambda$deleteCacheDirAndShutdown$25(VertxImpl.java:830) > at io.vertx.core.impl.ContextImpl.lambda$null$0(ContextImpl.java:284) > at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:320) > at > io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) > at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) > at > io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.NoClassDefFoundError: > io/vertx/core/impl/VertxImpl$SharedWorkerPool > ... 10 more > Caused by: java.lang.ClassNotFoundException: > io.vertx.core.impl.VertxImpl$SharedWorkerPool > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at > org.apache.flink.runtime.execution.librarycache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeClassLoaders.java:129) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 10 more > > On Wed, 15 May 2019 at 12:00, Andrey Zagrebin > wrote: > >> Hi John, >> >> could you share the full stack trace or better logs? >> It looks like something is trying to be executed in vertx.io code after >> the local task has been stopped and the class loader for the user code has >> been unloaded. Maybe from some daemon thread pool. >> >> Best, >> Andrey >> >> >> On Wed, May 15, 2019 at 4:58 PM John Smith >> wrote: >> >>> Hi, >>> >>> I'm using vertx.io as an async JDBC client for a RichAsyncFunction it >>> works fine but when I stop the job I get... >>> >>> java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: >>> io/vertx/core/impl/VertxImpl$SharedWorkerPool >>> >>> Is there a way to avoid/fix this? >>> >> --0000000000006b73f20588fcdcf5 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Could you share the source code of your RichAsyncFunction?=
Looks like netty threads of vertx are still being shutdown after user = code class loader has been shutdown.
It probably means that RichA= syncFunction was not closed properly or not all resources have been fully f= reed there (logging your RichAsyncFunction.close could help).
Do = you mean cancelation by stopping the job?

On Wed, May 15, 2019 at 10:0= 2 PM John Smith <java.dev.mtl@= gmail.com> wrote:
So these are the two exception= s I see in the logs...

Exception in thread "ve= rt.x-worker-thread-0" Exception in thread "vert.x-internal-blocki= ng-0" java.lang.NoClassDefFoundError: io/netty/util/concurrent/FastThr= eadLocal
at io.netty= .util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:3= 2)
at java.lang.Thre= ad.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundExcepti= on: io.netty.util.concurrent.FastThreadLocal
at java.net.URLClassLoader.findClass(URLClassLoade= r.java:381)
at java.= lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.runtime.execution.libraryc= ache.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCod= eClassLoaders.java:129)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 2 more
java.lang.NoC= lassDefFoundError: io/netty/util/concurrent/FastThreadLocal
at io.netty.util.concurrent.FastThr= eadLocalRunnable.run(FastThreadLocalRunnable.java:32)
at java.lang.Thread.run(Thread.java:748)<= /div>
May 15, 2019 10:42:07 AM io.vertx.core.impl.ContextImpl
SEVERE: Unhandled exception
java.lang.BootstrapMethodError: java= .lang.NoClassDefFoundError: io/vertx/core/impl/VertxImpl$SharedWorkerPool
at io.vertx.core.impl= .VertxImpl.lambda$deleteCacheDirAndShutdown$25(VertxImpl.java:830)
at io.vertx.core.impl.Contex= tImpl.lambda$null$0(ContextImpl.java:284)
at io.vertx.core.impl.ContextImpl.executeTask(Context= Impl.java:320)
at io= .vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.ja= va:38)
at io.netty.u= til.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java= :163)
at io.netty.ut= il.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecut= or.java:404)
at io.n= etty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
at io.netty.util.concurrent.SingleThre= adEventExecutor$5.run(SingleThreadEventExecutor.java:897)
at io.netty.util.concurrent.FastThrea= dLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)<= /div>
Caused by: java.lang.NoClassDefFoundError: io/vertx/core/impl/Ver= txImpl$SharedWorkerPool
... 10 more
Caused by: java.lang.ClassNotFoundException: io.v= ertx.core.impl.VertxImpl$SharedWorkerPool
at java.net.URLClassLoader.findClass(URLClassLoader.j= ava:381)
at java.lan= g.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.flink.runtime.execution.librarycach= e.FlinkUserCodeClassLoaders$ChildFirstClassLoader.loadClass(FlinkUserCodeCl= assLoaders.java:129)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
On Wed, = 15 May 2019 at 12:00, Andrey Zagrebin <andrey@ververica.com> wrote:
Hi=C2=A0John,

could you share the full stack trace o= r better logs?
It looks like something is trying to be executed i= n vertx.io code after the= local task has been stopped and the class loader for the user code has bee= n unloaded. Maybe from some daemon thread pool.=C2=A0

<= div>Best,
Andrey


On Wed, May 15, 2019= at 4:58 PM John Smith <java.dev.mtl@gmail.com> wrote:
Hi,=C2= =A0

I'm using vertx.io as an async JDBC client for a RichAs= yncFunction it works fine but when I stop the job I get...

java.lang.BootstrapMethodError: java.lang.NoClass= DefFoundError: io/vertx/core/impl/VertxImpl$SharedWorkerPool
=
Is there a way to avoid/fix this?
--0000000000006b73f20588fcdcf5--