Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 5C2C318129 for ; Wed, 20 Apr 2016 21:50:06 +0000 (UTC) Received: (qmail 80955 invoked by uid 500); 20 Apr 2016 21:50:01 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 80866 invoked by uid 500); 20 Apr 2016 21:50:01 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 80856 invoked by uid 99); 20 Apr 2016 21:50:01 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 20 Apr 2016 21:50:01 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 873E91A05BC for ; Wed, 20 Apr 2016 21:50:00 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.28 X-Spam-Level: * X-Spam-Status: No, score=1.28 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=braintreepayments-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id vY3VI9OWF_kO for ; Wed, 20 Apr 2016 21:49:56 +0000 (UTC) Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id AB5B25FB19 for ; Wed, 20 Apr 2016 21:49:55 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id ys16so16899812lbb.3 for ; Wed, 20 Apr 2016 14:49:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=braintreepayments-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=mHqJw7Cnvnui+DNfRhULRg+2u81G2hf+hRqN29xNJJg=; b=Sn+BRc29up7OC/kMWe2/rMCJwIf6WLCbTfQNVhT5Bpkbp/qY4gPWML1gUVlqhDW7SC CbD1NYtZBiyy564Rpj5Rv1N7JhJYbXtl4+6Mxx0iR2c9doqkrT4WDqvcCfWZDyBgf4xD G+zbxccshTS9vSsqTK2diJ3CD9AtFco3+zfO5sibfQaCUjJffUUStwNg2uMb2ZmEcOiD LhEGLQmoKSQOMSuUCs/XgydXN2z2U6pWVSdSTcggrblucODef7EvtYXgmYZUtBERQqPw qzBP5LOWKhodH5MMAkEbNgMp9pnmrfswlBQjKlTX4GsDFsL0/CJqMDukiH+1d+ZCG8Ck 7q8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=mHqJw7Cnvnui+DNfRhULRg+2u81G2hf+hRqN29xNJJg=; b=lNAGy2WEL+cKuIFwZpE7ifbyTXdtqmv41lU1xuV/Yvas4kv0Iv8ZB3YQtCgHiYxcyt Uum8WLwQZmNYMX5pRXaEzlzTNG7vFEj2+A4Vf85p9dVLmIwaf75463iqLLbfSrBDnq9J agf5iGF/5zbD3rTkj11PG9qQEqd6tSboCPlk07AvK3nGFsY3shQ0LDOgDxDkg5fIzbQ5 kLWd8yYqClekoTzP8XNRDVVrOS+JVPuai3806xICcuBlT/lpNyG4RRXkh97tgi2Xl2V2 919tAMkg5DInx2vKqttVsV8hHtF8r3NBEbDl4e20Z8WJyGTOgUrcIBDokZlmBhW7G9uD 55gA== X-Gm-Message-State: AOPr4FU+A9EFbtXaLtG7uJL/QICxVDW8/YA2Hf8wkaGph3bQFAUTVVFVG6ZM8QiPaWXzVUaTYekyVOwNaaNh/nnU X-Received: by 10.112.129.169 with SMTP id nx9mr4718594lbb.96.1461188994300; Wed, 20 Apr 2016 14:49:54 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: David Kim Date: Wed, 20 Apr 2016 21:49:44 +0000 Message-ID: Subject: Re: asm IllegalArgumentException with 1.0.0 To: user@flink.apache.org Content-Type: multipart/alternative; boundary=047d7b3441daf493fc0530f19213 --047d7b3441daf493fc0530f19213 Content-Type: text/plain; charset=UTF-8 Hi Stephan! Following up on this issue, it seems the issue doesn't show itself when using version 1.0.1. I'm able to run our unit tests in IntelliJ now. Thanks! David On Wed, Apr 13, 2016 at 1:59 PM Stephan Ewen wrote: > Does this problem persist? (It might have been caused by maven caches with > bad artifacts). > > The many transitive dependencies you see often come from the connectors - > that is also why we do not put the connectors into the lib folder directly, > so that these libraries are not always part of every Flink program. > > > On Mon, Mar 14, 2016 at 9:16 PM, Zach Cox wrote: > >> Yes pretty much - we use sbt to run the job in a local environment, not >> Intellij, but should be the same thing. We were also seeing that exception >> running unit tests locally. We did not see the exception when assembling a >> fat jar and submitting to a remote Flink cluster. >> >> It seems like the flink-connector-elasticsearch jar should not have >> shaded classes in it. Maybe that jar in maven central was built incorrectly? >> >> We worked around this by just not depending on that elasticsearch >> connector at all, since we wrote our own connector for Elasticsearch 2.x. >> >> -Zach >> >> >> On Mon, Mar 14, 2016 at 2:03 PM Andrew Whitaker < >> andrew.whitaker@braintreepayments.com> wrote: >> >>> We're having the same issue (we also have a dependency on >>> flink-connector-elasticsearch). It's only happening to us in IntelliJ >>> though. Is this the case for you as well? >>> >>> On Thu, Mar 10, 2016 at 3:20 PM, Zach Cox wrote: >>> >>>> After some poking around I noticed >>>> that flink-connector-elasticsearch_2.10-1.0.0.jar contains shaded asm >>>> classes. If I remove that dependency from my project then I do not get the >>>> IllegalArgumentException. >>>> >>>> >>>> On Thu, Mar 10, 2016 at 11:51 AM Zach Cox wrote: >>>> >>>>> Here are the jars on the classpath when I try to run our Flink job in >>>>> a local environment (via `sbt run`): >>>>> >>>>> >>>>> https://gist.githubusercontent.com/zcox/0992aba1c517b51dc879/raw/7136ec034c2beef04bd65de9f125ce3796db511f/gistfile1.txt >>>>> >>>>> There are many transitive dependencies pulled in from internal library >>>>> projects that probably need to be cleaned out. Maybe we are including >>>>> something that conflicts? Or maybe something important is being excluded? >>>>> >>>>> Are the asm classes included in Flink jars in some shaded form? >>>>> >>>>> Thanks, >>>>> Zach >>>>> >>>>> >>>>> On Thu, Mar 10, 2016 at 5:06 AM Stephan Ewen wrote: >>>>> >>>>>> Dependency shading changed a bit between RC4 and RC5 - maybe a >>>>>> different minor ASM version is now included in the "test" scope. >>>>>> >>>>>> Can you share the dependencies of the problematic project? >>>>>> >>>>>> On Thu, Mar 10, 2016 at 12:26 AM, Zach Cox wrote: >>>>>> >>>>>>> I also noticed when I try to run this application in a local >>>>>>> environment, I get the same IllegalArgumentException. >>>>>>> >>>>>>> When I assemble this application into a fat jar and run it on a >>>>>>> Flink cluster using the CLI tools, it seems to run fine. >>>>>>> >>>>>>> Maybe my local classpath is missing something that is provided on >>>>>>> the Flink task managers? >>>>>>> >>>>>>> -Zach >>>>>>> >>>>>>> >>>>>>> On Wed, Mar 9, 2016 at 5:16 PM Zach Cox wrote: >>>>>>> >>>>>>>> Hi - after upgrading to 1.0.0, I'm getting this exception now in a >>>>>>>> unit test: >>>>>>>> >>>>>>>> IllegalArgumentException: (null:-1) >>>>>>>> org.apache.flink.shaded.org.objectweb.asm.ClassVisitor.(Unknown >>>>>>>> Source) >>>>>>>> org.apache.flink.shaded.org.objectweb.asm.ClassVisitor.(Unknown >>>>>>>> Source) >>>>>>>> >>>>>>>> org.apache.flink.api.scala.InnerClosureFinder.(ClosureCleaner.scala:279) >>>>>>>> >>>>>>>> org.apache.flink.api.scala.ClosureCleaner$.getInnerClasses(ClosureCleaner.scala:95) >>>>>>>> >>>>>>>> org.apache.flink.api.scala.ClosureCleaner$.clean(ClosureCleaner.scala:115) >>>>>>>> >>>>>>>> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.scalaClean(StreamExecutionEnvironment.scala:568) >>>>>>>> >>>>>>>> org.apache.flink.streaming.api.scala.StreamExecutionEnvironment.addSource(StreamExecutionEnvironment.scala:498) >>>>>>>> >>>>>>>> The line that causes that exception is just adding >>>>>>>> a FlinkKafkaConsumer08 source. >>>>>>>> >>>>>>>> ClassVisitor [1] seems to throw that IllegalArgumentException when >>>>>>>> it is not given a valid api version number, but InnerClosureFinder [2] >>>>>>>> looks fine to me. >>>>>>>> >>>>>>>> Any idea what might be causing this? This unit test worked fine >>>>>>>> with 1.0.0-rc0 jars. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Zach >>>>>>>> >>>>>>>> [1] >>>>>>>> http://websvn.ow2.org/filedetails.php?repname=asm&path=%2Ftrunk%2Fasm%2Fsrc%2Forg%2Fobjectweb%2Fasm%2FClassVisitor.java >>>>>>>> [2] >>>>>>>> https://github.com/apache/flink/blob/master/flink-scala/src/main/scala/org/apache/flink/api/scala/ClosureCleaner.scala#L279 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>> >>> >>> -- >>> Andrew Whitaker | andrew.whitaker@braintreepayments.com >>> -- >>> Note: this information is confidential. It is prohibited to share, post >>> online or otherwise publicize without Braintree's prior written consent. >>> >> > --047d7b3441daf493fc0530f19213 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Stephan!

Following up on this issue,= it seems the issue doesn't show itself when using version 1.0.1. I'= ;m able to run our unit tests in IntelliJ now.

Tha= nks!
David

On Wed, Apr 13, 2016 at 1:59 PM Stephan Ewen <sewen@apache.org> wrote:
Does this problem persist? (It might have be= en caused by maven caches with bad artifacts).

The many = transitive dependencies you see often come from the connectors - that is al= so why we do not put the connectors into the lib folder directly, so that t= hese libraries are not always part of every Flink program.


On Mon,= Mar 14, 2016 at 9:16 PM, Zach Cox <zcox522@gmail.com> wrote= :
Yes pretty much - we u= se sbt to run the job in a local environment, not Intellij, but should be t= he same thing. We were also seeing that exception running unit tests locall= y. We did not see the exception when assembling a fat jar and submitting to= a remote Flink cluster.

It seems like the flink-connect= or-elasticsearch jar should not have shaded classes in it. Maybe that jar i= n maven central was built incorrectly?

We worked a= round this by just not depending on that elasticsearch connector at all, si= nce we wrote our own connector for Elasticsearch 2.x.

-Zach

<= /div>

On Mon, Mar = 14, 2016 at 2:03 PM Andrew Whitaker <andrew.whitaker@braintreepayments.c= om> wrote:
= We're having the same issue (we also have a dependency on flink-connect= or-elasticsearch). It's only happening to us in IntelliJ though. Is thi= s the case for you as well?

On Thu, Mar 10, 2016 at 3:= 20 PM, Zach Cox <zcox522@gmail.com> wrote:
After some poking around I noticed that= =C2=A0flink-connector-elasticsearch_2.10-1.0.0.jar contains shaded asm clas= ses. If I remove that dependency from my project then I do not get the Ille= galArgumentException.


On Thu, Mar 10, 2016 at 11:51 AM Zach Cox <zcox522@gmail.com>= ; wrote:
Here are = the jars on the classpath when I try to run our Flink job in a local enviro= nment (via `sbt run`):=C2=A0


There are many transitiv= e dependencies pulled in from internal library projects that probably need = to be cleaned out. Maybe we are including something that conflicts? Or mayb= e something important is being excluded?

Are the a= sm classes included in Flink jars in some shaded form?

=
Thanks,
Zach


On Thu, Mar 10, 2016 at 5:06 AM Stephan Ewen <= ;sewen@apache.org= > wrote:
Depend= ency shading changed a bit between RC4 and RC5 - maybe a different minor AS= M version is now included in the "test" scope.

Can you share the dependencies of the problematic project?

On Thu, Mar 10, 2016 = at 12:26 AM, Zach Cox <zcox522@gmail.com> wrote:
I also noticed when I try to r= un this application in a local environment, I get the same IllegalArgumentE= xception.

When I assemble this application into a fat j= ar and run it on a Flink cluster using the CLI tools, it seems to run fine.= =C2=A0

Maybe my local classpath is missing something tha= t is provided on the Flink task managers?

-Zach


On Wed, Mar 9, 2016 at 5= :16 PM Zach Cox <= zcox522@gmail.com> wrote:
Hi - after upgrading to 1.0.0, I'm getting this exceptio= n now in a unit test:

=C2=A0 =C2=A0IllegalArgumentE= xception: =C2=A0 (null:-1)
org.apache.flink.shaded.org.objectweb.= asm.ClassVisitor.<init>(Unknown Source)
org.apache.flink.sh= aded.org.objectweb.asm.ClassVisitor.<init>(Unknown Source)
= org.apache.flink.api.scala.InnerClosureFinder.<init>(ClosureCleaner.s= cala:279)
org.apache.flink.api.scala.ClosureCleaner$.getInnerClas= ses(ClosureCleaner.scala:95)
org.apache.flink.api.scala.ClosureCl= eaner$.clean(ClosureCleaner.scala:115)
org.apache.flink.streaming= .api.scala.StreamExecutionEnvironment.scalaClean(StreamExecutionEnvironment= .scala:568)
org.apache.flink.streaming.api.scala.StreamExecutionE= nvironment.addSource(StreamExecutionEnvironment.scala:498)
=
The line that causes that exception is just adding a=C2=A0Fl= inkKafkaConsumer08 source.

ClassVisitor [1] seems = to throw that IllegalArgumentException when it is not given a valid api ver= sion number, but InnerClosureFinder [2] looks fine to me.

Any idea what might be causing this? This unit test worked fine wit= h 1.0.0-rc0 jars.

Thanks,
Zach







<= div class=3D"gmail_extra">--
--
Note: this information is con= fidential. It is prohibited to share, post online or otherwise publicize wi= thout Braintree's prior written consent.

--047d7b3441daf493fc0530f19213--