Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id E7056200C6A for ; Wed, 19 Apr 2017 20:42:54 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id E58B7160B9C; Wed, 19 Apr 2017 18:42:54 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id E04B1160B94 for ; Wed, 19 Apr 2017 20:42:53 +0200 (CEST) Received: (qmail 89334 invoked by uid 500); 19 Apr 2017 18:42:53 -0000 Mailing-List: contact issues-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@flink.apache.org Delivered-To: mailing list issues@flink.apache.org Received: (qmail 89325 invoked by uid 99); 19 Apr 2017 18:42:53 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 19 Apr 2017 18:42:53 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id B15231857B2 for ; Wed, 19 Apr 2017 18:42:52 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id AmcG4PVbWvfY for ; Wed, 19 Apr 2017 18:42:50 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id A5F255FB6A for ; Wed, 19 Apr 2017 18:42:49 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id E3D1BE0641 for ; Wed, 19 Apr 2017 18:42:43 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id A5CA721B48 for ; Wed, 19 Apr 2017 18:42:41 +0000 (UTC) Date: Wed, 19 Apr 2017 18:42:41 +0000 (UTC) From: "Luke Hutchison (JIRA)" To: issues@flink.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Wed, 19 Apr 2017 18:42:55 -0000 [ https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975268#comment-15975268 ] Luke Hutchison commented on FLINK-3328: --------------------------------------- [~StephanEwen] It's only not a problem if you know that in every case, the version of every single shared library will be the same between all Flink jars. And even if you know that will be the case, it's hardly ideal space-wise, because you end up pulling in the same dependencies multiple times for every Flink project, until a fat jar is built. In general, there is a good reason for that Maven warning, because if there's any way that multiple different versions of a dep can occur, you can trigger bugs that are maddeningly hard to trace. If every Flink jar (like `flink-runtime`) depends on one of the core Flink jars (e.g. `flink-core`), couldn't all the shared dependencies just be put in that one jar? > Incorrectly shaded dependencies in flink-runtime > ------------------------------------------------ > > Key: FLINK-3328 > URL: https://issues.apache.org/jira/browse/FLINK-3328 > Project: Flink > Issue Type: Bug > Components: Build System > Affects Versions: 1.0.0 > Reporter: Stephan Ewen > Assignee: Robert Metzger > Priority: Blocker > Fix For: 1.0.0 > > > There are apparently some dependencies shaded into {{flink-runtime}} fat jar that are not relocated. (the flink-runtime jar is now 70 MB) > From the output of the shading in flink-dist, it looks as if this concerns at least > - Zookeeper > - slf4j > - jline > - netty (3.x) > Possible more. > {code} > [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440 overlapping classes: > [WARNING] - org.apache.zookeeper.server.NettyServerCnxnFactory > [WARNING] - org.apache.jute.compiler.JFile > [WARNING] - org.apache.zookeeper.server.SessionTracker$Session > [WARNING] - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1 > [WARNING] - org.apache.jute.compiler.JLong > [WARNING] - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState > [WARNING] - org.apache.zookeeper.server.auth.KerberosName$Rule > [WARNING] - org.apache.jute.CsvOutputArchive > [WARNING] - org.apache.zookeeper.server.quorum.QuorumPeer > [WARNING] - org.apache.zookeeper.ZooKeeper$DataWatchRegistration > [WARNING] - 430 more... > [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24 overlapping classes: > [WARNING] - org.slf4j.spi.MarkerFactoryBinder > [WARNING] - org.slf4j.helpers.SubstituteLogger > [WARNING] - org.slf4j.helpers.BasicMarker > [WARNING] - org.slf4j.helpers.Util > [WARNING] - org.slf4j.LoggerFactory > [WARNING] - org.slf4j.Marker > [WARNING] - org.slf4j.helpers.NamedLoggerBase > [WARNING] - org.slf4j.Logger > [WARNING] - org.slf4j.spi.LocationAwareLogger > [WARNING] - org.slf4j.ILoggerFactory > [WARNING] - 14 more... > [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes: > [WARNING] - org.fusesource.jansi.Ansi$Erase > [WARNING] - org.fusesource.jansi.Ansi > [WARNING] - org.fusesource.jansi.AnsiOutputStream > [WARNING] - org.fusesource.jansi.internal.CLibrary > [WARNING] - org.fusesource.jansi.Ansi$2 > [WARNING] - org.fusesource.jansi.WindowsAnsiOutputStream > [WARNING] - org.fusesource.jansi.AnsiRenderer$Code > [WARNING] - org.fusesource.jansi.AnsiConsole > [WARNING] - org.fusesource.jansi.Ansi$Attribute > [WARNING] - org.fusesource.jansi.internal.Kernel32 > [WARNING] - 13 more... > [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar, commons-beanutils-1.7.0.jar define 10 overlapping classes: > [WARNING] - org.apache.commons.collections.FastHashMap$EntrySet > [WARNING] - org.apache.commons.collections.ArrayStack > [WARNING] - org.apache.commons.collections.FastHashMap$1 > [WARNING] - org.apache.commons.collections.FastHashMap$KeySet > [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView > [WARNING] - org.apache.commons.collections.BufferUnderflowException > [WARNING] - org.apache.commons.collections.Buffer > [WARNING] - org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator > [WARNING] - org.apache.commons.collections.FastHashMap$Values > [WARNING] - org.apache.commons.collections.FastHashMap > [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar, flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar, flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar, flink-scala_2.10-1.0-SNAPSHOT.jar, flink-clients_2.10-1.0-SNAPSHOT.jar, flink-optimizer_2.10-1.0-SNAPSHOT.jar, flink-runtime-web_2.10-1.0-SNAPSHOT.jar define 1690 overlapping classes: > [WARNING] - org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap > [WARNING] - org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource > [WARNING] - org.apache.flink.shaded.com.google.common.escape.Platform > [WARNING] - org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture > [WARNING] - org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator > [WARNING] - org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2 > [WARNING] - org.apache.flink.shaded.com.google.common.escape.Escaper$1 > [WARNING] - org.apache.flink.shaded.com.google.common.collect.MultimapBuilder$SetMultimapBuilder > [WARNING] - org.apache.flink.shaded.com.google.common.collect.Ordering$ArbitraryOrdering > [WARNING] - org.apache.flink.shaded.com.google.common.collect.Synchronized$SynchronizedAsMapEntries$1 > [WARNING] - 1680 more... > [WARNING] flink-scala_2.10-1.0-SNAPSHOT.jar, flink-java-1.0-SNAPSHOT.jar, flink-streaming-scala_2.10-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 25 overlapping classes: > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Context > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.FieldWriter > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.TypePath > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Handler > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.TypeReference > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.signature.SignatureVisitor > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.Frame > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.FieldVisitor > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.ByteVector > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.ClassVisitor > [WARNING] - 15 more... > [WARNING] jline-0.9.94.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 35 overlapping classes: > [WARNING] - jline.ArgumentCompletor$ArgumentList > [WARNING] - jline.UnsupportedTerminal > [WARNING] - jline.Terminal > [WARNING] - jline.WindowsTerminal$ReplayPrefixOneCharInputStream > [WARNING] - jline.History > [WARNING] - jline.WindowsTerminal$1 > [WARNING] - jline.ConsoleReader > [WARNING] - jline.ClassNameCompletor > [WARNING] - jline.SimpleCompletor$SimpleCompletorFilter > [WARNING] - jline.CandidateCycleCompletionHandler > [WARNING] - 25 more... > [WARNING] netty-3.8.0.Final.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 810 overlapping classes: > [WARNING] - org.jboss.netty.handler.codec.http.websocketx.WebSocketClientHandshaker > [WARNING] - org.jboss.netty.handler.codec.base64.Base64Decoder > [WARNING] - org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink$1 > [WARNING] - org.jboss.netty.util.VirtualExecutorService > [WARNING] - org.jboss.netty.util.DefaultObjectSizeEstimator > [WARNING] - org.jboss.netty.util.internal.ConcurrentIdentityHashMap$HashEntry > [WARNING] - org.jboss.netty.channel.socket.oio.OioDatagramChannel > [WARNING] - org.jboss.netty.logging.InternalLoggerFactory > [WARNING] - org.jboss.netty.handler.codec.spdy.DefaultSpdyDataFrame > [WARNING] - org.jboss.netty.channel.LifeCycleAwareChannelHandler > [WARNING] - 800 more... > [WARNING] flink-java-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 141 overlapping classes: > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.tree.ClassNode > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.commons.JSRInlinerAdapter$Instantiation > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.tree.analysis.BasicValue > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SingleDocElement > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.tree.TypeAnnotationNode > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.commons.CodeSizeEvaluator > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SAXWriter > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.util.TraceFieldVisitor > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$ContentHandlerFactory > [WARNING] - org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$FrameTypeRule > [WARNING] - 131 more... > [WARNING] commons-beanutils-core-1.8.0.jar, commons-beanutils-1.7.0.jar define 82 overlapping classes: > [WARNING] - org.apache.commons.beanutils.Converter > [WARNING] - org.apache.commons.beanutils.WrapDynaBean > [WARNING] - org.apache.commons.beanutils.converters.IntegerConverter > [WARNING] - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean > [WARNING] - org.apache.commons.beanutils.locale.converters.DoubleLocaleConverter > [WARNING] - org.apache.commons.beanutils.locale.converters.DecimalLocaleConverter > [WARNING] - org.apache.commons.beanutils.converters.ShortConverter > [WARNING] - org.apache.commons.beanutils.converters.StringArrayConverter > [WARNING] - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean > [WARNING] - org.apache.commons.beanutils.LazyDynaClass > [WARNING] - 72 more... > [WARNING] commons-lang3-3.3.2.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 217 overlapping classes: > [WARNING] - org.apache.commons.lang3.builder.DiffResult > [WARNING] - org.apache.commons.lang3.CharRange > [WARNING] - org.apache.commons.lang3.builder.ToStringStyle$ShortPrefixToStringStyle > [WARNING] - org.apache.commons.lang3.concurrent.ConcurrentException > [WARNING] - org.apache.commons.lang3.builder.DiffBuilder$1 > [WARNING] - org.apache.commons.lang3.builder.DiffBuilder > [WARNING] - org.apache.commons.lang3.builder.Diff > [WARNING] - org.apache.commons.lang3.time.FastDatePrinter$TwoDigitYearField > [WARNING] - org.apache.commons.lang3.ObjectUtils$Null > [WARNING] - org.apache.commons.lang3.reflect.MemberUtils > [WARNING] - 207 more... > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)