flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Hutchison (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
Date Wed, 19 Apr 2017 18:42:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975268#comment-15975268
] 

Luke Hutchison commented on FLINK-3328:
---------------------------------------

[~StephanEwen] It's only not a problem if you know that in every case, the version of every
single shared library will be the same between all Flink jars. And even if you know that will
be the case, it's hardly ideal space-wise, because you end up pulling in the same dependencies
multiple times for every Flink project, until a fat jar is built. In general, there is a good
reason for that Maven warning, because if there's any way that multiple different versions
of a dep can occur, you can trigger bugs that are maddeningly hard to trace.

If every Flink jar (like `flink-runtime`) depends on one of the core Flink jars (e.g. `flink-core`),
couldn't all the shared dependencies just be put in that one jar?


> Incorrectly shaded dependencies in flink-runtime
> ------------------------------------------------
>
>                 Key: FLINK-3328
>                 URL: https://issues.apache.org/jira/browse/FLINK-3328
>             Project: Flink
>          Issue Type: Bug
>          Components: Build System
>    Affects Versions: 1.0.0
>            Reporter: Stephan Ewen
>            Assignee: Robert Metzger
>            Priority: Blocker
>             Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar that are
not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at least
>   - Zookeeper
>   - slf4j
>   - jline
>   - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440 overlapping
classes: 
> [WARNING]   - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING]   - org.apache.jute.compiler.JFile
> [WARNING]   - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING]   - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING]   - org.apache.jute.compiler.JLong
> [WARNING]   - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING]   - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING]   - org.apache.jute.CsvOutputArchive
> [WARNING]   - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING]   - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING]   - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24 overlapping
classes: 
> [WARNING]   - org.slf4j.spi.MarkerFactoryBinder
> [WARNING]   - org.slf4j.helpers.SubstituteLogger
> [WARNING]   - org.slf4j.helpers.BasicMarker
> [WARNING]   - org.slf4j.helpers.Util
> [WARNING]   - org.slf4j.LoggerFactory
> [WARNING]   - org.slf4j.Marker
> [WARNING]   - org.slf4j.helpers.NamedLoggerBase
> [WARNING]   - org.slf4j.Logger
> [WARNING]   - org.slf4j.spi.LocationAwareLogger
> [WARNING]   - org.slf4j.ILoggerFactory
> [WARNING]   - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes: 
> [WARNING]   - org.fusesource.jansi.Ansi$Erase
> [WARNING]   - org.fusesource.jansi.Ansi
> [WARNING]   - org.fusesource.jansi.AnsiOutputStream
> [WARNING]   - org.fusesource.jansi.internal.CLibrary
> [WARNING]   - org.fusesource.jansi.Ansi$2
> [WARNING]   - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING]   - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING]   - org.fusesource.jansi.AnsiConsole
> [WARNING]   - org.fusesource.jansi.Ansi$Attribute
> [WARNING]   - org.fusesource.jansi.internal.Kernel32
> [WARNING]   - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar, commons-beanutils-1.7.0.jar
define 10 overlapping classes: 
> [WARNING]   - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING]   - org.apache.commons.collections.ArrayStack
> [WARNING]   - org.apache.commons.collections.FastHashMap$1
> [WARNING]   - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING]   - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING]   - org.apache.commons.collections.BufferUnderflowException
> [WARNING]   - org.apache.commons.collections.Buffer
> [WARNING]   - org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING]   - org.apache.commons.collections.FastHashMap$Values
> [WARNING]   - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar, flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar, flink-scala_2.10-1.0-SNAPSHOT.jar,
flink-clients_2.10-1.0-SNAPSHOT.jar, flink-optimizer_2.10-1.0-SNAPSHOT.jar, flink-runtime-web_2.10-1.0-SNAPSHOT.jar
define 1690 overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING]   - org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING]   - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING]   - org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING]   - org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING]   - org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING]   - org.apache.flink.shaded.com.google.common.escape.Escaper$1
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.MultimapBuilder$SetMultimapBuilder
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.Ordering$ArbitraryOrdering
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.Synchronized$SynchronizedAsMapEntries$1
> [WARNING]   - 1680 more...
> [WARNING] flink-scala_2.10-1.0-SNAPSHOT.jar, flink-java-1.0-SNAPSHOT.jar, flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
flink-runtime_2.10-1.0-SNAPSHOT.jar define 25 overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Context
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.FieldWriter
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.TypePath
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Handler
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.TypeReference
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.signature.SignatureVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Frame
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.FieldVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.ByteVector
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.ClassVisitor
> [WARNING]   - 15 more...
> [WARNING] jline-0.9.94.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 35 overlapping
classes: 
> [WARNING]   - jline.ArgumentCompletor$ArgumentList
> [WARNING]   - jline.UnsupportedTerminal
> [WARNING]   - jline.Terminal
> [WARNING]   - jline.WindowsTerminal$ReplayPrefixOneCharInputStream
> [WARNING]   - jline.History
> [WARNING]   - jline.WindowsTerminal$1
> [WARNING]   - jline.ConsoleReader
> [WARNING]   - jline.ClassNameCompletor
> [WARNING]   - jline.SimpleCompletor$SimpleCompletorFilter
> [WARNING]   - jline.CandidateCycleCompletionHandler
> [WARNING]   - 25 more...
> [WARNING] netty-3.8.0.Final.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 810 overlapping
classes: 
> [WARNING]   - org.jboss.netty.handler.codec.http.websocketx.WebSocketClientHandshaker
> [WARNING]   - org.jboss.netty.handler.codec.base64.Base64Decoder
> [WARNING]   - org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink$1
> [WARNING]   - org.jboss.netty.util.VirtualExecutorService
> [WARNING]   - org.jboss.netty.util.DefaultObjectSizeEstimator
> [WARNING]   - org.jboss.netty.util.internal.ConcurrentIdentityHashMap$HashEntry
> [WARNING]   - org.jboss.netty.channel.socket.oio.OioDatagramChannel
> [WARNING]   - org.jboss.netty.logging.InternalLoggerFactory
> [WARNING]   - org.jboss.netty.handler.codec.spdy.DefaultSpdyDataFrame
> [WARNING]   - org.jboss.netty.channel.LifeCycleAwareChannelHandler
> [WARNING]   - 800 more...
> [WARNING] flink-java-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 141
overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.ClassNode
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.commons.JSRInlinerAdapter$Instantiation
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.analysis.BasicValue
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SingleDocElement
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.TypeAnnotationNode
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.commons.CodeSizeEvaluator
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SAXWriter
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.util.TraceFieldVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$ContentHandlerFactory
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$FrameTypeRule
> [WARNING]   - 131 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-beanutils-1.7.0.jar define 82 overlapping
classes: 
> [WARNING]   - org.apache.commons.beanutils.Converter
> [WARNING]   - org.apache.commons.beanutils.WrapDynaBean
> [WARNING]   - org.apache.commons.beanutils.converters.IntegerConverter
> [WARNING]   - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean
> [WARNING]   - org.apache.commons.beanutils.locale.converters.DoubleLocaleConverter
> [WARNING]   - org.apache.commons.beanutils.locale.converters.DecimalLocaleConverter
> [WARNING]   - org.apache.commons.beanutils.converters.ShortConverter
> [WARNING]   - org.apache.commons.beanutils.converters.StringArrayConverter
> [WARNING]   - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean
> [WARNING]   - org.apache.commons.beanutils.LazyDynaClass
> [WARNING]   - 72 more...
> [WARNING] commons-lang3-3.3.2.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 217 overlapping
classes: 
> [WARNING]   - org.apache.commons.lang3.builder.DiffResult
> [WARNING]   - org.apache.commons.lang3.CharRange
> [WARNING]   - org.apache.commons.lang3.builder.ToStringStyle$ShortPrefixToStringStyle
> [WARNING]   - org.apache.commons.lang3.concurrent.ConcurrentException
> [WARNING]   - org.apache.commons.lang3.builder.DiffBuilder$1
> [WARNING]   - org.apache.commons.lang3.builder.DiffBuilder
> [WARNING]   - org.apache.commons.lang3.builder.Diff
> [WARNING]   - org.apache.commons.lang3.time.FastDatePrinter$TwoDigitYearField
> [WARNING]   - org.apache.commons.lang3.ObjectUtils$Null
> [WARNING]   - org.apache.commons.lang3.reflect.MemberUtils
> [WARNING]   - 207 more...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message