flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
Date Wed, 19 Apr 2017 18:22:41 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15975227#comment-15975227
] 

Stephan Ewen commented on FLINK-3328:
-------------------------------------

That is not a problem. Guava and asm must be in each artifact ({{flink-core}}, {{flink-runtime}},
etc) to make them self contained.
When assembling the final jar, they all have guava, so you have an overlap of the exact same
classes.

The alternative is to relocate it to a different namespace in each artifact, but then we get
guava 10 times into the {{flink-dist}}, and that is not very desirable.

> Incorrectly shaded dependencies in flink-runtime
> ------------------------------------------------
>
>                 Key: FLINK-3328
>                 URL: https://issues.apache.org/jira/browse/FLINK-3328
>             Project: Flink
>          Issue Type: Bug
>          Components: Build System
>    Affects Versions: 1.0.0
>            Reporter: Stephan Ewen
>            Assignee: Robert Metzger
>            Priority: Blocker
>             Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar that are
not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at least
>   - Zookeeper
>   - slf4j
>   - jline
>   - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440 overlapping
classes: 
> [WARNING]   - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING]   - org.apache.jute.compiler.JFile
> [WARNING]   - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING]   - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING]   - org.apache.jute.compiler.JLong
> [WARNING]   - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING]   - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING]   - org.apache.jute.CsvOutputArchive
> [WARNING]   - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING]   - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING]   - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24 overlapping
classes: 
> [WARNING]   - org.slf4j.spi.MarkerFactoryBinder
> [WARNING]   - org.slf4j.helpers.SubstituteLogger
> [WARNING]   - org.slf4j.helpers.BasicMarker
> [WARNING]   - org.slf4j.helpers.Util
> [WARNING]   - org.slf4j.LoggerFactory
> [WARNING]   - org.slf4j.Marker
> [WARNING]   - org.slf4j.helpers.NamedLoggerBase
> [WARNING]   - org.slf4j.Logger
> [WARNING]   - org.slf4j.spi.LocationAwareLogger
> [WARNING]   - org.slf4j.ILoggerFactory
> [WARNING]   - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes: 
> [WARNING]   - org.fusesource.jansi.Ansi$Erase
> [WARNING]   - org.fusesource.jansi.Ansi
> [WARNING]   - org.fusesource.jansi.AnsiOutputStream
> [WARNING]   - org.fusesource.jansi.internal.CLibrary
> [WARNING]   - org.fusesource.jansi.Ansi$2
> [WARNING]   - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING]   - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING]   - org.fusesource.jansi.AnsiConsole
> [WARNING]   - org.fusesource.jansi.Ansi$Attribute
> [WARNING]   - org.fusesource.jansi.internal.Kernel32
> [WARNING]   - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar, commons-beanutils-1.7.0.jar
define 10 overlapping classes: 
> [WARNING]   - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING]   - org.apache.commons.collections.ArrayStack
> [WARNING]   - org.apache.commons.collections.FastHashMap$1
> [WARNING]   - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING]   - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING]   - org.apache.commons.collections.BufferUnderflowException
> [WARNING]   - org.apache.commons.collections.Buffer
> [WARNING]   - org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING]   - org.apache.commons.collections.FastHashMap$Values
> [WARNING]   - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar, flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar, flink-scala_2.10-1.0-SNAPSHOT.jar,
flink-clients_2.10-1.0-SNAPSHOT.jar, flink-optimizer_2.10-1.0-SNAPSHOT.jar, flink-runtime-web_2.10-1.0-SNAPSHOT.jar
define 1690 overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING]   - org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING]   - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING]   - org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING]   - org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING]   - org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING]   - org.apache.flink.shaded.com.google.common.escape.Escaper$1
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.MultimapBuilder$SetMultimapBuilder
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.Ordering$ArbitraryOrdering
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.Synchronized$SynchronizedAsMapEntries$1
> [WARNING]   - 1680 more...
> [WARNING] flink-scala_2.10-1.0-SNAPSHOT.jar, flink-java-1.0-SNAPSHOT.jar, flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
flink-runtime_2.10-1.0-SNAPSHOT.jar define 25 overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Context
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.FieldWriter
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.TypePath
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Handler
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.TypeReference
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.signature.SignatureVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Frame
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.FieldVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.ByteVector
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.ClassVisitor
> [WARNING]   - 15 more...
> [WARNING] jline-0.9.94.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 35 overlapping
classes: 
> [WARNING]   - jline.ArgumentCompletor$ArgumentList
> [WARNING]   - jline.UnsupportedTerminal
> [WARNING]   - jline.Terminal
> [WARNING]   - jline.WindowsTerminal$ReplayPrefixOneCharInputStream
> [WARNING]   - jline.History
> [WARNING]   - jline.WindowsTerminal$1
> [WARNING]   - jline.ConsoleReader
> [WARNING]   - jline.ClassNameCompletor
> [WARNING]   - jline.SimpleCompletor$SimpleCompletorFilter
> [WARNING]   - jline.CandidateCycleCompletionHandler
> [WARNING]   - 25 more...
> [WARNING] netty-3.8.0.Final.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 810 overlapping
classes: 
> [WARNING]   - org.jboss.netty.handler.codec.http.websocketx.WebSocketClientHandshaker
> [WARNING]   - org.jboss.netty.handler.codec.base64.Base64Decoder
> [WARNING]   - org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink$1
> [WARNING]   - org.jboss.netty.util.VirtualExecutorService
> [WARNING]   - org.jboss.netty.util.DefaultObjectSizeEstimator
> [WARNING]   - org.jboss.netty.util.internal.ConcurrentIdentityHashMap$HashEntry
> [WARNING]   - org.jboss.netty.channel.socket.oio.OioDatagramChannel
> [WARNING]   - org.jboss.netty.logging.InternalLoggerFactory
> [WARNING]   - org.jboss.netty.handler.codec.spdy.DefaultSpdyDataFrame
> [WARNING]   - org.jboss.netty.channel.LifeCycleAwareChannelHandler
> [WARNING]   - 800 more...
> [WARNING] flink-java-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 141
overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.ClassNode
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.commons.JSRInlinerAdapter$Instantiation
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.analysis.BasicValue
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SingleDocElement
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.TypeAnnotationNode
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.commons.CodeSizeEvaluator
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SAXWriter
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.util.TraceFieldVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$ContentHandlerFactory
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$FrameTypeRule
> [WARNING]   - 131 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-beanutils-1.7.0.jar define 82 overlapping
classes: 
> [WARNING]   - org.apache.commons.beanutils.Converter
> [WARNING]   - org.apache.commons.beanutils.WrapDynaBean
> [WARNING]   - org.apache.commons.beanutils.converters.IntegerConverter
> [WARNING]   - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean
> [WARNING]   - org.apache.commons.beanutils.locale.converters.DoubleLocaleConverter
> [WARNING]   - org.apache.commons.beanutils.locale.converters.DecimalLocaleConverter
> [WARNING]   - org.apache.commons.beanutils.converters.ShortConverter
> [WARNING]   - org.apache.commons.beanutils.converters.StringArrayConverter
> [WARNING]   - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean
> [WARNING]   - org.apache.commons.beanutils.LazyDynaClass
> [WARNING]   - 72 more...
> [WARNING] commons-lang3-3.3.2.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 217 overlapping
classes: 
> [WARNING]   - org.apache.commons.lang3.builder.DiffResult
> [WARNING]   - org.apache.commons.lang3.CharRange
> [WARNING]   - org.apache.commons.lang3.builder.ToStringStyle$ShortPrefixToStringStyle
> [WARNING]   - org.apache.commons.lang3.concurrent.ConcurrentException
> [WARNING]   - org.apache.commons.lang3.builder.DiffBuilder$1
> [WARNING]   - org.apache.commons.lang3.builder.DiffBuilder
> [WARNING]   - org.apache.commons.lang3.builder.Diff
> [WARNING]   - org.apache.commons.lang3.time.FastDatePrinter$TwoDigitYearField
> [WARNING]   - org.apache.commons.lang3.ObjectUtils$Null
> [WARNING]   - org.apache.commons.lang3.reflect.MemberUtils
> [WARNING]   - 207 more...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message