flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephan Ewen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3328) Incorrectly shaded dependencies in flink-runtime
Date Thu, 20 Apr 2017 09:35:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976371#comment-15976371
] 

Stephan Ewen commented on FLINK-3328:
-------------------------------------

To do that, one would have to explicitly code against the shaded namespaces, meaning explicitly
use {{org.apache.flink.shaded.com.google.common.collect.Lists}} rather than {{com.google.common.collect.Lists}}.
That's not ideal either.

I think the best think is to actually avoid libraries that are used in many modules need shading.
We started to do that with Guava. With ASM it will not be possible, I think.

An alternative that I could see is to have a single {{flink-shaded-dependencies}} project
in which we have the libraries shaded once, and then we develop against the shaded namespaces
directly.

> Incorrectly shaded dependencies in flink-runtime
> ------------------------------------------------
>
>                 Key: FLINK-3328
>                 URL: https://issues.apache.org/jira/browse/FLINK-3328
>             Project: Flink
>          Issue Type: Bug
>          Components: Build System
>    Affects Versions: 1.0.0
>            Reporter: Stephan Ewen
>            Assignee: Robert Metzger
>            Priority: Blocker
>             Fix For: 1.0.0
>
>
> There are apparently some dependencies shaded into {{flink-runtime}} fat jar that are
not relocated. (the flink-runtime jar is now 70 MB)
> From the output of the shading in flink-dist, it looks as if this concerns at least
>   - Zookeeper
>   - slf4j
>   - jline
>   - netty (3.x)
> Possible more.
> {code}
> [WARNING] zookeeper-3.4.6.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 440 overlapping
classes: 
> [WARNING]   - org.apache.zookeeper.server.NettyServerCnxnFactory
> [WARNING]   - org.apache.jute.compiler.JFile
> [WARNING]   - org.apache.zookeeper.server.SessionTracker$Session
> [WARNING]   - org.apache.zookeeper.server.quorum.AuthFastLeaderElection$1
> [WARNING]   - org.apache.jute.compiler.JLong
> [WARNING]   - org.apache.zookeeper.client.ZooKeeperSaslClient$SaslState
> [WARNING]   - org.apache.zookeeper.server.auth.KerberosName$Rule
> [WARNING]   - org.apache.jute.CsvOutputArchive
> [WARNING]   - org.apache.zookeeper.server.quorum.QuorumPeer
> [WARNING]   - org.apache.zookeeper.ZooKeeper$DataWatchRegistration
> [WARNING]   - 430 more...
> [WARNING] slf4j-api-1.7.7.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 24 overlapping
classes: 
> [WARNING]   - org.slf4j.spi.MarkerFactoryBinder
> [WARNING]   - org.slf4j.helpers.SubstituteLogger
> [WARNING]   - org.slf4j.helpers.BasicMarker
> [WARNING]   - org.slf4j.helpers.Util
> [WARNING]   - org.slf4j.LoggerFactory
> [WARNING]   - org.slf4j.Marker
> [WARNING]   - org.slf4j.helpers.NamedLoggerBase
> [WARNING]   - org.slf4j.Logger
> [WARNING]   - org.slf4j.spi.LocationAwareLogger
> [WARNING]   - org.slf4j.ILoggerFactory
> [WARNING]   - 14 more...
> [WARNING] jansi-1.4.jar, jline-2.10.4.jar define 23 overlapping classes: 
> [WARNING]   - org.fusesource.jansi.Ansi$Erase
> [WARNING]   - org.fusesource.jansi.Ansi
> [WARNING]   - org.fusesource.jansi.AnsiOutputStream
> [WARNING]   - org.fusesource.jansi.internal.CLibrary
> [WARNING]   - org.fusesource.jansi.Ansi$2
> [WARNING]   - org.fusesource.jansi.WindowsAnsiOutputStream
> [WARNING]   - org.fusesource.jansi.AnsiRenderer$Code
> [WARNING]   - org.fusesource.jansi.AnsiConsole
> [WARNING]   - org.fusesource.jansi.Ansi$Attribute
> [WARNING]   - org.fusesource.jansi.internal.Kernel32
> [WARNING]   - 13 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-collections-3.2.2.jar, commons-beanutils-1.7.0.jar
define 10 overlapping classes: 
> [WARNING]   - org.apache.commons.collections.FastHashMap$EntrySet
> [WARNING]   - org.apache.commons.collections.ArrayStack
> [WARNING]   - org.apache.commons.collections.FastHashMap$1
> [WARNING]   - org.apache.commons.collections.FastHashMap$KeySet
> [WARNING]   - org.apache.commons.collections.FastHashMap$CollectionView
> [WARNING]   - org.apache.commons.collections.BufferUnderflowException
> [WARNING]   - org.apache.commons.collections.Buffer
> [WARNING]   - org.apache.commons.collections.FastHashMap$CollectionView$CollectionViewIterator
> [WARNING]   - org.apache.commons.collections.FastHashMap$Values
> [WARNING]   - org.apache.commons.collections.FastHashMap
> [WARNING] flink-streaming-scala_2.10-1.0-SNAPSHOT.jar, flink-core-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar,
flink-java-1.0-SNAPSHOT.jar, flink-streaming-java_2.10-1.0-SNAPSHOT.jar, flink-scala_2.10-1.0-SNAPSHOT.jar,
flink-clients_2.10-1.0-SNAPSHOT.jar, flink-optimizer_2.10-1.0-SNAPSHOT.jar, flink-runtime-web_2.10-1.0-SNAPSHOT.jar
define 1690 overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.LinkedListMultimap
> [WARNING]   - org.apache.flink.shaded.com.google.common.io.ByteSource$AsCharSource
> [WARNING]   - org.apache.flink.shaded.com.google.common.escape.Platform
> [WARNING]   - org.apache.flink.shaded.com.google.common.util.concurrent.Futures$ImmediateFailedCheckedFuture
> [WARNING]   - org.apache.flink.shaded.com.google.common.primitives.SignedBytes$LexicographicalComparator
> [WARNING]   - org.apache.flink.shaded.com.google.common.cache.LocalCache$WriteQueue$2
> [WARNING]   - org.apache.flink.shaded.com.google.common.escape.Escaper$1
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.MultimapBuilder$SetMultimapBuilder
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.Ordering$ArbitraryOrdering
> [WARNING]   - org.apache.flink.shaded.com.google.common.collect.Synchronized$SynchronizedAsMapEntries$1
> [WARNING]   - 1680 more...
> [WARNING] flink-scala_2.10-1.0-SNAPSHOT.jar, flink-java-1.0-SNAPSHOT.jar, flink-streaming-scala_2.10-1.0-SNAPSHOT.jar,
flink-runtime_2.10-1.0-SNAPSHOT.jar define 25 overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Context
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.FieldWriter
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.TypePath
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Handler
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.TypeReference
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.signature.SignatureVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.Frame
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.FieldVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.ByteVector
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.ClassVisitor
> [WARNING]   - 15 more...
> [WARNING] jline-0.9.94.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 35 overlapping
classes: 
> [WARNING]   - jline.ArgumentCompletor$ArgumentList
> [WARNING]   - jline.UnsupportedTerminal
> [WARNING]   - jline.Terminal
> [WARNING]   - jline.WindowsTerminal$ReplayPrefixOneCharInputStream
> [WARNING]   - jline.History
> [WARNING]   - jline.WindowsTerminal$1
> [WARNING]   - jline.ConsoleReader
> [WARNING]   - jline.ClassNameCompletor
> [WARNING]   - jline.SimpleCompletor$SimpleCompletorFilter
> [WARNING]   - jline.CandidateCycleCompletionHandler
> [WARNING]   - 25 more...
> [WARNING] netty-3.8.0.Final.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 810 overlapping
classes: 
> [WARNING]   - org.jboss.netty.handler.codec.http.websocketx.WebSocketClientHandshaker
> [WARNING]   - org.jboss.netty.handler.codec.base64.Base64Decoder
> [WARNING]   - org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink$1
> [WARNING]   - org.jboss.netty.util.VirtualExecutorService
> [WARNING]   - org.jboss.netty.util.DefaultObjectSizeEstimator
> [WARNING]   - org.jboss.netty.util.internal.ConcurrentIdentityHashMap$HashEntry
> [WARNING]   - org.jboss.netty.channel.socket.oio.OioDatagramChannel
> [WARNING]   - org.jboss.netty.logging.InternalLoggerFactory
> [WARNING]   - org.jboss.netty.handler.codec.spdy.DefaultSpdyDataFrame
> [WARNING]   - org.jboss.netty.channel.LifeCycleAwareChannelHandler
> [WARNING]   - 800 more...
> [WARNING] flink-java-1.0-SNAPSHOT.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 141
overlapping classes: 
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.ClassNode
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.commons.JSRInlinerAdapter$Instantiation
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.analysis.BasicValue
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SingleDocElement
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.tree.TypeAnnotationNode
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.commons.CodeSizeEvaluator
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$SAXWriter
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.util.TraceFieldVisitor
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.Processor$ContentHandlerFactory
> [WARNING]   - org.apache.flink.shaded.org.objectweb.asm.xml.ASMContentHandler$FrameTypeRule
> [WARNING]   - 131 more...
> [WARNING] commons-beanutils-core-1.8.0.jar, commons-beanutils-1.7.0.jar define 82 overlapping
classes: 
> [WARNING]   - org.apache.commons.beanutils.Converter
> [WARNING]   - org.apache.commons.beanutils.WrapDynaBean
> [WARNING]   - org.apache.commons.beanutils.converters.IntegerConverter
> [WARNING]   - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean
> [WARNING]   - org.apache.commons.beanutils.locale.converters.DoubleLocaleConverter
> [WARNING]   - org.apache.commons.beanutils.locale.converters.DecimalLocaleConverter
> [WARNING]   - org.apache.commons.beanutils.converters.ShortConverter
> [WARNING]   - org.apache.commons.beanutils.converters.StringArrayConverter
> [WARNING]   - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean
> [WARNING]   - org.apache.commons.beanutils.LazyDynaClass
> [WARNING]   - 72 more...
> [WARNING] commons-lang3-3.3.2.jar, flink-runtime_2.10-1.0-SNAPSHOT.jar define 217 overlapping
classes: 
> [WARNING]   - org.apache.commons.lang3.builder.DiffResult
> [WARNING]   - org.apache.commons.lang3.CharRange
> [WARNING]   - org.apache.commons.lang3.builder.ToStringStyle$ShortPrefixToStringStyle
> [WARNING]   - org.apache.commons.lang3.concurrent.ConcurrentException
> [WARNING]   - org.apache.commons.lang3.builder.DiffBuilder$1
> [WARNING]   - org.apache.commons.lang3.builder.DiffBuilder
> [WARNING]   - org.apache.commons.lang3.builder.Diff
> [WARNING]   - org.apache.commons.lang3.time.FastDatePrinter$TwoDigitYearField
> [WARNING]   - org.apache.commons.lang3.ObjectUtils$Null
> [WARNING]   - org.apache.commons.lang3.reflect.MemberUtils
> [WARNING]   - 207 more...
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message