flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7973) Fix service shading relocation for S3 file systems
Date Wed, 08 Nov 2017 16:52:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16244296#comment-16244296
] 

ASF GitHub Bot commented on FLINK-7973:
---------------------------------------

Github user aljoscha commented on the issue:

    https://github.com/apache/flink/pull/4961
  
    I added two end-to-end tests and it seems for presto this currently fails with:
    ```
    ------------------------------------------------------------
     The program finished with the following exception:
    
    org.apache.flink.client.program.ProgramInvocationException: The program execution failed:
Failed to submit job ae34d825b54b7afe2e973627d396e722 (WordCount Example)
    	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:492)
    	at org.apache.flink.client.program.StandaloneClusterClient.submitJob(StandaloneClusterClient.java:105)
    	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:456)
    	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:444)
    	at org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:62)
    	at org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:86)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:525)
    	at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:417)
    	at org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:396)
    	at org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:802)
    	at org.apache.flink.client.CliFrontend.run(CliFrontend.java:282)
    	at org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:1054)
    	at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1101)
    	at org.apache.flink.client.CliFrontend$1.call(CliFrontend.java:1098)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at javax.security.auth.Subject.doAs(Subject.java:422)
    	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
    	at org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
    	at org.apache.flink.client.CliFrontend.main(CliFrontend.java:1098)
    Caused by: org.apache.flink.runtime.client.JobExecutionException: Failed to submit job
ae34d825b54b7afe2e973627d396e722 (WordCount Example)
    	at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1325)
    	at org.apache.flink.runtime.jobmanager.JobManager$$anonfun$handleMessage$1.applyOrElse(JobManager.scala:447)
    	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    	at org.apache.flink.runtime.LeaderSessionMessageFilter$$anonfun$receive$1.applyOrElse(LeaderSessionMessageFilter.scala:38)
    	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36)
    	at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:33)
    	at org.apache.flink.runtime.LogMessages$$anon$1.apply(LogMessages.scala:28)
    	at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
    	at org.apache.flink.runtime.LogMessages$$anon$1.applyOrElse(LogMessages.scala:28)
    	at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
    	at org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:122)
    	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
    	at akka.actor.ActorCell.invoke(ActorCell.scala:495)
    	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
    	at akka.dispatch.Mailbox.run(Mailbox.scala:224)
    	at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
    	at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    	at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    	at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    	at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
    Caused by: org.apache.flink.runtime.JobException: Creating the input splits caused an
error: null
    	at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:262)
    	at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:801)
    	at org.apache.flink.runtime.executiongraph.ExecutionGraphBuilder.buildGraph(ExecutionGraphBuilder.java:180)
    	at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1277)
    	... 19 more
    Caused by: java.lang.ExceptionInInitializerError
    	at org.apache.flink.fs.s3presto.shaded.com.amazonaws.ClientConfiguration.<clinit>(ClientConfiguration.java:65)
    	at org.apache.flink.fs.s3presto.shaded.com.facebook.presto.hive.PrestoS3FileSystem.initialize(PrestoS3FileSystem.java:203)
    	at org.apache.flink.fs.s3presto.S3FileSystemFactory.create(S3FileSystemFactory.java:133)
    	at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:377)
    	at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:307)
    	at org.apache.flink.core.fs.Path.getFileSystem(Path.java:293)
    	at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:472)
    	at org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:62)
    	at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:248)
    	... 22 more
    Caused by: org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.LogConfigurationException:
java.lang.ClassNotFoundException: org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.impl.LogFactoryImpl
(Caused by java.lang.ClassNotFoundException: org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.impl.LogFactoryImpl)
    	at org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.LogFactory.createFactory(LogFactory.java:1201)
    	at org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.LogFactory$2.run(LogFactory.java:1003)
    	at java.security.AccessController.doPrivileged(Native Method)
    	at org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.LogFactory.newFactory(LogFactory.java:1000)
    	at org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.LogFactory.getFactory(LogFactory.java:626)
    	at org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.LogFactory.getLog(LogFactory.java:657)
    	at org.apache.flink.fs.s3presto.shaded.com.amazonaws.util.VersionInfoUtils.<clinit>(VersionInfoUtils.java:47)
    	... 31 more
    Caused by: java.lang.ClassNotFoundException: org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.impl.LogFactoryImpl
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	at org.apache.flink.fs.s3presto.shaded.org.apache.commons.logging.LogFactory.createFactory(LogFactory.java:1063)
    	... 37 more
    ```
    
    This is my branch with the end-to-end tests: https://github.com/aljoscha/flink/tree/finish-pr-4961-s3-shading-fixing


> Fix service shading relocation for S3 file systems
> --------------------------------------------------
>
>                 Key: FLINK-7973
>                 URL: https://issues.apache.org/jira/browse/FLINK-7973
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Stephan Ewen
>            Assignee: Nico Kruber
>            Priority: Blocker
>             Fix For: 1.4.0
>
>
> The shade plugin relocates services incorrectly currently, applying relocation patterns
multiple times.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message