spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Or (JIRA)" <j...@apache.org>
Subject [jira] [Closed] (SPARK-4078) New FsPermission instance w/o FsPermission.createImmutable in eventlog
Date Thu, 30 Oct 2014 22:54:34 GMT

     [ https://issues.apache.org/jira/browse/SPARK-4078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Or closed SPARK-4078.
----------------------------
    Resolution: Fixed

> New FsPermission instance w/o FsPermission.createImmutable in eventlog
> ----------------------------------------------------------------------
>
>                 Key: SPARK-4078
>                 URL: https://issues.apache.org/jira/browse/SPARK-4078
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.1.0
>            Reporter: Jie Huang
>            Assignee: Jie Huang
>             Fix For: 1.2.0
>
>
> By default, Spark builds its package against Hadoop 1.0.4 version. In that version, it
has some FsPermission bug (see HADOOP-7629 by Todd Lipcon). This bug got fixed since 1.1 version.
By using that FsPermission.createImmutable() API, end-user may see some RPC exception like
below (if turn on eventlog over HDFS). 
> {quote}
> Exception in thread "main" java.io.IOException: Call to sr484/10.1.2.84:54310 failed
on local exception: java.io.EOFException
>         at org.apache.hadoop.ipc.Client.wrapException(Client.java:1150)
>         at org.apache.hadoop.ipc.Client.call(Client.java:1118)
>         at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
>         at $Proxy6.setPermission(Unknown Source)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85)
>         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62)
>         at $Proxy6.setPermission(Unknown Source)
>         at org.apache.hadoop.hdfs.DFSClient.setPermission(DFSClient.java:1285)
>         at org.apache.hadoop.hdfs.DistributedFileSystem.setPermission(DistributedFileSystem.java:572)
>         at org.apache.spark.util.FileLogger.createLogDir(FileLogger.scala:138)
>         at org.apache.spark.util.FileLogger.start(FileLogger.scala:115)
>         at org.apache.spark.scheduler.EventLoggingListener.start(EventLoggingListener.scala:74)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:324)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message