hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3592) DistCp should not setTimes on directories
Date Wed, 21 Dec 2011 19:05:33 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13174316#comment-13174316
] 

Konstantin Shvachko commented on MAPREDUCE-3592:
------------------------------------------------

This happened in -update mode, when the source and destination turned out to be in sync.
I think DistCp does not need to update anything at all. But it does and gets this exception.
The exception is confusing, but it means setTimes() is not supported for directories.
{code}
11/12/20 15:55:49 INFO tools.DistCp: sourcePathsCount(files+directories)=11602
11/12/20 15:55:49 INFO tools.DistCp: filesToCopyCount=0
11/12/20 15:55:49 INFO tools.DistCp: bytesToCopyCount=0
11/12/20 15:55:49 INFO tools.DistCp: filesToSkipCopyCount=5676
11/12/20 15:55:49 INFO tools.DistCp: bytesToSkipCopyCount=5339.3g
11/12/20 15:56:01 INFO tools.DistCp: deletedPathsFromDestCount(files+directories)=0
11/12/20 15:56:01 INFO tools.DistCp: sourcePathsCount=11602
11/12/20 15:56:01 INFO tools.DistCp: filesToCopyCount=0
11/12/20 15:56:01 INFO tools.DistCp: bytesToCopyCount=0
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.io.FileNotFoundException: File /my-backup/my-data-input does not exist.
     at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimes(FSNamesystem.java:1096)
     at org.apache.hadoop.hdfs.server.namenode.NameNode.setTimes(NameNode.java:1269)
     at sun.reflect.GeneratedMethodAccessor34.invoke(Unknown Source)
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
     at java.lang.reflect.Method.invoke(Method.java:597)
     at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:349)
     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1482)
     at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1478)
     at java.security.AccessController.doPrivileged(Native Method)
     at javax.security.auth.Subject.doAs(Subject.java:396)
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1153)
     at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1476)
 
     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
     at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90)
     at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57)
     at org.apache.hadoop.hdfs.DFSClient.setTimes(DFSClient.java:1347)
     at org.apache.hadoop.hdfs.DistributedFileSystem.setTimes(DistributedFileSystem.java:777)
     at org.apache.hadoop.tools.DistCp.updateDestStatus(DistCp.java:827)
     at org.apache.hadoop.tools.DistCp.finalize(DistCp.java:853)
     at org.apache.hadoop.tools.DistCp.copy(DistCp.java:794)
     at org.apache.hadoop.tools.DistCp.run(DistCp.java:1028)
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
     at org.apache.hadoop.tools.DistCp.main(DistCp.java:1055)
{code}
                
> DistCp should not setTimes on directories
> -----------------------------------------
>
>                 Key: MAPREDUCE-3592
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3592
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: distcp
>    Affects Versions: 0.22.0
>            Reporter: Konstantin Shvachko
>
> On finalize() stage DistCp attempts to updateDestStatus(), which includes calling setTimes().
In HDFS setTimes() is not supported for directories, therefore DistCp fails if there are directories
in the tree.
> It looks the failure occurs only when there are no files to copy.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message