hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Paduano (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12689) S3 filesystem operations stopped working correctly
Date Mon, 04 Jan 2016 23:41:39 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15082024#comment-15082024
] 

Matthew Paduano commented on HADOOP-12689:
------------------------------------------

The attached patch reverts the one line edit by HADOOP-10542 and includes code to address
the possible NPE if a block is requested but not found in the S3 store.  This seems the easiest
way to resolve the possible NPE from 10542 and not rewrite many of the S3 file system operations
that expect null to be returned by get() when an object is missing (and/or a FileNotFoundExecption
to be raised, not IOException)

> S3 filesystem operations stopped working correctly
> --------------------------------------------------
>
>                 Key: HADOOP-12689
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12689
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 2.8.0, 3.0.0
>            Reporter: Matthew Paduano
>            Assignee: Matthew Paduano
>            Priority: Minor
>             Fix For: 2.7.1
>
>         Attachments: HADOOP-12689.01.patch
>
>
> HADOOP-10542 was resolved by replacing "return null;" with throwing  IOException.   This
causes several S3 filesystem operations to fail (possibly more code is expecting that null
return value; these are just the calls I noticed):
> S3FileSystem.getFileStatus() (which no longer raises FileNotFoundException but instead
IOException)
> FileSystem.exists() (which no longer returns false but instead raises IOException)
> S3FileSystem.create() (which no longer succeeds but instead raises IOException)
> Run command:
> hadoop distcp hdfs://localhost:9000/test s3://xxx:yyy@com.bar.foo/
> Resulting stack trace:
> 2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1449826461866_0005_m_000006_0 - exited : java.io.IOException: /test doesn't
exist
> at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170)
> at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source)
> at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230)
> at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> changing the "raise IOE..." to "return null" fixes all of the above code sites and allows
distcp to succeed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message