Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Tue, 29 Dec 2015 01:57:49 +0000 (UTC)
From: "Matthew Paduano (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12710793.1398612450000.19355.1451354269872@Atlassian.JIRA>
In-Reply-To: <JIRA.12710793.1398612450000@Atlassian.JIRA>
References: <JIRA.12710793.1398612450000@Atlassian.JIRA>
 <JIRA.12710793.1398612450858@arcas>
Subject: [jira] [Commented] (HADOOP-10542) Potential null pointer
 dereference in Jets3tFileSystemStore#retrieveBlock()
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-10542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15073363#comment-15073363 ] 

Matthew Paduano commented on HADOOP-10542:
------------------------------------------

This change seems to break some things.  In particular, have a closer look at:

S3FileSystem.getFileStatus()   (which no longer raises FileNotFoundException but instead IOException)
FileSystem.exists()                   (which no longer returns false but instead raises IOException)
S3FileSystem.create()              (which no longer succeeds but instead raises IOException)

While the javadoc suggests that the API permits one to raise IOException, most of the code I have
encountered while debugging a command like "hadoop distcp hdfs://localhost:9000/test s3://xxx:yyy@com.bar.foo/"
seems to expect (1) retrieveINode() to return null and (2) FileNotFoundException to be raised when a file is not 
found (i.e. when get() fails!).

2015-12-11 10:04:34,030 FATAL [IPC Server handler 6 on 44861] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task: attempt_1449826461866_0005_m_000006_0 - exited : java.io.IOException: /test doesn't exist
	at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.get(Jets3tFileSystemStore.java:170)
	at org.apache.hadoop.fs.s3.Jets3tFileSystemStore.retrieveINode(Jets3tFileSystemStore.java:221)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy17.retrieveINode(Unknown Source)
	at org.apache.hadoop.fs.s3.S3FileSystem.getFileStatus(S3FileSystem.java:340)
	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:230)
	at org.apache.hadoop.tools.mapred.CopyMapper.map(CopyMapper.java:50)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

changing the "raise IOE..." to "return null" fixes all of the above code sites and 
allows distcp to succeed.


> Potential null pointer dereference in Jets3tFileSystemStore#retrieveBlock()
> ---------------------------------------------------------------------------
>
>                 Key: HADOOP-10542
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10542
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.6.0
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Minor
>             Fix For: 2.7.0
>
>         Attachments: hadoop-10542-001.patch
>
>
> {code}
>       in = get(blockToKey(block), byteRangeStart);
>       out = new BufferedOutputStream(new FileOutputStream(fileBlock));
>       byte[] buf = new byte[bufferSize];
>       int numRead;
>       while ((numRead = in.read(buf)) >= 0) {
> {code}
> get() may return null.
> The while loop dereferences in without null check.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)