hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-1675) Null pointer exception in InMemoryFileSystem
Date Fri, 03 Aug 2007 18:39:52 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12517601

Devaraj Das commented on HADOOP-1675:

This can happen if the InMemoryFileSystem instance is "closed". The InMemoryFileSystem instance
is closed prematurely if the merge of the files in the InMemoryFileSystem throws an exception.
When close() is invoked on the instance of the InMemoryFileSystem, it will esentially clear
out the mapping of the corresponding URI from the FileSystem's CACHE map, thereby invalidating
the reference to the object. However, some of the copier threads might still be active and
they might get NPEs if they try to invoke InMemoryFileSystem operations on that invalid object.
In essence, although the code could be tweaked to handle this exception, it is harmless, if
there was some other fatal error before (like in-memory merge throwing an exception) this
exception is thrown since the task is anyway going to die. Could you please confirm whether
you see some other fatal exception before this NPE?

> Null pointer exception in InMemoryFileSystem
> --------------------------------------------
>                 Key: HADOOP-1675
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1675
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.13.0
>            Reporter: Runping Qi
> In a job running release .13, I saw a lot of failures due to the following errors:
> 2007-08-03 16:13:39,516 ERROR org.apache.hadoop.mapred.ReduceTask: Map output copy failure:
> 	at org.apache.hadoop.fs.InMemoryFileSystem$RawInMemoryFileSystem.reserveSpace(InMemoryFileSystem.java:361)
> 	at org.apache.hadoop.fs.InMemoryFileSystem.reserveSpaceWithCheckSum(InMemoryFileSystem.java:474)
> 	at org.apache.hadoop.mapred.MapOutputLocation.getFile(MapOutputLocation.java:220)
> 	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput(ReduceTask.java:680)
> 	at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:641)

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message