hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Koji Noguchi (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6766) Spill can fail with bad call to Random
Date Sun, 16 May 2010 04:16:42 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12867950#action_12867950
] 

Koji Noguchi commented on HADOOP-6766:
--------------------------------------

Peter, if you're using yahoo clusters it could be ops accidentally deleting the local dirs
after TaskTracker came up. 

> Spill can fail with bad call to Random
> --------------------------------------
>
>                 Key: HADOOP-6766
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6766
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs
>    Affects Versions: 0.20.2
>            Reporter: Peter Arthur Ciccolo
>            Priority: Minor
>
> java.lang.IllegalArgumentException: n must be positive
>         at java.util.Random.nextInt(Random.java:250)
>         at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.confChanged(LocalDirAllocator.java:243)
>         at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathForWrite(LocalDirAllocator.java:289)
>         at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathForWrite(LocalDirAllocator.java:124)
>         at org.apache.hadoop.mapred.MapOutputFile.getSpillFileForWrite(MapOutputFile.java:107)
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1221)
>         at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1129)
>         at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:549)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:623)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
>         at org.apache.hadoop.mapred.Child.main(Child.java:159)
> confChanged assumes that the list of dirs it creates (LocalDirAllocator.java:215) has
at least one element in it by the end of the function. If, for each local dir, either the
conditional on line 221 is false, or the call to DiskChecker.checkDir() throws an exception,
this assumption will not hold. In this case, dirIndexRandomizer.nextInt() is called on the
number of elements in dirs, which is 0. Since dirIndexRandomizer (195) is an instance of Random(),
it needs a positive (non-zero) argument to nextInt().

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message