hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-4278) cannot run two local jobs in parallel from the same gateway.
Date Tue, 09 Oct 2012 14:21:03 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13472419#comment-13472419

Tom White commented on MAPREDUCE-4278:

You're right - it's not easy to create a unit test where the job IDs collide with the current
code. Can you run a manual test without the patch that runs two jobs and produces a collision,
and then test that with the patch there is no collision as a sanity check?

> Also, I realized that with my approach the randids could get mixed if two jobs were submitted
concurrently using the same LocalJobRunner. Is this a concern?

LocalJobRunner doesn't support running multiple jobs concurrently, so I don't think your change
makes things worse. We could add some class javadoc to clarify what it supports (i.e. use
an instance of LJR per job to run multiple jobs in a single JVM). 
> cannot run two local jobs in parallel from the same gateway.
> ------------------------------------------------------------
>                 Key: MAPREDUCE-4278
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4278
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions:
>            Reporter: Araceli Henley
>         Attachments: MAPREDUCE-4278-branch1.patch
> I cannot run two local mode jobs from Pig in parallel from the same gateway, this is
a typical use case. If I re-run the tests sequentially, then the test pass. This seems to
be a problem from Hadoop.
> Additionally, the pig harness, expects to be able to run Pig-version-undertest against
Pig-version-stable from the same gateway.
> To replicate the error:
> I have two clusters running from the same gateway.
> If I run the Pig regression suites nightly.conf in local mode in paralell - once on each
cluster. Conflicts in M/R local mode result in failures in the tests. 
> org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> output/file.out in any of the configured local directories
>         at
> org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429)
>         at
> org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160)
>         at
> org.apache.hadoop.mapred.MapOutputFile.getOutputFile(MapOutputFile.java:56)
>         at org.apache.hadoop.mapred.Task.calculateOutputSize(Task.java:944)
>         at org.apache.hadoop.mapred.Task.sendLastUpdate(Task.java:924)
>         at org.apache.hadoop.mapred.Task.done(Task.java:875)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:374)
> ---
> 2012-05-17 20:25:36,762 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> -
> HadoopJobId: job_local_0001
> 2012-05-17 20:25:36,778 [Thread-3] INFO  org.apache.hadoop.mapred.Task -
> Using ResourceCalculatorPlugin : org.apache.
> hadoop.util.LinuxResourceCalculatorPlugin@ffa490e
> 2012-05-17 20:25:36,837 [Thread-3] WARN
> org.apache.hadoop.mapred.LocalJobRunner - job_local_0001
> java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
>         at java.util.ArrayList.RangeCheck(ArrayList.java:547)
>         at java.util.ArrayList.get(ArrayList.java:322)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getLoadFunc(PigInputFormat.java
> :153)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.createRecordReader(PigInputForm
> at.java:106)
>         at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.<init>(MapTask.java:489)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>         at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
> 2012-05-17 20:25:41,291 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message