hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6320) Configuration of retrieved Job via Cluster is not properly set-up
Date Sun, 19 Apr 2015 17:54:59 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502030#comment-14502030

Hadoop QA commented on MAPREDUCE-6320:

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  against trunk revision 5459b24.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:red}-1 javac{color:red}.  The patch appears to cause the build to fail.

Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5411//console

This message is automatically generated.

> Configuration of retrieved Job via Cluster is not properly set-up
> -----------------------------------------------------------------
>                 Key: MAPREDUCE-6320
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6320
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Jens Rabe
>         Attachments: MAPREDUCE-6320.001.patch
> When getting a Job via the Cluster API, it is not correctly configured.
> To reproduce this:
> # Submit a MR job, and set some arbitrary parameter to its configuration
> {code:java}
> job.getConfiguration().set("foo", "bar");
> job.setJobName("foo-bug-demo");
> {code}
> # Get the job in a client:
> {code:java}
> final Cluster c = new Cluster(conf);
> final JobStatus[] statuses = c.getAllJobStatuses();
> final JobStatus s = ... // get the status for the job named foo-bug-demo
> final Job j = c.getJob(s.getJobId());
> final Configuration conf = job.getConfiguration();
> {code}
> # Get its "foo" entry
> {code:java}
> final String s = conf.get("foo");
> {code}
> # Expected: s is "bar"; But: s is null.
> The reason is that the job's configuration is stored on HDFS (the Configuration has a
resource with a *hdfs://* URL) and in the *loadResource* it is changed to a path on the local
file system (hdfs://host.domain:port/tmp/hadoop-yarn/... is changed to /tmp/hadoop-yarn/...),
which does not exist, and thus the configuration is not populated.
> The bug happens in the *Cluster* class, where *JobConfs* are created from *status.getJobFile()*.
A quick fix would be to copy this job file to a temporary file in the local file system and
populate the JobConf from this file.

This message was sent by Atlassian JIRA

View raw message