hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "shenhong (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (MAPREDUCE-5749) TestRMContainerAllocator#testReportedAppProgress Failed
Date Mon, 10 Feb 2014 05:51:19 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-5749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

shenhong updated MAPREDUCE-5749:
--------------------------------

    Description: 
When execute "mvn test -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed
with message:
{code}
Caused by: java.io.FileNotFoundException: File /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml
does not exist
{code}
But in fact, the job.xml exits:
{code}
-rw-rw-r-- 1 yuling.sh yuling.sh 65791  2月 10 13:13 /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml
{code}
See the following code:
{code}
public Job submit(Configuration conf, boolean mapSpeculative,
      boolean reduceSpeculative) throws Exception {
    String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
        .getCurrentUser().getShortUserName());
    conf.set(MRJobConfig.USER_NAME, user);
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
    conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
    // TODO: fix the bug where the speculator gets events with
    // not-fully-constructed objects. For now, disable speculative exec
    conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
    conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);

    init(conf);
    start();
    DefaultMetricsSystem.shutdown();
    Job job = getContext().getAllJobs().values().iterator().next();
    if (assignedQueue != null) {
      job.setQueueName(assignedQueue);
    }

    // Write job.xml
    String jobFile = MRApps.getJobFile(conf, user,
        TypeConverter.fromYarn(job.getID()));
    LOG.info("Writing job conf to " + jobFile);
    new File(jobFile).getParentFile().mkdirs();
    conf.writeXml(new FileOutputStream(jobFile));

    return job;
  }
{code}
At first, user is "yuling.sh",  but the UGI will be set to attemptId at "start();", after
that, job.xml write to yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job
is running, MRAppMaster can't find the job.xml at appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.

  was:
When execute "mvn test -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed
with message:
{code}
Caused by: java.io.FileNotFoundException: File /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml
does not exist
{code}
If fact, the job.xml exits in:
/home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml

See the following code:
{code}
public Job submit(Configuration conf, boolean mapSpeculative,
      boolean reduceSpeculative) throws Exception {
    String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
        .getCurrentUser().getShortUserName());
    conf.set(MRJobConfig.USER_NAME, user);
    conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
    conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
    // TODO: fix the bug where the speculator gets events with
    // not-fully-constructed objects. For now, disable speculative exec
    conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
    conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);

    init(conf);
    start();
    DefaultMetricsSystem.shutdown();
    Job job = getContext().getAllJobs().values().iterator().next();
    if (assignedQueue != null) {
      job.setQueueName(assignedQueue);
    }

    // Write job.xml
    String jobFile = MRApps.getJobFile(conf, user,
        TypeConverter.fromYarn(job.getID()));
    LOG.info("Writing job conf to " + jobFile);
    new File(jobFile).getParentFile().mkdirs();
    conf.writeXml(new FileOutputStream(jobFile));

    return job;
  }
{code}
At first, user is "yuling.sh",  but the UGI will be set to attemptId at "start();", after
that, job.xml write to yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job
is running, MRAppMaster can't find the job.xml at appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.


> TestRMContainerAllocator#testReportedAppProgress Failed
> -------------------------------------------------------
>
>                 Key: MAPREDUCE-5749
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5749
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: trunk
>            Reporter: shenhong
>
> When execute "mvn test -Dtest=TestRMContainerAllocator#testReportedAppProgress", It failed
with message:
> {code}
> Caused by: java.io.FileNotFoundException: File /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001/job.xml
does not exist
> {code}
> But in fact, the job.xml exits:
> {code}
> -rw-rw-r-- 1 yuling.sh yuling.sh 65791  2月 10 13:13 /home/yuling.sh/hadoop-common/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator/yuling.sh/.staging/job_1392009213299_0001/job.xml
> {code}
> See the following code:
> {code}
> public Job submit(Configuration conf, boolean mapSpeculative,
>       boolean reduceSpeculative) throws Exception {
>     String user = conf.get(MRJobConfig.USER_NAME, UserGroupInformation
>         .getCurrentUser().getShortUserName());
>     conf.set(MRJobConfig.USER_NAME, user);
>     conf.set(MRJobConfig.MR_AM_STAGING_DIR, testAbsPath.toString());
>     conf.setBoolean(MRJobConfig.MR_AM_CREATE_JH_INTERMEDIATE_BASE_DIR, true);
>     // TODO: fix the bug where the speculator gets events with
>     // not-fully-constructed objects. For now, disable speculative exec
>     conf.setBoolean(MRJobConfig.MAP_SPECULATIVE, mapSpeculative);
>     conf.setBoolean(MRJobConfig.REDUCE_SPECULATIVE, reduceSpeculative);
>     init(conf);
>     start();
>     DefaultMetricsSystem.shutdown();
>     Job job = getContext().getAllJobs().values().iterator().next();
>     if (assignedQueue != null) {
>       job.setQueueName(assignedQueue);
>     }
>     // Write job.xml
>     String jobFile = MRApps.getJobFile(conf, user,
>         TypeConverter.fromYarn(job.getID()));
>     LOG.info("Writing job conf to " + jobFile);
>     new File(jobFile).getParentFile().mkdirs();
>     conf.writeXml(new FileOutputStream(jobFile));
>     return job;
>   }
> {code}
> At first, user is "yuling.sh",  but the UGI will be set to attemptId at "start();", after
that, job.xml write to yuling.sh/.staging/job_1392009213299_0001/job.xml. But when the job
is running, MRAppMaster can't find the job.xml at appattempt_1392009213299_0001_000001/.staging/job_1392009213299_0001.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message