hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nitika Gupta (Created) (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-2583) Hadoop mapreduce job to process S3 logs gets hung at INFO mapred.JobClient: map 0% reduce 0%.
Date Tue, 22 Nov 2011 03:10:41 GMT
Hadoop mapreduce job to process S3 logs gets hung at INFO mapred.JobClient:  map 0% reduce
0%.
----------------------------------------------------------------------------------------------

                 Key: HDFS-2583
                 URL: https://issues.apache.org/jira/browse/HDFS-2583
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: Nitika Gupta
            Priority: Blocker


I am trying to run a mapreduce job to process the Amazon S3 logs. However, the code hangs
at INFO mapred.JobClient:  map 0% reduce 0% and does not even attempt to launch the tasks.
The sample code for the job setup is given below:
public int run(CommandLine cl) throws Exception 
{
       Configuration conf = getConf();
       String inputPath = "";
       String outputPath = "";
       try
       {
           Job job = new Job(conf, "Dummy");
           job.setNumReduceTasks(0);
           job.setMapperClass(Mapper.class);
           inputPath = cl.getOptionValue("input"); //input is an s3n path
           outputPath = cl.getOptionValue("output");
           FileInputFormat.setInputPaths(job, inputPath);
           FileOutputFormat.setOutputPath(job, new Path(outputPath));
           _log.info("Input path set as " + inputPath);
           _log.info("Output path set as " + outputPath);
           job.waitForCompletion(true);
           return 0;
       } catch (Exception ex)
       {
           _log.error(ex);
           return 1;
       }
}
The above code works on the staging machine. However, it fails on the production machine which
is same as the staging machine with more capacity.

Does anyone know what could be the possible reason for the error? 

Thanks in advance!

Nitika

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message