hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unmesha sreeveni <unmeshab...@gmail.com>
Subject Re: Counters in MapReduce
Date Thu, 12 Jun 2014 08:31:23 GMT
yes rectified that error
But after 1 st iteration when it enters to second iteration
showing

 java.io.FileNotFoundException: for  *Path out1 = new Path(CL);*
*Why is it so .*
*Normally that should be in this way only the o/p folder should not exist*

* //other configuration*
*     job1.setMapperClass(ID3ClsLabelMapper.class);*
*     job1.setReducerClass(ID3ClsLabelReducer.class);*
*     Path in = new Path(args[0]);*
*     Path out1 = new Path(CL);*
*    //delete the file if exist*
*     if(counter == 0){*
*            FileInputFormat.addInputPath(job1, in);*
*     }*
*     else{*
*            FileInputFormat.addInputPath(job1, out5);   *
*     }*
    *  FileOutputFormat.setOutputPath(job1,out1);*
*     job1.waitForCompletion(true);*


On Thu, Jun 12, 2014 at 10:29 AM, unmesha sreeveni <unmeshabiju@gmail.com>
wrote:

> I tried out by setting an enum to count no. of lines in output file from
> job3.
>
> But I am getting
> 14/06/12 10:12:30 INFO mapred.JobClient:     Total committed heap usage
> (bytes)=1238630400
> conf3
> Exception in thread "main" java.lang.IllegalStateException: Job in state
> DEFINE instead of RUNNING
> at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:116)
>  at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:491)
>
>
> Below is my current code
>
> *static enum UpdateCounter {*
> *        INCOMING_ATTR*
> *    }*
>
> *public static void main(String[] args) throws Exception {*
> *    Configuration conf = new Configuration();*
> *    int res = ToolRunner.run(conf, new Driver(), args);*
> *    System.exit(res);*
> *}*
>
>
> *@Override*
> *public int run(String[] args) throws Exception {*
> *while(counter >= 0){*
>
> *      Configuration conf = getConf();*
> *     /**
> *     * Job 1: *
> *     */*
> *     Job job1 = new Job(conf, "");*
> *     //other configuration*
> *     job1.setMapperClass(ID3ClsLabelMapper.class);*
> *     job1.setReducerClass(ID3ClsLabelReducer.class);*
> *     Path in = new Path(args[0]);*
> *     Path out1 = new Path(CL);*
> *     if(counter == 0){*
> *            FileInputFormat.addInputPath(job1, in);*
> *     }*
> *     else{*
> *            FileInputFormat.addInputPath(job1, out5);   *
> *     }*
> *     FileInputFormat.addInputPath(job1, in);*
> *     FileOutputFormat.setOutputPath(job1,out1);*
> *     job1.waitForCompletion(true);*
> *    /**
> *     * Job 2: *
> *     *  *
> *     */*
> *    Configuration conf2 = getConf();*
> *    Job job2 = new Job(conf2, "");*
> *    Path out2 = new Path(ANC);*
> *    FileInputFormat.addInputPath(job2, in);*
> *    FileOutputFormat.setOutputPath(job2,out2);*
> *   job2.waitForCompletion(true);*
>
>  *   /**
> *     * Job3*
> *    */*
> *    Configuration conf3 = getConf();*
> *    Job job3 = new Job(conf3, "");*
> *    System.out.println("conf3");*
> *    Path out5 = new Path(args[1]);*
> *    if(fs.exists(out5)){*
> *        fs.delete(out5, true);*
> *    }*
> *    FileInputFormat.addInputPath(job3,out2);*
> *    FileOutputFormat.setOutputPath(job3,out5);*
> *    job3.waitForCompletion(true);*
> *    FileInputFormat.addInputPath(job3,new Path(args[0]));*
> *    FileOutputFormat.setOutputPath(job3,out5);*
> *    job3.waitForCompletion(true);*
> *    counter =
> job3.getCounters().findCounter(UpdateCounter.INCOMING_ATTR).getValue();*
> *  }*
> * return 0;*
>
>  Am I doing anything wrong?
>
>
> On Mon, Jun 9, 2014 at 4:37 PM, Krishna Kumar <kkumar@nanigans.com> wrote:
>
>> You should use FileStatus to  decide what files you want to include in
>> the InputPath, and use the FileSystem class to delete or process the
>> intermediate / final paths. Moving each job in your iteration logic into
>> different methods would help keep things simple.
>>
>>
>>
>> From: unmesha sreeveni <unmeshabiju@gmail.com>
>> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
>> Date: Monday, June 9, 2014 at 6:02 AM
>> To: User Hadoop <user@hadoop.apache.org>
>> Subject: Re: Counters in MapReduce
>>
>> Ok I will check out with counters.
>> And after I st iteration the input file to job1 will be the output file
>> of job 3.How to give that.
>> *Inorder to satisfy 2 conditions*
>> First iteration : users input file
>> after first iteration :job 3 's output file as job 1 s input.
>>
>>
>>
>>> --
>>> *Thanks & Regards*
>>>
>>>
>>> *Unmesha Sreeveni U.B*
>>> *Hadoop, Bigdata Developer*
>>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>>> http://www.unmeshasreeveni.blogspot.in/
>>>
>>>
>>>
>>>   ------------------------------
>>> *Kai Voigt* Am Germaniahafen 1 k@123.org
>>>  24143 Kiel +49 160 96683050
>>>  Germany @KaiVoigt
>>>
>>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>>
>> *Unmesha Sreeveni U.B*
>> *Hadoop, Bigdata Developer*
>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>
>
>
> --
> *Thanks & Regards *
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>


-- 
*Thanks & Regards *


*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/

Mime
View raw message