hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unmesha sreeveni <unmeshab...@gmail.com>
Subject Re: Counters in MapReduce
Date Thu, 12 Jun 2014 04:59:11 GMT
I tried out by setting an enum to count no. of lines in output file from
job3.

But I am getting
14/06/12 10:12:30 INFO mapred.JobClient:     Total committed heap usage
(bytes)=1238630400
conf3
Exception in thread "main" java.lang.IllegalStateException: Job in state
DEFINE instead of RUNNING
at org.apache.hadoop.mapreduce.Job.ensureState(Job.java:116)
 at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:491)


Below is my current code

*static enum UpdateCounter {*
*        INCOMING_ATTR*
*    }*

*public static void main(String[] args) throws Exception {*
*    Configuration conf = new Configuration();*
*    int res = ToolRunner.run(conf, new Driver(), args);*
*    System.exit(res);*
*}*


*@Override*
*public int run(String[] args) throws Exception {*
*while(counter >= 0){*

*      Configuration conf = getConf();*
*     /**
*     * Job 1: *
*     */*
*     Job job1 = new Job(conf, "");*
*     //other configuration*
*     job1.setMapperClass(ID3ClsLabelMapper.class);*
*     job1.setReducerClass(ID3ClsLabelReducer.class);*
*     Path in = new Path(args[0]);*
*     Path out1 = new Path(CL);*
*     if(counter == 0){*
*            FileInputFormat.addInputPath(job1, in);*
*     }*
*     else{*
*            FileInputFormat.addInputPath(job1, out5);   *
*     }*
*     FileInputFormat.addInputPath(job1, in);*
*     FileOutputFormat.setOutputPath(job1,out1);*
*     job1.waitForCompletion(true);*
*    /**
*     * Job 2: *
*     *  *
*     */*
*    Configuration conf2 = getConf();*
*    Job job2 = new Job(conf2, "");*
*    Path out2 = new Path(ANC);*
*    FileInputFormat.addInputPath(job2, in);*
*    FileOutputFormat.setOutputPath(job2,out2);*
*   job2.waitForCompletion(true);*

 *   /**
*     * Job3*
*    */*
*    Configuration conf3 = getConf();*
*    Job job3 = new Job(conf3, "");*
*    System.out.println("conf3");*
*    Path out5 = new Path(args[1]);*
*    if(fs.exists(out5)){*
*        fs.delete(out5, true);*
*    }*
*    FileInputFormat.addInputPath(job3,out2);*
*    FileOutputFormat.setOutputPath(job3,out5);*
*    job3.waitForCompletion(true);*
*    FileInputFormat.addInputPath(job3,new Path(args[0]));*
*    FileOutputFormat.setOutputPath(job3,out5);*
*    job3.waitForCompletion(true);*
*    counter =
job3.getCounters().findCounter(UpdateCounter.INCOMING_ATTR).getValue();*
*  }*
* return 0;*

 Am I doing anything wrong?


On Mon, Jun 9, 2014 at 4:37 PM, Krishna Kumar <kkumar@nanigans.com> wrote:

> You should use FileStatus to  decide what files you want to include in the
> InputPath, and use the FileSystem class to delete or process the
> intermediate / final paths. Moving each job in your iteration logic into
> different methods would help keep things simple.
>
>
>
> From: unmesha sreeveni <unmeshabiju@gmail.com>
> Reply-To: "user@hadoop.apache.org" <user@hadoop.apache.org>
> Date: Monday, June 9, 2014 at 6:02 AM
> To: User Hadoop <user@hadoop.apache.org>
> Subject: Re: Counters in MapReduce
>
> Ok I will check out with counters.
> And after I st iteration the input file to job1 will be the output file of
> job 3.How to give that.
> *Inorder to satisfy 2 conditions*
> First iteration : users input file
> after first iteration :job 3 's output file as job 1 s input.
>
>
>
>> --
>> *Thanks & Regards*
>>
>>
>> *Unmesha Sreeveni U.B*
>> *Hadoop, Bigdata Developer*
>> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
>> http://www.unmeshasreeveni.blogspot.in/
>>
>>
>>
>>   ------------------------------
>> *Kai Voigt* Am Germaniahafen 1 k@123.org
>>  24143 Kiel +49 160 96683050
>>  Germany @KaiVoigt
>>
>>
>
>
> --
> *Thanks & Regards*
>
>
> *Unmesha Sreeveni U.B*
> *Hadoop, Bigdata Developer*
> *Center for Cyber Security | Amrita Vishwa Vidyapeetham*
> http://www.unmeshasreeveni.blogspot.in/
>
>
>


-- 
*Thanks & Regards *


*Unmesha Sreeveni U.B*
*Hadoop, Bigdata Developer*
*Center for Cyber Security | Amrita Vishwa Vidyapeetham*
http://www.unmeshasreeveni.blogspot.in/

Mime
View raw message