hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hardik Pandya <smarty.ju...@gmail.com>
Subject Re: when it's safe to read map-reduce result?
Date Fri, 28 Mar 2014 18:06:31 GMT
if the job complets without any failures exitCode should be 0 and safe
to read the result
public class MyApp extends Configured implements Tool {

       public int run(String[] args) throws Exception {
         // Configuration processed by ToolRunner
         Configuration conf = getConf();

         // Create a JobConf using the processed conf
         JobConf job = new JobConf(conf, MyApp.class);

         // Process custom command-line options
         Path in = new Path(args[1]);
         Path out = new Path(args[2]);

         // Specify various job-specific parameters
         job.setJobName("my-app");
         job.setInputPath(in);
         job.setOutputPath(out);
         job.setMapperClass(MyMapper.class);
         job.setReducerClass(MyReducer.class);

         // Submit the job, then poll for progress until the job is complete
         JobClient.runJob(job);
         return 0;
       }

       public static void main(String[] args) throws Exception {
         // Let ToolRunner handle generic command-line options
         int res = ToolRunner.run(new Configuration(), new MyApp(), args);

         System.exit(res);
       }
     }



On Fri, Mar 28, 2014 at 4:41 AM, Li Li <fancyerii@gmail.com> wrote:

> thanks. is the following codes safe?
> int exitCode=ToolRunner.run()
> if(exitCode==0){
>    //safe to read result
> }
>
> On Fri, Mar 28, 2014 at 4:36 PM, Dieter De Witte <drdwitte@gmail.com>
> wrote:
> > _SUCCES implies that the job has succesfully terminated, so this seems
> like
> > a reasonable criterion.
> >
> > Regards, Dieter
> >
> >
> > 2014-03-28 9:33 GMT+01:00 Li Li <fancyerii@gmail.com>:
> >
> >> I have a program that do some map-reduce job and then read the result
> >> of the job.
> >> I learned that hdfs is not strong consistent. when it's safe to read the
> >> result?
> >> as long as output/_SUCCESS exist?
> >
> >
>

Mime
View raw message