hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 梁景明 <futur...@gmail.com>
Subject Re: How to skip fail map to done the job
Date Thu, 22 Oct 2009 02:06:38 GMT
thank you ,it works

2009/10/21 Amogh Vasekar <amogh@yahoo-inc.com>

> For skipping failed tasks try : mapred.max.map.failures.percent
>
> Amogh
> On 10/21/09 8:58 AM, "梁景明" <futureha@gmail.com> wrote:
>
> hi, I use hadoop0.20 and 8 nodes, there is a job that has 130 map to run
> ,and  completed 128 map,
> but only 2 map fail ,and its fail in my case is accepted ,but the job fail
> ,the last 128 map also fail,
> i can't get any result.
> i found that SkipBadRecords maybe work for my case,but i used it ,and it
> doesnt work,maybe something
> wrong,here is my code , how to skip the bad fail map ,thanks for any help
>
>  JobConf conf = new JobConf(ProductMR.class);
>  conf.setJobName("ProductMR");
>  conf.setOutputKeyClass(Text.class);
>  conf.setOutputValueClass(Product.class);
>  conf.setMapperClass(Map.class);
>  conf.setReducerClass(Reduce.class);
>  conf.setMapOutputCompressorClass(DefaultCodec.class);
>  conf.setInputFormat(SequenceFileInputFormat.class);
>  conf.setOutputFormat(SequenceFileOutputFormat.class);
>  String objpath = "abc1";
>  SequenceFileInputFormat.addInputPath(conf, new Path(objpath));
>  SkipBadRecords.setMapperMaxSkipRecords(conf, Long.MAX_VALUE);
>  SkipBadRecords.setAttemptsToStartSkipping(conf, 0);
>  SkipBadRecords.setSkipOutputPath(conf, new Path("data/product/skip/"));
>  String output = "abc";
>  SequenceFileOutputFormat.setOutputPath(conf, new Path(output));
>  JobClient.runJob(conf);
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message