hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alex Kozlov <ale...@cloudera.com>
Subject Re: Distance Calculation problem
Date Wed, 28 Jul 2010 18:56:08 GMT
Hi Erik,

Your assumption is that the values are coming in the same order as they were
emitted by the mapper.  This is not part of the MapReduce contract.

MapReduce just gives you all the values for a given key.

For your particular problem I don't think you need a reducer at all: just
compute the distance in the mapper and set the # of reducers to 0 (*
mapred.reduce.tasks*).  You do not need the reduce step.

Alex K

On Wed, Jul 28, 2010 at 11:44 AM, Erik Test <erik.shiken@gmail.com> wrote:

> Hello,
>
> I've implemented a program using map reduce for a simple distance
> calculations between two 2D points. I've set up my input such that all
> calculations should be the same but they are not. This makes me think I'm
> doing something wrong within a map and/or reduce function. Here is my
> output.
>
> 1,7,13,19       18.973665961010276
> 10,16,22,28     8.48528137423857
> 11,17,23,29     8.48528137423857
> 12,18,24,30     8.48528137423857
> 13,19,25,31     8.48528137423857
> 14,20,26,32     8.48528137423857
> 15,21,27,33     18.973665961010276
> 16,22,28,34     18.973665961010276
> 17,23,29,35     18.973665961010276
> 18,24,30,36     8.48528137423857
> 19,25,31,37     18.973665961010276
> 2,8,14,20       8.48528137423857
> 20,26,32,38     8.48528137423857
> 21,27,33,39     8.48528137423857
> 22,28,34,40     8.48528137423857
> 23,29,35,41     8.48528137423857
> 24,30,36,42     8.48528137423857
> 25,31,37,43     18.973665961010276
> 26,32,38,44     18.973665961010276
> 27,33,39,45     18.973665961010276
> 28,34,40,46     16.97056274847714
> 29,35,41,47     8.48528137423857
> 3,9,15,21       8.48528137423857
> 30,36,42,48     18.973665961010276
> 31,37,43,49     18.973665961010276
> 32,38,44,50     8.48528137423857
> 33,39,45,51     8.48528137423857
> 34,40,46,52     8.48528137423857
> 35,41,47,53     18.973665961010276
> 4,10,16,22      18.973665961010276
> 5,11,17,23      18.973665961010276
> 6,12,18,24      8.48528137423857
> 7,13,19,25      8.48528137423857
> 8,14,20,26      8.48528137423857
> 9,15,21,27      18.973665961010276
>
> My key is the entire line entry of the input file and the value is the
> distance calculated. Any value that doesn't begin with an 8 is wrong.
>
> This is what I do in the mapper.
>
>  public static class Map extends MapReduceBase
>    implements Mapper<LongWritable, Text,
>      Text, DoubleWritable>
>        {
>          private Text word = new Text();
>
>          public void map(LongWritable key, Text value,
>            OutputCollector<Text, DoubleWritable> output,
>              Reporter reporter) throws IOException
>                {
>                  String line = value.toString();
>                  String[] tokens = line.split("[,]");
>
>                  for(int i = 0; i < tokens.length; i++)
>                  {
>                    word.set(line);
>                    output.collect(word, new
> DoubleWritable(Double.parseDouble(tokens[i])));
>                  }
>                }//public void map
>        }//public static class Map
>
> And this is what I do in the reducer.
>
> public static class Reduce extends MapReduceBase
>    implements Reducer<Text, DoubleWritable,
>      Text, DoubleWritable>
>        {
>          public void reduce(Text key, Iterator<DoubleWritable> values,
>            OutputCollector<Text, DoubleWritable> output, Reporter reporter)
>              throws IOException
>                {
>                  double distance  = 0;
>                  double x1 = 0;
>                  double x2 = 0;
>                  double y1 = 0;
>                  double y2 = 0;
>
>                  if(values.hasNext())
>                  {
>                    x1 = values.next().get();
>                  }
>
>                  if(values.hasNext())
>                  {
>                    x2 = values.next().get();
>                  }
>
>                  if(values.hasNext())
>                  {
>                    y1 = values.next().get();
>                  }
>
>                  if(values.hasNext())
>                  {
>                    y2 = values.next().get();
>                  }
>
>                  distance = StrictMath.sqrt(StrictMath.pow(x2 - x1, 2.0) +
> StrictMath.pow(y2 - y1, 2.0));
>                  output.collect(key, new DoubleWritable(distance));
>                }
>        }//public static class Reduce
>
> Any suggestions would be appreciated.
>
> Erik
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message