hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From alx...@aim.com
Subject Re: reducer gets values with empty attributes
Date Wed, 01 May 2013 18:12:50 GMT

Here is the map and reduce part of the code

  public void map(Text key, Writable  value, OutputCollector<Text, Document> output,
Reporter reporter) throws IOException {

      ParseData parseData = (ParseData) value;
      Metadata parseMeta = parseData.getParseMeta();
        String[] id = parseMeta.getValues("id");      
        String [] ids=id[0].split(",");       
        int len=ids.length;      
        for(int i=0;i<len;i++)
           Document doc = new Document();
           doc.add("id", ids[i]);
           output.collect(new Text(ids[i]), doc);
          //    for testing
            List list=doc.getField("id").getValues();
           for(int j=0;j<list.size();j++){
              System.out.println("map:sending to reducer id field for key "+ids[i]+" as "+list.get(j));


public void reduce(Text key, Iterator<Document> values, OutputCollector<Text, Document>
output, Reporter reporter) throws IOException{
    Document doc=values.next();
    System.out.println("reducer keys are  "+key +" and size of doc is  "+doc.getField("id").getValues().size()

    List list=doc.getField("id").getValues();
    System.out.println("reducer: list size is "+list.size() +"\n");
    for(int j=0;j<list.size();j++){
    System.out.println("reduce:id field for key "+key+" is "+list.get(j));
    //output.collect(key, doc);

 System.out.println in map function outputs expected values for id field for each document.
However, System.out.println in reducer function outputs keys but no values for id field.

Thanks in advance.

-----Original Message-----
From: Mahesh Balija <balijamahesh.mca@gmail.com>
To: user <user@hadoop.apache.org>
Sent: Mon, Apr 29, 2013 11:33 pm
Subject: Re: reducer gets values with empty attributes

Hi Alex,

             Can you please attach your code? and the sample input data.

Mahesh Balija,

Calsoft Labs.

On Tue, Apr 30, 2013 at 2:29 AM,  <alxsss@aim.com> wrote:


I try to write mapreduce program in hadoop -1.0.4. using mapred libs. I have a map function
which gets

keys and creates a different object with a few attributes like id and etc and passes it to
reducer function using

 output.collect(key, value);

Reducer gets keys, but values  has empty  fields (like id and etc) , although map correctly
assigns these fields to each value.

Any ideas why this happens?


View raw message