hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From unmesha sreeveni <unmeshab...@gmail.com>
Subject Re: Get dynamic values in a user defined class from reducer.
Date Thu, 19 Dec 2013 04:15:27 GMT
Thanks for ur reply Robert Dyer.
Thanks for spending ur valuable time for clearing my doubts.

I need to pass some other values too


Reducer
------------
public  class Reduce extends MapReduceBase
implements Reducer<Text, IntWritable, Text, IntWritable> {

static int cnt =0;
ArrayList<String> ar = new ArrayList<String>();
String data = null;
public void reduce(Text key, Iterator<IntWritable> values,
OutputCollector<Text, IntWritable> output,
Reporter reporter) throws IOException {
System.out.println("In reducer");
 int sum = 0;
String line = key.toString();
StringTokenizer itr = new StringTokenizer(line);
while (values.hasNext()) {
sum += values.next().get();
}
output.collect(key, new IntWritable(sum));

String data = key+" "+sum;
ar.add(data);
               writeToFile(ar);
ar.add("\n");
int index=Integer.parseInt(itr.nextToken());
String value=itr.nextToken();
String classLabel=itr.nextToken();
int count=sum;

}

public static void writeToFile(ArrayList<String>  text) {
System.out.println("In reduce write to file ");
 try {

cnt++;
C45 id=new C45();
System.out.println("count "+cnt);

Path input = new Path("C45/intermediate"+id.current_index+".txt");
 Configuration conf = new Configuration();
                        FileSystem fs = FileSystem.get(conf);
BufferedWriter bw = new BufferedWriter(
new OutputStreamWriter(fs.create(input, true)));
 System.out.println("Text from Reducer: "+
"C45/intermediate"+id.current_index+".txt"+ text);
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+fs.exists(input));

for(String str: text) {
bw.write(str);
}

bw.newLine();
bw.close();
 } catch (Exception e) {
System.out.println("File is not creating in reduce");
}
}

}

Driver
--------
GainRatio  gainObj = new GainRatio();
gainObj.getcount();



GainRatio
--------------

public void getcount()
{
System.out.println("In gainratio --- getcount ");
C45 id=new C45();
FileInputStream fstream;
try {

Configuration config = new Configuration();
FileSystem dfs = FileSystem.get(config);
Path path = new Path("C45/intermediate"+id.current_index+".txt");
System.out.println("file
exists:C45/intermediate"+id.current_index+".txt"+dfs.exists(path));
System.out.println(path);
BufferedReader br=new BufferedReader(new InputStreamReader(dfs.open(path)));
String line;
StringTokenizer itr;
while ((line = br.readLine()) != null)   {
System.out.println("In gain Ratio : "+line);
itr= new StringTokenizer(line);
count[linenumber][0]=itr.nextToken();
count[linenumber][1]=itr.nextToken();
count[linenumber][2]=itr.nextToken();
count[linenumber][3]=itr.nextToken();
int i=linenumber;
linenumber++;
}
System.out.println("outsidec while");
count[linenumber][0]=null;
count[linenumber][1]=null;
count[linenumber][2]=null;
count[linenumber][3]=null;
br.close();
emptyfile(id.current_index);

} catch (Exception e) {

e.printStackTrace();

}
}



Here from reducer many files are created
eg: intermediate0.txt
     .....upto intermediate7.txt according to my input dataset.

But here only intermediate0.txt is created all other creation of file get
failed.
Is that the same reason you explained before.

For this also should i go for couters?



On Wed, Dec 18, 2013 at 11:09 PM, Robert Dyer <psybers@gmail.com> wrote:

> Generally speaking, static fields are not useful in Hadoop.
>
> The issue you are seeing is that the reducer is running in a separate VM
> (possibly on a different node!) and thus the static value you are reading
> inside of Mid is actually a separate instantiation of that class and field.
>
> If you have an integer you need to set and read from the driver/mid, you
> can use a counter.  E.g.:
>
>
> http://www.philippeadjiman.com/blog/2010/01/07/hadoop-tutorial-series-issue-3-counters-in-action/
>
> - Robert
>
> On Wed, Dec 18, 2013 at 4:02 AM, unmesha sreeveni <unmeshabiju@gmail.com>wrote:
>
>> Can any one pls suggest a good way
>> My scenario:
>> i hav
>> 1.Driver class
>> 2.Mapper class
>> 3.reducer class
>> 4.Mid class
>> After completing mapper it goes to reduce. From reducer it will be going
>> to Driver and from driver to Mid class
>>
>> so i need to get a data from reducer class to Mid class
>>  So for that i declared a static variable in reduce and tried to access
>> that variable in Mid class.But while executing i am only getting the 0
>> value. How to overcome this.
>>
>>
>> Reducer.java
>> --------------------
>> public  class Reduce extends MapReduceBase
>> implements Reducer<Text, IntWritable, Text, IntWritable> {
>>  int ig;
>> public void reduce(Text key, Iterator<IntWritable> values,
>>  OutputCollector<Text, IntWritable> output,
>> Reporter reporter) throws IOException {
>>                               ig = 9;
>> }
>> }
>> Driver.java
>>        gainObj.getcount();
>>
>>
>> Mid.java
>> public class Mid{
>> public void getcount(String args) throws IOException
>> {
>>                    //WANT TO GET THE IG VALUE IE "9" HERE So i did like
>> this
>>                         Reduce r = new Reduce();
>>  System.out.println("red "+ r.ig);
>> }
>> }
>> But my result is "0". Not 9.
>> How to solve this?
>>
>>
>>
>>
>> --
>> *Thanks & Regards*
>>
>> Unmesha Sreeveni U.B
>>
>> *Junior Developer*
>>
>>
>>
>


-- 
*Thanks & Regards*

Unmesha Sreeveni U.B

*Junior Developer*

Mime
View raw message