Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 6FDAF1066B for ; Fri, 20 Dec 2013 08:04:50 +0000 (UTC) Received: (qmail 17414 invoked by uid 500); 20 Dec 2013 08:04:41 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 17057 invoked by uid 500); 20 Dec 2013 08:04:41 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 17049 invoked by uid 99); 20 Dec 2013 08:04:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Dec 2013 08:04:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of xiaotao.cs.nju@gmail.com designates 209.85.216.47 as permitted sender) Received: from [209.85.216.47] (HELO mail-qa0-f47.google.com) (209.85.216.47) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 20 Dec 2013 08:04:33 +0000 Received: by mail-qa0-f47.google.com with SMTP id w5so5645870qac.20 for ; Fri, 20 Dec 2013 00:04:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=nM2ugU/d+2WT0xXCZE+vosn/tis8Pbhec5BjNxWoEqw=; b=V9LS5FrA7An4dWXoVfUpRego0wQhsC5gNc/hc7G/XnEoKnOQowZ60IjHz8Du6b4D1s hwsNB2TwFOl2nCkSvfrgvHoodZNzIYY73h5eXNtV2s7hE4uMqePxpa4/VirI/WzrhiFv a9BUzKziHPMyuCzMBm8H/ch/OvRjagVQ5shkavZWlncEKYUN0uAnz+PsuG9Xh2m+t9YJ 5102ytxGt3UrheWIwCe4+xlI1DY43NYl68z7d4xfNcO1fiax8NaXv0n1uCAsHmiGTsS/ qq64rk93EtYvv2PAz9F/6NhXVCIQXvC9WM4JmzRuQap1guZaZvn3blrjfv9uE9rJLQrl o+mg== MIME-Version: 1.0 X-Received: by 10.224.16.204 with SMTP id p12mr11474216qaa.26.1387526652617; Fri, 20 Dec 2013 00:04:12 -0800 (PST) Received: by 10.96.198.97 with HTTP; Fri, 20 Dec 2013 00:04:12 -0800 (PST) In-Reply-To: References: Date: Fri, 20 Dec 2013 16:04:12 +0800 Message-ID: Subject: Re: LOGGING in MapReduce From: Tao Xiao To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7bdc7dac3ee70204edf2b9c3 X-Virus-Checked: Checked by ClamAV on apache.org --047d7bdc7dac3ee70204edf2b9c3 Content-Type: text/plain; charset=ISO-8859-1 You can't log your mappers' System.out.println() or reducers' System.out.println() in console, because in console you just submit your job, but actually the map tasks and reduce tasks are running in different processes or different nodes which, is not running in the console. But you can find you log produced by mappers' System.out.println() or reducers' System.out.println() in the corresponding log files. 2013/12/20 unmesha sreeveni > I want to log my System.out.println() in console > How to do that. > I did the below code but it is not displaying any thing. I am using mapred > api the old one. > Did i do anything wrong? > > Code > ------------------------------------------------------------ > > package tech; > > import java.io.BufferedWriter; > import java.io.IOException; > import java.io.OutputStreamWriter; > import java.util.ArrayList; > import java.util.Iterator; > import java.util.StringTokenizer; > > import org.apache.hadoop.conf.Configuration; > import org.apache.hadoop.fs.FileSystem; > import org.apache.hadoop.fs.Path; > import org.apache.hadoop.io.IntWritable; > import org.apache.hadoop.io.Text; > import org.apache.hadoop.mapred.MapReduceBase; > import org.apache.hadoop.mapred.OutputCollector; > import org.apache.hadoop.mapred.Reducer; > import org.apache.hadoop.mapred.Reporter; > import com.sun.org.apache.commons.logging.LogFactory; > > public class Reduce extends MapReduceBase > implements Reducer { > > static int cnt =0; > ArrayList ar = new ArrayList(); > String data = null; > > *public static final com.sun.org.apache.commons.logging.Log LOG = > LogFactory.getLog(Reduce.class);* > > public void reduce(Text key, Iterator values, > OutputCollector output, > Reporter reporter) throws IOException { > > *System.out.println("In reducer");* > *LOG.info("In Reducer");* > > int sum = 0; > String line = key.toString(); > StringTokenizer itr = new StringTokenizer(line); > while (values.hasNext()) { > sum += values.next().get(); > } > output.collect(key, new IntWritable(sum)); > String data = key+" "+sum; > ar.add(data); > *writeToFile(ar);* > System.out.println("Wrote to file"); > *if(LOG.isDebugEnabled()){* > * LOG.debug("Log: In Reducer");* > * }* > > ar.add("\n"); > int index=Integer.parseInt(itr.nextToken()); > String value=itr.nextToken(); > String classLabel=itr.nextToken(); > int count=sum; > > } > > public static void *writeToFile*(ArrayList text) throws > IOException { > System.out.println("In reduce write to file "); > C45 id=new C45(); > System.out.println("count "+cnt); > > Path input = new Path("C45/intermediate"+id.current_index+".txt"); > Configuration conf = new Configuration(); > FileSystem fs = FileSystem.get(conf); > BufferedWriter bw = new BufferedWriter(new > > OutputStreamWriter(fs.create(input, true))); > for(String str: text) { > bw.write(str); > } > bw.newLine(); > bw.close(); > } > > } > 1. And how to log details: if my reducer data goes into function *writeToFile() > *and creates the file in hdfs. > > -- > *Thanks & Regards* > > Unmesha Sreeveni U.B > > *Junior Developer* > > > --047d7bdc7dac3ee70204edf2b9c3 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
You can't log your mappers' System.out.printl= n()=A0 or reducers' System.out.println()=A0 in console, because in cons= ole you just submit your job, but actually the map tasks and reduce tasks a= re running in different processes or different nodes which,=A0 is not runni= ng in the console.

But you can find you log produced by mappers' System.out.prin= tln()=A0 or reducers' System.out.println()=A0 in the corresponding log= files.


2013/12/20 unmesha sreeveni <unmeshabiju@gmail.com>
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
I want to log my System.out.println() in console
How to do that= .
I did= the below code but it is not displaying any thing. I am using mapred api t= he old one.
Did i do anything wrong?

Code
-------------------------= -----------------------------------

<= /div>
packag= e tech;

<= font face=3D"verdana, sans-serif">import java.io.IOException;
<= div class=3D"gmail_default"> import java.io.OutputStreamWriter;
impo= rt java.util.ArrayList;
import java.util.Iterator;
import java= .util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;=
import org.= apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;=
import org.apache.hadoop.mapred.MapRed= uceBase;
import org.= apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Redu= cer;
import org.= apache.hadoop.mapred.Reporter;
import com.sun.org.apache.commons.logging.L= ogFactory;

=
publi= c =A0class Reduce extends MapReduceBase
implements Reducer<Text, IntWri= table, Text, IntWritable> {

=
static int cnt =3D0;
ArrayList<String> ar =3D new ArrayList<String>();
String data =3D null;

=
public static final com.sun.org.= apache.commons.logging.Log LOG =3D LogFactory.getLog(Reduce.class);

=
public void reduce(Text key, Iterat= or<IntWritable> values,
OutputCollector<Text, IntWritable>= ; output,
Reporter reporter= ) throws IOException {

=
System.out.println("= ;In reducer");
LOG.info("In Reducer");=
int sum =3D 0;
=
String line =3D key.toString();
StringTokenizer itr =3D new StringTokeni= zer(line);
while (values.has= Next()) {
sum +=3D values.next().get();
}
output.collect(key, new IntWritable(sum)= );
String data =3D key+"= ; "+sum;
ar.add(data);
writeToFile(ar);
System.out.println("Wrote to file&q= uot;);
if(LOG.isDebugEnabled()){<= /div>
LOG.debug("Log: In Reducer= ");
}

<= /font>
ar.add("\n"= );
int index=3DInteger.parseInt(itr.nextTok= en());
String value=3Ditr.ne= xtToken();
String classLabel=3Ditr.nextToken();
int count=3Dsum;

=
}

public static void writeToFile(ArrayList<String> = =A0text) throws IOException {
System.out.println("In reduce write= to file ");
= C45 id=3Dnew C45();
System.out.println("count "+c= nt);

Path input =3D new Path(&q= uot;C45/intermediate"+id.current_index+".txt");
Configuration conf =3D new Configuratio= n();
FileSystem fs =3D File= System.get(conf);
BufferedWriter bw =3D new BufferedWrite= r(new
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 OutputStreamWriter(fs.create(input, tru= e)));
for(String str: text) {
bw.write(str);
}
bw.newLine();
bw.close();
}

}
1. And how to log details: if = my reducer data goes into function writeToFile() and creates the fil= e in hdfs.

-- <= br>
Thanks & Regards

Unmesha Sreeveni U.B
Junior Developer


--047d7bdc7dac3ee70204edf2b9c3--