Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 501E210A60 for ; Sat, 28 Dec 2013 20:14:44 +0000 (UTC) Received: (qmail 52326 invoked by uid 500); 28 Dec 2013 20:14:39 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 52132 invoked by uid 500); 28 Dec 2013 20:14:39 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 52125 invoked by uid 99); 28 Dec 2013 20:14:39 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Dec 2013 20:14:39 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of dg.dg.23@gmail.com designates 209.85.212.175 as permitted sender) Received: from [209.85.212.175] (HELO mail-wi0-f175.google.com) (209.85.212.175) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 28 Dec 2013 20:14:34 +0000 Received: by mail-wi0-f175.google.com with SMTP id hi5so15248652wib.8 for ; Sat, 28 Dec 2013 12:14:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=ab5NXSw/tlg2s/fIhF/dM+onP3ZiEHAosNoIDy3XM38=; b=F0njsDnOFhbw9/pmlzRjBqffq0VRIormD7p1Pna6cD+2613PlXpUwZB0v3qvJP+HHA Esj7tjQPX1RbUQlJcaRwgZuqbvzAS40f1vuxStZZ4Du1CaTRyoCWFTcUoIUaJnlJ4Mhh AwhhgrTdfIxXAO1T17RHtt2hUQezbu6K6O/nbkJvtF0pLzaa/qBFdheMwfCKnR+XJ50u dlmBWLiAajuPzJ8XdU0lnYDM7uP6QDWAPwX7xJXda8BOSNtYTAGEmd57BRFrkM8NvpWT VfCbL78Uk2Hfwer5m7fGBCZz/MksZUgC0H4YcheLxU5N9BLIA/p4PqGaGckq+uiq/4/P EkJw== MIME-Version: 1.0 X-Received: by 10.180.105.199 with SMTP id go7mr37867805wib.53.1388261652879; Sat, 28 Dec 2013 12:14:12 -0800 (PST) Sender: dg.dg.23@gmail.com Received: by 10.216.205.138 with HTTP; Sat, 28 Dec 2013 12:14:12 -0800 (PST) In-Reply-To: References: Date: Sat, 28 Dec 2013 15:14:12 -0500 X-Google-Sender-Auth: VdEpQR4cyD3J5wca9edBm9LC9GY Message-ID: Subject: Re: Request for a pointer to a MapReduce Program tutorial From: Diego Gutierrez To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=f46d04182592acdeea04ee9dda30 X-Virus-Checked: Checked by ClamAV on apache.org --f46d04182592acdeea04ee9dda30 Content-Type: text/plain; charset=ISO-8859-1 Hi, For the first question, the name of the file containing the response of MR process is like that: part-r-00000. This file is inside the output directory. The second question, to locate the file in your linux file system you have to go to your datanode directory. This directory path is in your hdfs-site.xml file in {HADOOP_INSTALLATION}/etc/hadoop. In my case the full path of the MR response file is: /home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-1387647958374/current/finalized. where my datanode directory is: /home/user/hadoop/data2.2/hdfs/namenode. In the /finalized directory, there are files named like that: blk_1073741836. In one of those files is the MR response file. This aproach is not good if you want to open the response with vi for example, instead you can copy the response file into your linux file system using *copyToLocal* command( https://hadoop.apache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.html#copyToLocal). The syntax could be like that: bin/hadoop dfs -copyToLocal /output/part-r-00000 /home/user/ Hope this help 2013/12/28 Sitaraman Vilayannur > Hi all, > Thanks much for the pointers. I ran word count it appears, > successfully. Currently in inspected the output like so, > /usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat > /user/sitaraman/output/* > Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library > /usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which > might have disabled stack guard. The VM will try to fix the stack > guard now. > It's highly recommended that you fix the library with 'execstack -c > ', or link it with '-z noexecstack'. > 13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load > native-hadoop library for your platform... using builtin-java classes > where applicable > as 2 > do 1 > do: 1 > i 2 > not 1 > say 1 > > What is the name of the file in the output directory? Can i read it > say using emacs or vi where will the directory /user/sitaraman/output > be in the linux fs, that is how to locate where it is.... > Thanks again for all the help. > Sitaraman > > On 12/28/13, Hardik Pandya wrote: > > I recently blogged about it - hope it helps > > > > > http://letsdobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-from-eclipse-kepler/ > > > > Regards, > > Hardik > > > > On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur < > > vrsitaramanietflists@gmail.com> wrote: > > > >> Hi, > >> Would much appreciate a pointer to a mapreduce tutorial which explains > >> how i can run a simulated cluster of mapreduce nodes on a single PC and > >> write a Java program with the MapReduce Paradigm. > >> Thanks very much. > >> Sitaraman > >> > > > --f46d04182592acdeea04ee9dda30 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,

For the first question, th= e name of the file containing the response of MR process is like that:=A0 p= art-r-00000. This file is inside the output directory.

The sec= ond question, to locate the file in your linux file system you have to go t= o your datanode directory. This directory path is in your hdfs-site.xml fil= e in {HADOOP_INSTALLATION}/etc/hadoop. In my case the full path of the MR r= esponse file is:
/home/user/hadoop/data2.2/hdfs/datanode/current/BP-1422783414-127.0.1.1-138= 7647958374/current/finalized.

where my datanode directory= is: /home/user/hadoop/data2.2/hdfs/namenode.

In the /fin= alized directory, there are files named like that:=A0 blk_1073741836. In on= e of those files is the MR response file. This aproach is not good if you w= ant to open the response with vi for example, instead you can copy the resp= onse file into your linux file system using copyToLocal command( https://hadoop.a= pache.org/docs/current2/hadoop-project-dist/hadoop-common/FileSystemShell.h= tml#copyToLocal ). The syntax could be like that: bin/hadoop dfs -copyT= oLocal /output/part-r-00000 /home/user/

Hope this help


2013/12/28 Sitaraman Vilayannur <vrsitaraman= ietflists@gmail.com>
Hi all,
=A0 Thanks much for the pointers. I ran word count it appears,
successfully. Currently in inspected the output like so,
/usr/local/Software/hadoop-2.2.0/bin/hadoop fs -cat /user/sitaraman/output/= *
Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library
/usr/local/Software/hadoop-2.2.0/lib/native/libhadoop.so.1.0.0 which
might have disabled stack guard. The VM will try to fix the stack
guard now.
It's highly recommended that you fix the library with 'execstack -c=
<libfile>', or link it with '-z noexecstack'.
13/12/28 15:13:06 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes
where applicable
as =A0 =A0 =A02
do =A0 =A0 =A01
do: =A0 =A0 1
i =A0 =A0 =A0 2
not =A0 =A0 1
say =A0 =A0 1

What is the name of the file in the output directory? =A0Can i read it
say using emacs or vi where will the directory /user/sitaraman/output
be in the linux fs, that is how to locate where it is....
Thanks again for all the help.
Sitaraman

On 12/28/13, Hardik Pandya <sm= arty.juice@gmail.com> wrote:
> I recently blogged about it - hope it helps
>
> http://lets= dobigdata.wordpress.com/2013/12/07/running-hadoop-mapreduce-application-fro= m-eclipse-kepler/
>
> Regards,
> Hardik
>
> On Fri, Dec 27, 2013 at 6:53 AM, Sitaraman Vilayannur <
> vrsitaramanietflists= @gmail.com> wrote:
>
>> Hi,
>> =A0Would much appreciate a pointer to a mapreduce tutorial which e= xplains
>> how i can run a simulated cluster of mapreduce nodes on a single P= C and
>> write a Java program with the MapReduce Paradigm.
>> =A0Thanks very much.
>> Sitaraman
>>
>

--f46d04182592acdeea04ee9dda30--