Return-Path: X-Original-To: apmail-hadoop-common-user-archive@www.apache.org Delivered-To: apmail-hadoop-common-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 18CEE1047D for ; Thu, 22 Aug 2013 08:14:42 +0000 (UTC) Received: (qmail 49193 invoked by uid 500); 22 Aug 2013 08:14:36 -0000 Delivered-To: apmail-hadoop-common-user-archive@hadoop.apache.org Received: (qmail 48799 invoked by uid 500); 22 Aug 2013 08:14:35 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Delivered-To: moderator for user@hadoop.apache.org Received: (qmail 51082 invoked by uid 99); 21 Aug 2013 12:09:37 -0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of rabmdu@gmail.com designates 209.85.220.196 as permitted sender) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=O28qjcc+3ZwwOOYaL/CB/TTU+wCGKHeanjOi3SA23LE=; b=UHt/gckc0T+fUrzTeBrM5cMYfrkG8e/JrkuMOAN5EXh5cko+jYcibHSOd8IBZ/rn0Y 0rdKtal9TCvt/Qc9rA8VQo8NQuusJF6RZtHhICyLF5OfYJzsORqt4uUCS2vbmb8cavfO UTl+MVIRCaAQRKE66BwuvUrrCdQGzT+rl5ix+UIqtbqGrHY5BfksHX1qI1vB9Adyss+W 68NFAb9n1AXRc13+GoInStIuCrhMGvPE6AYXkk4YdKHSEKpdvgFoAM6CZE7BLPlo2u70 QSAK07fX/EuhjPFaroOh21lebxS4beOJQqvuMbg69bb2VYdl2d28lA//PAu1SVJ+qzuU 6Xig== MIME-Version: 1.0 X-Received: by 10.52.171.38 with SMTP id ar6mr5200036vdc.22.1377086950251; Wed, 21 Aug 2013 05:09:10 -0700 (PDT) Date: Wed, 21 Aug 2013 17:39:10 +0530 Message-ID: Subject: running map tasks in remote node From: rab ra To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=047d7b6d95e67ea39304e4740a45 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b6d95e67ea39304e4740a45 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Hello, Here is the new bie question of the day. For one of my use cases, I want to use hadoop map reduce without HDFS. Here, I will have a text file containing a list of file names to process. Assume that I have 10 lines (10 files to process) in the input text file and I wish to generate 10 map tasks and execute them in parallel in 10 nodes. I started with basic tutorial on hadoop and could setup single node hadoop cluster and successfully tested wordcount code. Now, I took two machines A (master) and B (slave). I did the below configuration in these machines to setup a two node cluster. hdfs-site.xml dfs.replication 1 dfs.name.dir /tmp/hadoop-bala/dfs/name dfs.data.dir /tmp/hadoop-bala/dfs/data mapred.job.tracker A:9001 mapred-site.xml mapred.job.tracker A:9001 mapreduce.tasktracker.map.tasks.maximum 1 core-site.xml fs.default.name hdfs://A:9000 In A and B, I do have a file named =91slaves=92 with an entry =91B=92 in it= and another file called =91masters=92 wherein an entry =91A=92 is there. I have kept my input file at A. I see the map method process the input file line by line but they are all processed in A. Ideally, I would expect those processing to take place in B. Can anyone highlight where I am going wrong? regards rab --047d7b6d95e67ea39304e4740a45 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
Hello,
=A0
Here is the new bie question of the day.
=A0
For one of my use cases, I want to use h= adoop map reduce without HDFS. Here, I will have a text file containing a l= ist of file names to process. Assume that I have 10 lines (10 files to proc= ess) in the input text file and I wish to generate 10 map tasks and execute= them in parallel in 10 nodes. I started with basic tutorial on hadoop and = could setup single node hadoop cluster and successfully tested wordcount co= de.
=A0
Now, I took two machines A (master) and B (slave). = I did the below configuration in these machines to setup a two node cluster= .
=A0
hdfs-site.xml
=A0
<?x= ml version=3D"1.0"?>
<?xml-stylesheet type= =3D"text/xsl" href=3D"configuration.xsl"?>
<!-- Put site-specific property overrides in this= file. -->
<configuration>
&= lt;property>
=A0=A0=A0=A0=A0=A0=A0=A0=A0 <name>dfs.replication= </name>
=A0=A0=A0=A0=A0=A0=A0=A0=A0 &= lt;value>1</value>
</property>
<p= roperty>
=A0 <name>dfs.name.dir</name>
<= div style=3D"font-family:Calibri,sans-serif">=A0 <value>/tmp/hadoop-bala/dfs/name</value>=
</property>
<p= roperty>
=A0 <name>dfs.data.dir</name>
<= div style=3D"font-family:Calibri,sans-serif">=A0 <value>/tmp/hadoop-bala/dfs/data</value>=
</property>
<p= roperty>
=A0=A0=A0=A0 <name>mapred.job.tracker</name>= ;
=A0=A0=A0 <value>A:9001</value&g= t;
</property>
=A0
</configuration>
=A0
mapred-site.xml
=A0
= <?xml version=3D"1.0"?>
<?xml-stylesheet type= =3D"text/xsl" href=3D"configuration.xsl"?>
=A0
<!-- Put site-specific property overrides in this fi= le. -->
=A0
<configuration>
&= lt;property>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <name>mapred.jo= b.tracker</name>
=A0=A0=A0=A0=A0=A0= =A0=A0=A0=A0=A0 <value>A:9001</value>
</property>
<p= roperty>
=A0=A0=A0=A0=A0=A0=A0=A0=A0 <name>mapreduce.taskt= racker.map.tasks.maximum</name>
=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0 <value>1</value>
</property>
</= configuration>
=A0
core-site.xml
=A0
<?xml version=3D"1.0"?>
<?xml-stylesheet type=3D"text/xsl" href=3D&qu= ot;configuration.xsl"?>
<!-- Put site-specific property overrides in this fi= le. -->
<configuration>
=A0=A0=A0=A0=A0=A0=A0=A0 <property>
<= div style=3D"font-family:Calibri,sans-serif">=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <name&g= t;fs.default.name= </name>
=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 <value= >hdfs://A:9000</value>
=A0=A0=A0= =A0=A0=A0=A0 </property>
</configuration>
= =A0
=A0
In A and B, I do have a file named = =91slaves=92 with an entry =91B=92 in it and another file called =91masters= =92 wherein an entry =91A=92 is there.
=A0
I have kept my input file at A. I see the map metho= d process the input file line by line but they are all processed in A. Idea= lly, I would expect those processing to take place in B.
=A0
Can anyone highlight where I am going wrong?=
=A0
=A0regard= s
rab
--047d7b6d95e67ea39304e4740a45--