Return-Path: X-Original-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-mapreduce-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E738210168 for ; Thu, 17 Apr 2014 02:07:22 +0000 (UTC) Received: (qmail 98853 invoked by uid 500); 17 Apr 2014 02:07:15 -0000 Delivered-To: apmail-hadoop-mapreduce-user-archive@hadoop.apache.org Received: (qmail 98643 invoked by uid 500); 17 Apr 2014 02:07:14 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 98576 invoked by uid 99); 17 Apr 2014 02:07:13 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 02:07:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of vinodkv@hortonworks.com designates 209.85.220.49 as permitted sender) Received: from [209.85.220.49] (HELO mail-pa0-f49.google.com) (209.85.220.49) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 02:07:07 +0000 Received: by mail-pa0-f49.google.com with SMTP id lj1so11554349pab.22 for ; Wed, 16 Apr 2014 19:06:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:from:message-id:mime-version:subject:date :references:to:in-reply-to:content-type; bh=dloxB8XVeKuOFmiE8Wgo/NOtOkakk0uTCAXxp84fjnQ=; b=k1BMlZw0XGQDhMO+AraOkEOmE2xVaEGTQhGkDZx9fKHAMokhFDwb1vS0Eu2P4DYIQS ov+uAO6+a78+G3nlOkRUJZkSHZpBPvtrlkBQ/GuDcOaopSPjJKQNvFlXC1ZipUL+8Dk9 oCVVHAlTIC+n/yjd5+o2sHDhX+MwqRyu/7GKZ8IA3o4EE9rZBkcsKXOSfYGS1Pj8NHJR ZfAuMN4UHDEURYTEUnfHw1Y9fPiCyWKYDpTpB14uGKucPHHihSdyk7s8SxQtF4nsPAM3 sFt76e2KnMRiy/hT2Zopi8Kg5TRL/KQ9gPAFOgzWDV7JQ2tZNJ1IBHQcLxdEcJovZIaM bdDQ== X-Gm-Message-State: ALoCoQn4L9lxYEoy9IgpfC5ZLOAW7pdG4bBgaZpocO5e/eTLXcK0KN2G5LyVFhUuk7Dvu9vtZ1ZtN2nKZZvzoZwfb3TIW2Qm4evl+i0XhVNhwJZ3BYiz34k= X-Received: by 10.68.170.131 with SMTP id am3mr12184227pbc.97.1397700407005; Wed, 16 Apr 2014 19:06:47 -0700 (PDT) Received: from [10.11.3.74] ([192.175.27.2]) by mx.google.com with ESMTPSA id ei4sm49907921pbb.42.2014.04.16.19.06.38 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 16 Apr 2014 19:06:38 -0700 (PDT) Sender: Vinod Kumar Vavilapalli From: Vinod Kumar Vavilapalli Message-Id: <57A6E8D2-6312-49D0-81A3-1AFB96F0C22F@apache.org> Mime-Version: 1.0 (Mac OS X Mail 7.2 \(1874\)) Subject: Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster Date: Wed, 16 Apr 2014 19:06:36 -0700 References: To: user@hadoop.apache.org In-Reply-To: X-Mailer: Apple Mail (2.1874) Content-Type: multipart/mixed; boundary=047d7b86f7dc46192804f7337ce1 X-Virus-Checked: Checked by ClamAV on apache.org --047d7b86f7dc46192804f7337ce1 Content-Type: multipart/alternative; boundary="Apple-Mail=_521F8ECA-4BD3-430B-A893-51F649B780A6" --Apple-Mail=_521F8ECA-4BD3-430B-A893-51F649B780A6 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=ISO-8859-1 You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported n= or even possible. +Vinod On Apr 16, 2014, at 2:27 PM, Kim Chew wrote: > I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapre= duce.framework.name" is set to "classic" therefore the ResourceManager is n= ot running. >=20 > On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to = the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like, >=20 > > > mapred.job.tracker > 172.31.3.150:8021 > >=20 > Not surprisingly I got a version mismatched when I submit my job using th= e Hadoop-1.1.1 jars, >=20 > org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot commun= icate with client version 4 > at org.apache.hadoop.ipc.Client.call(Client.java:1107) >=20 > So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 = jars. Here is how my Hadoop 2 mapred-site.xml looks like, >=20 > > > mapreduce.job.tracker.address > 172.31.3.150:8021 > > > mapreduce.framework.name > yarn > >=20 > Note that I have to set "mapreduce.framework.name" to "yarn" otherwise th= e job will be run locally instead of on the targeted cluster. But my target= ed cluster is not running YARN as stated above, >=20 > 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /= 172.31.3.150:8032 > 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-hos= t1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); retry policy i= s RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sleepTime=3D1 SECOND= S) >=20 > (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in y= arn-site.xml on my client.) >=20 > Therefore it seems to me that it does not matter I have to recompile my j= ob with Hadoop 2 or not. The question is what should I do to enable submitt= ing my job remotely to the Hadoop 2 cluster ? What are the configurations I= need to set on the client side? >=20 > The only solution I can think of is to enable YARN on the Hadoop 2 cluste= r but is it necessary? >=20 > I am running out of pointers and stuck 8-( >=20 > TIA=20 >=20 > Kim >=20 --=20 CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to= =20 which it is addressed and may contain information that is confidential,=20 privileged and exempt from disclosure under applicable law. If the reader= =20 of this message is not the intended recipient, you are hereby notified that= =20 any printing, copying, dissemination, distribution, disclosure or=20 forwarding of this communication is strictly prohibited. If you have=20 received this communication in error, please contact the sender immediately= =20 and delete it from your system. Thank You. --Apple-Mail=_521F8ECA-4BD3-430B-A893-51F649B780A6 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=US-ASCII You cannot run JobTracker/= TaskTracker in Hadoop 2. It's neither supported nor even possible.

=
+Vinod

On Apr 16, 2014, at 2:27 PM, Kim = Chew <kchew534@gmail.com> w= rote:

I have a cl= uster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name" is set to "classic= " therefore the ResourceManager is not running.

On the Client side, I want to submit a job compiled with Hadoop-1= .1.1 to the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks l= ike,

<property>
        = <!-- Pointed to the remote JobTracker -->
        <name>mapred.job.tracker&l= t;/name>
        <value>172.31.3.150:8021</value>
&n= bsp; </property>

Not surprisingly I got a version mismat= ched when I submit my job using the Hadoop-1.1.1 jars,

org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot comm= unicate with client version 4
    at org.apache.hadoop.ip= c.Client.call(Client.java:1107)

So I recompiled my job with Ha= doop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 ma= pred-site.xml looks like,

<property>
    <!-- Pointed to the remote Jo= bTracker -->
        <name>m= apreduce.job.tracker.address</name>
      = ;  <value>172.31.3.150:802= 1</value>
    </property>
    <property>=
        <name>mapreduce.framework.name</name>
 =        <value>yarn</value>
 &n= bsp;  </property>

Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be run = locally instead of on the targeted cluster. But my targeted cluster is not = running YARN as stated above,

14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at= /172.31.3.150:8032
14/04/16 1= 3:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31= .3.150:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumC= ountWithFixedSleep(maxRetries=3D10, sleepTime=3D1 SECONDS)

(Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.1= 50" in yarn-site.xml on my client.)

Therefore it seems to me t= hat it does not matter I have to recompile my job with Hadoop 2 or not. The= question is what should I do to enable submitting my job remotely to the H= adoop 2 cluster ? What are the configurations I need to set on the client s= ide?

The only solution I can think of is to enable YARN on the Ha= doop 2 cluster but is it necessary?

I am running ou= t of pointers and stuck 8-(

TIA

Kim



CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u. --Apple-Mail=_521F8ECA-4BD3-430B-A893-51F649B780A6-- --047d7b86f7dc46192804f7337ce1 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - http://gpgtools.org iQEcBAEBAgAGBQJTTzcsAAoJECVa31bDbF8P7F0H/1ytkrU3KqXofJwztoIpYAbM wNNhYXqwKalNi2RE/nh1MB9cV7nFsCT+99451P2t2i/0854sq4Ma18+2p7ackVWq GhgY3TyWOyDBMqxKGN11UZhtb3LUPXWGgdE/3UedzdL/Q4VSHdcLTKypgvjZsvkj mAUj7x2uBj9IjMBzZfydGxRUDNt3jvtrzFmeSFWc+gNN6ZfTSqRj7ziFhy4kmR2U saGjjl2QjQdOzKjlouSiT6yPxK10Vg6Sxe8EztuBLsjOQckYfyQ7lqrQCWPeeob5 U+Zkm2jQdU3vQDa9EQ7VzLy4Fl/keJuJJaj2qfhjvRTa2rpFyfJsVExniuVsjIs= =dTlb -----END PGP SIGNATURE----- --047d7b86f7dc46192804f7337ce1--