Return-Path: X-Original-To: apmail-hadoop-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 28068105D4 for ; Thu, 17 Apr 2014 04:17:16 +0000 (UTC) Received: (qmail 62700 invoked by uid 500); 17 Apr 2014 04:17:08 -0000 Delivered-To: apmail-hadoop-user-archive@hadoop.apache.org Received: (qmail 62136 invoked by uid 500); 17 Apr 2014 04:17:07 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 62125 invoked by uid 99); 17 Apr 2014 04:17:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 04:17:06 +0000 X-ASF-Spam-Status: No, hits=1.8 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,NORMAL_HTTP_TO_IP,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of kchew534@gmail.com designates 209.85.215.44 as permitted sender) Received: from [209.85.215.44] (HELO mail-la0-f44.google.com) (209.85.215.44) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 17 Apr 2014 04:17:02 +0000 Received: by mail-la0-f44.google.com with SMTP id c6so8912361lan.31 for ; Wed, 16 Apr 2014 21:16:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=00WEVaU4FZCMCg7vAFA92XxdyGluxtaf3YYqKz2dZbE=; b=WDmFAcLwsby6+93elbQTWQ74gcKqlofw/hD7SN3JQUBZWM4phRd6ebDbib70ddQlWG Grt60/rCZHL1OXfZkStdrXoOgoeqM+0lQ0xx3OBGDh82+Nf9w2Bhxh0vuCTEgmL4Hh5V 98Vr1wC2thdUBH3X5mTTM1b4cbieXzNeag0IrrgO70f0wZG7CXJEzSOOiwDboRaq7zQP EV/CCgGAEiBM0nnSBl9JeZP5tBLF8+T4ShfA94SLIFjM5SOp7d+z94bU8VQByZT1yD4B bAAg3WYKnNaRpqRtjMJxPMTMJUJgBkxYjBqLiPylxH6X9KjLV6TqzKRxfO1LXG0C1R/r 5SXg== MIME-Version: 1.0 X-Received: by 10.112.163.69 with SMTP id yg5mr5347214lbb.14.1397708201136; Wed, 16 Apr 2014 21:16:41 -0700 (PDT) Received: by 10.112.84.75 with HTTP; Wed, 16 Apr 2014 21:16:41 -0700 (PDT) In-Reply-To: <57A6E8D2-6312-49D0-81A3-1AFB96F0C22F@apache.org> References: <57A6E8D2-6312-49D0-81A3-1AFB96F0C22F@apache.org> Date: Wed, 16 Apr 2014 21:16:41 -0700 Message-ID: Subject: Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster From: Kim Chew To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=089e0118294ed415ed04f7354c9b X-Virus-Checked: Checked by ClamAV on apache.org --089e0118294ed415ed04f7354c9b Content-Type: text/plain; charset=UTF-8 Vinod, I am confused here. So could you please explain what actually happened under the hood if " mapreduce.framework.name" is set to "classic" on the cluster side? Or it is supposed to be set to "yarn" in the first place? Thanks. Kim On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar Vavilapalli wrote: > You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported > nor even possible. > > +Vinod > > On Apr 16, 2014, at 2:27 PM, Kim Chew wrote: > > I have a cluster running Hadoop 2 but it is not running YARN, i.e. " > mapreduce.framework.name" is set to "classic" therefore the > ResourceManager is not running. > > On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to > the above cluster. Here how my Hadoop-1.1.1 mapred-site.xml looks like, > > > > mapred.job.tracker > 172.31.3.150:8021 > > > Not surprisingly I got a version mismatched when I submit my job using the > Hadoop-1.1.1 jars, > > org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot > communicate with client version 4 > at org.apache.hadoop.ipc.Client.call(Client.java:1107) > > So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 > jars. Here is how my Hadoop 2 mapred-site.xml looks like, > > > > mapreduce.job.tracker.address > 172.31.3.150:8021 > > > mapreduce.framework.name > yarn > > > Note that I have to set "mapreduce.framework.name" to "yarn" otherwise > the job will be run locally instead of on the targeted cluster. But my > targeted cluster is not running YARN as stated above, > > 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at / > 172.31.3.150:8032 > 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: > hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s); > retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, > sleepTime=1 SECONDS) > > (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in > yarn-site.xml on my client.) > > Therefore it seems to me that it does not matter I have to recompile my > job with Hadoop 2 or not. The question is what should I do to enable > submitting my job remotely to the Hadoop 2 cluster ? What are the > configurations I need to set on the client side? > > The only solution I can think of is to enable YARN on the Hadoop 2 cluster > but is it necessary? > > I am running out of pointers and stuck 8-( > > TIA > > Kim > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity > to which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. --089e0118294ed415ed04f7354c9b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Vinod, I am confused here.

So = could you please explain what actually happened under the hood if "mapreduce.framework.name"= is set to "classic" on the cluster side? Or it is supposed to be= set to "yarn" in the first place?

Thanks.

Kim

=
On Wed, Apr 16, 2014 at 7:06 PM, Vinod Kumar= Vavilapalli <vinodkv@apache.org> wrote:
You cann= ot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor e= ven possible.

+Vinod

On Apr= 16, 2014, at 2:27 PM, Kim Chew <kchew534@gmail.com> wrote:

Not surprisingly I go= t a version mismatched when I submit my job using the Hadoop-1.1.1 jars,
org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot comm= unicate with client version 4
=C2=A0=C2=A0=C2=A0 at org.apache.hadoop.ip= c.Client.call(Client.java:1107)

So I recompiled my job with Ha= doop 2 and submitted it using the Hadoop 2 jars. Here is how my Hadoop 2 ma= pred-site.xml looks like,

<property>
=C2=A0=C2=A0=C2=A0 <!-- Pointed to the remote Jo= bTracker -->
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 <name>m= apreduce.job.tracker.address</name>
=C2=A0=C2=A0=C2=A0 =C2=A0=C2= =A0=C2=A0 <value>172.31.3.150:8021</value>
=C2=A0=C2=A0=C2=A0 </property>
=C2=A0=C2=A0=C2=A0 <property>=
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 <name>mapreduce.framework.name</= name>
=C2=A0=C2=A0=C2=A0 =C2=A0=C2=A0=C2=A0 <value>yarn</val= ue>
=C2=A0=C2=A0=C2=A0 </property>

Note that I have to set "mapreduce.framework.name" to "yar= n" otherwise the job will be run locally instead of on the targeted cl= uster. But my targeted cluster is not running YARN as stated above,

14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at= /172.31.3.150:8032=
14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032. Already tried 0 time(s= ); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=3D10, sle= epTime=3D1 SECONDS)

(Yes I have set my "yarn.resourcemanager.hostname" to &= quot;172.31.3.150" in yarn-site.xml on my client.)

Theref= ore it seems to me that it does not matter I have to recompile my job with = Hadoop 2 or not. The question is what should I do to enable submitting my j= ob remotely to the Hadoop 2 cluster ? What are the configurations I need to= set on the client side?

The only solution I can think of is to enable YARN on the Ha= doop 2 cluster but is it necessary?

I am running ou= t of pointers and stuck 8-(

TIA

Kim



CONFIDENTIALITY NOTICE
NOTICE: This message is = intended for the use of the individual or entity to which it is addressed a= nd may contain information that is confidential, privileged and exempt from= disclosure under applicable law. If the reader of this message is not the = intended recipient, you are hereby notified that any printing, copying, dis= semination, distribution, disclosure or forwarding of this communication is= strictly prohibited. If you have received this communication in error, ple= ase contact the sender immediately and delete it from your system. Thank Yo= u.

--089e0118294ed415ed04f7354c9b--