hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinod Kumar Vavilapalli <vino...@apache.org>
Subject Re: Submit a Hadoop 1.1.1 job remotely to a Hadoop 2 cluster
Date Thu, 17 Apr 2014 02:06:36 GMT
You cannot run JobTracker/TaskTracker in Hadoop 2. It's neither supported nor even possible.

+Vinod

On Apr 16, 2014, at 2:27 PM, Kim Chew <kchew534@gmail.com> wrote:

> I have a cluster running Hadoop 2 but it is not running YARN, i.e. "mapreduce.framework.name"
is set to "classic" therefore the ResourceManager is not running.
> 
> On the Client side, I want to submit a job compiled with Hadoop-1.1.1 to the above cluster.
Here how my Hadoop-1.1.1 mapred-site.xml looks like,
> 
> <property>
>         <!-- Pointed to the remote JobTracker -->
>         <name>mapred.job.tracker</name>
>         <value>172.31.3.150:8021</value>
>   </property>
> 
> Not surprisingly I got a version mismatched when I submit my job using the Hadoop-1.1.1
jars,
> 
> org.apache.hadoop.ipc.RemoteException: Server IPC version 9 cannot communicate with client
version 4
>     at org.apache.hadoop.ipc.Client.call(Client.java:1107)
> 
> So I recompiled my job with Hadoop 2 and submitted it using the Hadoop 2 jars. Here is
how my Hadoop 2 mapred-site.xml looks like,
> 
> <property>
>     <!-- Pointed to the remote JobTracker -->
>         <name>mapreduce.job.tracker.address</name>
>         <value>172.31.3.150:8021</value>
>     </property>
>     <property>
>         <name>mapreduce.framework.name</name>
>         <value>yarn</value>
>     </property>
> 
> Note that I have to set "mapreduce.framework.name" to "yarn" otherwise the job will be
run locally instead of on the targeted cluster. But my targeted cluster is not running YARN
as stated above,
> 
> 14/04/16 13:35:47 INFO client.RMProxy: Connecting to ResourceManager at /172.31.3.150:8032
> 14/04/16 13:35:49 INFO ipc.Client: Retrying connect to server: hadoop-host1.eng.narus.com/172.31.3.150:8032.
Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
sleepTime=1 SECONDS)
> 
> (Yes I have set my "yarn.resourcemanager.hostname" to "172.31.3.150" in yarn-site.xml
on my client.)
> 
> Therefore it seems to me that it does not matter I have to recompile my job with Hadoop
2 or not. The question is what should I do to enable submitting my job remotely to the Hadoop
2 cluster ? What are the configurations I need to set on the client side?
> 
> The only solution I can think of is to enable YARN on the Hadoop 2 cluster but is it
necessary?
> 
> I am running out of pointers and stuck 8-(
> 
> TIA 
> 
> Kim
> 


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

Mime
View raw message