hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: MapReduce jobs remotely
Date Thu, 03 May 2012 05:02:49 GMT
Kevin,

What version of Pig are you using?

Have you tried setting the right MR home directory to point Pig to the
local MR configuration for YARN?

$ HADOOP_MAPRED_HOME=/usr/lib/hadoop-mapreduce $PIG_HOME/bin/pig

Usually does it for me, so long as I have
/usr/lib/hadoop-mapreduce/conf configured properly for YARN+MR (and
considering that my YARN libs, etc. are all inside
/usr/lib/hadoop-mapreduce).

On Thu, May 3, 2012 at 12:11 AM, Kevin <kevin.macksamie@gmail.com> wrote:
> Hi,
>
> I have a cluster running YARN, and mapreduce jobs run as expected when they
> are executed from one of the nodes. However, when I run Pig scripts from a
> remote client, Pig connects to HDFS and HBase but runs its MapReduce job
> using the LocalJobRunner. Jobs finish successfully, but they aren't using
> the YARN architecture. I have placed all the configuration files in the Pig
> configuration directory, and this must be right otherwise Pig wouldn't
> connect to my cluster's HDFS and HBase.
>
> I have even put "mapreduce.framework.name=yarn" in the pig.properties file.
>
> Any ideas to get jobs submitted to a remote Hadoop cluster to work in
> distributed mode?
>
> -Kevin



-- 
Harsh J

Mime
View raw message