hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hitesh Shah <hit...@apache.org>
Subject Re: Best approach to detecting local mode in Hadoop 2.x
Date Sat, 27 Jun 2015 19:17:28 GMT
For 2.x, you should check the mapreduce.framework.name property
    - “yarn” implies cluster mode ( using YARN )
    - “local” implies local mode
    
yarn.resourcemanager.hostname set to “0.0.0.0” could also mean cluster mode with everything
installed on one machine.

— Hitesh  


On Jun 27, 2015, at 8:33 AM, Ken Krugler <kkrugler_lists@transpac.com> wrote:

> Hi all,
> 
> For some of the helicopter stunts we do while instrumenting Hadoop jobs, we need to know
whether we're running in local vs. distributed mode.
> 
> This is done on the client side, when setting up a Cascading-based workflow.
> 
> In the past (pre-2.x) we would get the mapred.job.tracker property out of the JobConf
and compare it to "local".
> 
> That no longer worked with 2.x, so we first see if the "yarn.resourcemanager.hostname"
property exists, and if so whether it's equal to "0.0.0.0"; if so, then it's local.
> 
> Unfortunately this isn't working for 2.4 on AWS's Elastic Mapreduce (EMR), as it's still
set to 0.0.0.0.
> 
> So is there a recommended approach to this that works with versions > 1.x?
> 
> Thanks!
> 
> -- Ken
> 
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
> 


Mime
View raw message