hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Krugler <kkrugler_li...@transpac.com>
Subject Best approach to detecting local mode in Hadoop 2.x
Date Sat, 27 Jun 2015 15:33:15 GMT
Hi all,

For some of the helicopter stunts we do while instrumenting Hadoop jobs, we need to know whether
we're running in local vs. distributed mode.

This is done on the client side, when setting up a Cascading-based workflow.

In the past (pre-2.x) we would get the mapred.job.tracker property out of the JobConf and
compare it to "local".

That no longer worked with 2.x, so we first see if the "yarn.resourcemanager.hostname" property
exists, and if so whether it's equal to ""; if so, then it's local.

Unfortunately this isn't working for 2.4 on AWS's Elastic Mapreduce (EMR), as it's still set

So is there a recommended approach to this that works with versions > 1.x?


-- Ken

Ken Krugler
+1 530-210-6378
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

View raw message