hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Jeltema <brian.jelt...@digitalenvoy.net>
Subject long startup time for MR job
Date Fri, 12 Sep 2014 15:56:23 GMT
Running an Hadoop 2.4/HBase 0.98 MR Job on a 12-node cluster, I’m seeing a long startup delay
(about 2.5 minutes):

14/09/12 11:46:05 INFO client.RMProxy: Connecting to ResourceManager at prod-hdfs-14.hdfs.digitalenvoy.net/
14/09/12 11:48:31 INFO mapreduce.JobSubmitter: number of splits:650

this seems like a long time. Is this due to the overhead of moving all of the JAR files into
place, or is there
other overhead involved? I’m using a -libjars option with a list of JAR files that is automatically
generated by
a home-grown tool, and is not optimized. I’m wondering if I need to make it smarter.

View raw message