hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Something Something <mailinglist...@gmail.com>
Subject Is Hadoop's TooRunner thread-safe?
Date Wed, 19 Mar 2014 17:15:15 GMT
I would like to trigger a few Hadoop jobs simultaneously.  I've created a
pool of threads using Executors.newFixedThreadPool.  Idea is that if the
pool size is 2, my code will trigger 2 Hadoop jobs at the same exact time
using 'ToolRunner.run'.  In my testing, I noticed that these 2 threads keep
stepping on each other.

When I looked under the hood, I noticed that ToolRunner creates
GenericOptionsParser which in turn calls a static method
'buildGeneralOptions'.  This method uses 'OptionBuilder.withArgName' which
uses an instance variable called, 'argName'.  This doesn't look thread safe
to me and I believe is the root cause of issues I am running into.

Any thoughts?

View raw message