hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Maximiliano Patricio Méndez <mmen...@despegar.com>
Subject Repeated localized resource in filecache
Date Thu, 17 Dec 2015 21:15:58 GMT
Hi,

I'm new to hadoop and I'm running the same map reduce job multiple times
with some unexpected behaviour.

In the mapreduce i'm adding a tmpjar dependency using the hbase utility:
TableMapReduceUtil.addDependencyJar(Redisson.class) - which finds the jar
based on the class and adds it to the "tmpjar" configuration of the
mapreduce job.

Every time I run the job a new jar added to
${yarn.nodemanager.local-dirs}/usercache/<myuser>/filecache

Is it normal to add the jar to this cache even if it's already there (under
a different ID for a previously submitted job)? Could it be that adding
dependencies with tmpjar is not the best way to add a commonly used
dependency to a mapreduce?

Thanks

Mime
View raw message