hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron Kimball <aa...@cloudera.com>
Subject Re: Distributed cache testing in local mode
Date Fri, 23 Jan 2009 00:14:47 GMT
Hi Bhupesh,

I've noticed the same problem -- LocalJobRunner makes the DistributedCache
effectively not work; so my code often winds up with two codepaths to
retrieve the local data :\

You could try running in pseudo-distributed mode to test, though then you
lose the ability to run a single-stepping debugger on the whole end-to-end

- Aaron

On Thu, Jan 22, 2009 at 11:29 AM, Bhupesh Bansal <bbansal@linkedin.com>wrote:

> Hey folks,
> I am trying to use Distributed cache in hadoop jobs to pass around
> configuration files , external-jars (job sepecific) and some archive data.
> I want to test Job end-to-end in local mode, but I think the distributed
> caches are localized in TaskTracker code which is not called in local mode
> Through LocalJobRunner.
> I can do some fairly simple workarounds for this but was just wondering if
> folks have more ideas about it.
> Thanks
> Bhupesh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message