hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peter Cogan <peter.co...@gmail.com>
Subject Re: Problem using distributed cache
Date Thu, 06 Dec 2012 21:29:57 GMT
Hi,

It's an instance created at the start of the program like this:

public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();


 Job job = new Job(conf, "wordcount");



 DistributedCache.addCacheFile(new URI("/user/peter/cacheFile/testCache1"),
conf);




On Thu, Dec 6, 2012 at 5:02 PM, Harsh J <harsh@cloudera.com> wrote:

> What is your conf object there? Is it job.getConfiguration() or an
> independent instance?
>
> On Thu, Dec 6, 2012 at 10:29 PM, Peter Cogan <peter.cogan@gmail.com>
> wrote:
> > Hi ,
> >
> > I want to use the distributed cache to allow my mappers to access data.
> In
> > main, I'm using the command
> >
> > DistributedCache.addCacheFile(new
> URI("/user/peter/cacheFile/testCache1"),
> > conf);
> >
> > Where /user/peter/cacheFile/testCache1 is a file that exists in hdfs
> >
> > Then, my setup function looks like this:
> >
> > public void setup(Context context) throws IOException,
> InterruptedException{
> >     Configuration conf = context.getConfiguration();
> >     Path[] localFiles = DistributedCache.getLocalCacheFiles(conf);
> >     //etc
> > }
> >
> > However, this localFiles array is always null.
> >
> > I was initially running on a single-host cluster for testing, but I read
> > that this will prevent the distributed cache from working. I tried with a
> > pseudo-distributed, but that didn't work either
> >
> > I'm using hadoop 1.0.3
> >
> > thanks Peter
> >
> >
>
>
>
> --
> Harsh J
>

Mime
View raw message