hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From bejoy.had...@gmail.com
Subject Re: Problem using distributed cache
Date Fri, 07 Dec 2012 15:13:02 GMT
Hi Peter

Can you try the following in your code
1. Driver class to implement Tools interface
2. Do a getConfiguration() rather than creating a new conf instance.

DC should be working with the above mentioned modifications to code.

Sent on my BlackBerry® from Vodafone

-----Original Message-----
From: Peter Cogan <peter.cogan@gmail.com>
Date: Fri, 7 Dec 2012 14:06:41 
To: <user@hadoop.apache.org>
Reply-To: user@hadoop.apache.org
Subject: Re: Problem using distributed cache

Hi,

any thoughts on this would be much appreciated

thanks
Peter


On Thu, Dec 6, 2012 at 9:29 PM, Peter Cogan <peter.cogan@gmail.com> wrote:

> Hi,
>
> It's an instance created at the start of the program like this:
>
> public static void main(String[] args) throws Exception {
>
> Configuration conf = new Configuration();
>
>
>  Job job = new Job(conf, "wordcount");
>
>
>
>  DistributedCache.addCacheFile(new URI("/user/peter/cacheFile/testCache1"),
> conf);
>
>
>
>
> On Thu, Dec 6, 2012 at 5:02 PM, Harsh J <harsh@cloudera.com> wrote:
>
>> What is your conf object there? Is it job.getConfiguration() or an
>> independent instance?
>>
>> On Thu, Dec 6, 2012 at 10:29 PM, Peter Cogan <peter.cogan@gmail.com>
>> wrote:
>> > Hi ,
>> >
>> > I want to use the distributed cache to allow my mappers to access data.
>> In
>> > main, I'm using the command
>> >
>> > DistributedCache.addCacheFile(new
>> URI("/user/peter/cacheFile/testCache1"),
>> > conf);
>> >
>> > Where /user/peter/cacheFile/testCache1 is a file that exists in hdfs
>> >
>> > Then, my setup function looks like this:
>> >
>> > public void setup(Context context) throws IOException,
>> InterruptedException{
>> >     Configuration conf = context.getConfiguration();
>> >     Path[] localFiles = DistributedCache.getLocalCacheFiles(conf);
>> >     //etc
>> > }
>> >
>> > However, this localFiles array is always null.
>> >
>> > I was initially running on a single-host cluster for testing, but I read
>> > that this will prevent the distributed cache from working. I tried with
>> a
>> > pseudo-distributed, but that didn't work either
>> >
>> > I'm using hadoop 1.0.3
>> >
>> > thanks Peter
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>
>

Mime
View raw message