hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <jason.had...@gmail.com>
Subject Re: SIMPLE QUESTIONS ABOUT DISTRIBUTEDCACHE...
Date Sat, 10 Oct 2009 18:38:48 GMT
The distributed cache does not behave as expected in PsuedoDistributed mode.

Since all of the tasks run within the same jvm, there is no easy way to
alter the class path to add distributed cache items, in addition there is no
way to change the working directory of the jvm to the 'per task directory'
that the normal task runner creates, where the distributed cache items made
available.

On Sat, Oct 10, 2009 at 10:48 AM, Alberto Luengo Cabanillas <
cabiwan@gmail.com> wrote:

> Hi everyone! I´m trying to distribute some common data between my nodes,
> but
> I´m unable to do it. I have some questions about this:
>
> 1.- I´m working in PseudoDistributed mode in Windows (under Cygwin) and my
> DFS server is running on a virtual machine (Ubuntu server). May I use
> DistributedCache normally with this configuration?
> 2.-In the case of the Mapper nodes, in the Driver class I add these two
> lines:
>
> URI uriMapper =new URI("/user/hadoop-user/data/mapper_configuration.dat");
> (I also verify this file already exists in the DFS).
> DistributedCache.addCacheFile(uriMapper,conf);
>
> 3.- In the Mapper class, I have this:
>
> @Override
>    protected void setup(Context cont) {
>       ...
>       String configureCacheName = new
> Path("/user/hadoop-user/data/mapper_configuration.dat").getName();
>       Path [] cacheFiles =
> DistributedCache.getLocalCacheFiles(cont.getConfiguration());
>       System.out.println("The file is: "+cacheFiles.toString());
>     ...
>    }
>
> Does anyone have an idea about what I´m doing wrong? (I´m going crazy,
> actually...)
>
> Thanks a lot in advance.
>
> --
> Alberto
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message