hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: LocalDirAllocator and getLocalPathForWrite
Date Wed, 05 Jan 2011 23:46:35 GMT
Hi Marc,

Yes, using LocalFileSystem would work fine, or you can just use the normal
java.io.File APIs.


On Wed, Jan 5, 2011 at 3:26 PM, Marc Sturlese <marc.sturlese@gmail.com>wrote:

> Hey Todd,
> >>LocalDirAllocator is an internal-facing API and you shouldn't be using it
> >>from user code. If you write into mapred.local.dir like this, you will
> end
> >>up with conflicts between different tasks running from the same node
>  I know it's a bit odd usage but the thing is that I need to create files
> in
> the local file system, work in there with them amb after that upload them
> to
> hdfs (I use the outputcomitter.) To avoid the conflicts you talk about, I
> create a folder which looks like "mapred.local.dir"/taskId/attemptId and I
> work there and aparently I am having no problems.
> >>and there isn't usually a good reason to write to multiple drives from
> >>within a task
> When I said I had a cluster of one node, was just to try to clarify my
> doubt
> and explain the example. My cluster is bigger than that actually and each
> node has more than 1 phisical disk. To have multuple task running at the
> same time is what I do. I would like each task to write just to a single
> local disk but don't know how to do it.
> >>The working directory of your MR task is already within one of the
> drives,
> Is there a way to get a working directory in the local disk from the
> reducer?
> Could I do something similar to:
> FileSystem fs = FileSystem.get(conf);
> LocalFileSystem localFs = fs.getLocal(conf);
> Path path = localFs.getWorkingDirectory();
> I would apreciate if you can tell me a bit more about this.
> I need to deal with these files just in local and want them copied to hdfs
> just when I finish working with them.
> Thanks in advance.
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/LocalDirAllocator-and-getLocalPathForWrite-tp2199517p2202221.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.

Todd Lipcon
Software Engineer, Cloudera

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message