hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 王洪军 <wanghj...@gmail.com>
Subject Re: MapReduce on Local FileSystem
Date Fri, 31 May 2013 10:14:27 GMT
Ingesting the data in HDFS is slow  ,Because it need a jvm process. But if
you don't use hdfs, you can't benifit from its features.   Without hdfs,the
big data will not be splited and distributed; I think  the initial time of
jvm is affordable if data is big, and hadoop is not good choice if the data
 is small.
file://   is cited local data, without distribution, other tasktracker
can't cite it until you copy it to the node all tasktrackers reside.


2013/5/31 Harsh J <harsh@cloudera.com>

> Then why not simply run with Write Replication Factor set to 1?
>
> On Fri, May 31, 2013 at 12:54 PM, Agarwal, Nikhil
> <Nikhil.Agarwal@netapp.com> wrote:
> > Hi,
> >
> >
> >
> > Thank you for your reply. One simple answer can be to reduce the time
> taken
> > for ingesting the data in HDFS.
> >
> >
> >
> > Regards,
> >
> > Nikhil
> >
> >
> >
> > From: Sanjay Subramanian [mailto:Sanjay.Subramanian@wizecommerce.com]
> > Sent: Friday, May 31, 2013 12:50 PM
> > To: <user@hadoop.apache.org>
> > Cc: user@hadoop.apache.org
> >
> >
> > Subject: Re: MapReduce on Local FileSystem
> >
> >
> >
> > Basic question. Why would u want to do that ? Also I think the Map R
> Hadoop
> > distribution has an NFS mountable HDFS
> >
> > Sanjay
> >
> > Sent from my iPhone
> >
> >
> > On May 30, 2013, at 11:37 PM, "Agarwal, Nikhil" <
> Nikhil.Agarwal@netapp.com>
> > wrote:
> >
> > Hi,
> >
> >
> >
> > Is it possible to run MapReduce on multiple nodes using Local File system
> > (file:///)  ?
> >
> > I am able to run it in single node setup but in a multiple node setup the
> > “slave” nodes are not able to access the “jobtoken” file which is
> present in
> > the Hadoop.tmp.dir in “master” node.
> >
> >
> >
> > Please let me know if it is possible to do this.
> >
> >
> >
> > Thanks & Regards,
> >
> > Nikhil
> >
> >
> >
> > CONFIDENTIALITY NOTICE
> > ======================
> > This email message and any attachments are for the exclusive use of the
> > intended recipient(s) and may contain confidential and privileged
> > information. Any unauthorized review, use, disclosure or distribution is
> > prohibited. If you are not the intended recipient, please contact the
> sender
> > by reply email and destroy all copies of the original message along with
> any
> > attachments, from your computer system. If you are the intended
> recipient,
> > please be advised that the content of this message is subject to access,
> > review and disclosure by the sender's Email System Administrator.
>
>
>
> --
> Harsh J
>

Mime
View raw message