hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vitaliy Semochkin <vitaliy...@gmail.com>
Subject Re: hdfs system crashes when loading files bigger than local space left
Date Wed, 21 Jul 2010 10:02:16 GMT
On Fri, Jul 16, 2010 at 10:07 PM, Allen Wittenauer <awittenauer@linkedin.com
> wrote:

>
> On Jul 16, 2010, at 3:15 AM, Vitaliy Semochkin wrote:
> > That is likely way too small.
> > Will setting 512Mb be better in case the whole volume size is only 190Gb?
>
> I'd recommend at least 5gb.  I'm also assuming this same disk space isn't
> getting used for MapReduce.

Thank you for advise. I'll increase the amount to 6 gb (hope it will be
enough).
Same disk is used for MapReduce but M/R is not executed during loading.


> > Does hadoop detect/distinct the client that uploads data from datanode
> and not from datanode?
> > lets say I execute
>
> Yes.
>
> > hadoop -put someFile hdfs://namenode.mycompany.com/
> >
> > from namenode.mycompany.com and from some other pc. Will it be any
> different for hadoop and will hadoop orgonize data more balanced in the last
> case?
>
> Yes.
>
> Again, namenode is irrelevant.

I was doing it from namenode which was acting as datanode as well.


> Do not do put's from a datanode if you want the data to be reasonably
> balanced.

Thank you very much. Will perform putting from pc outside the hadoop
cluster.


Regards,
Vitaliy S

Mime
View raw message