hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Peeyush Bishnoi <peeyu...@yahoo-inc.com>
Subject Re: Job failed when writing a huge file
Date Sun, 25 Jan 2009 17:58:06 GMT
Hello Tien ,

Hope this URL would help and illustrate you how blocks of data write on


It would be good if you paste the error you are getting while writing
the data on HDFS .

Thanks ,

On Sat, 2009-01-24 at 18:23 -0800, tienduc_dinh wrote:

> Hi everyone,
> I'm using now Hadoop 0.18.0 with 1 NameNode and 4 data nodes. By writing the
> file bigger than the maximal free space of each data node the job is often
> failed. 
> I've seen that the file is mostly written only on one node (e.g. N1) and if
> this node doesn't have enough space, Hadoop deletes the old chunks on node
> N1, tries on another node (e.g. N2) and so on. The job will be failed if the
> maximal retries are reached. 
> (I don't use the script "start-balancer.sh" or something like that for
> balancing my cluster in this test.)
> Sometimes it works after Hadoop really spread the file across the data
> nodes.
> I think it's not so good that Hadoop writes (and deletes) the whole huge
> file again and again instead of spreading it. 
> So my question is "how does the write algorithm work or how can I find such
> information ?"
> Any help is appreciated, thanks a lot.
> Tien Duc Dinh

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message