hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Amogh Vasekar <am...@yahoo-inc.com>
Subject Re: Hadoop dfs can't allocate memory with enough hard disk space when data gets huge
Date Mon, 19 Oct 2009 13:01:01 GMT
Hi,
It would be more helpful if you provide the exact error here.
Also, hadoop uses the local FS to store intermediate data, along with HDFS for final output.
If your job is memory intensive, try limiting the number of tasks you are running in parallel
on a machine.

Amogh


On 10/19/09 8:27 AM, "Kunsheng Chen" <keyek@yahoo.com> wrote:

I and running a hadoop program to perform MapReduce work on files inside a folder.

My program is basically doing Map and Reduce work, each line of any file is a pair of string,
and the result is a string associate with occurence inside all files.

The program works fine until the number of files grow to about 80,000,then the 'cannot allocate
memory' error occur for some reason.

Each of the file contains around 50 lines, but the total size of all files is no more than
1.5 GB. There are 3 datanodes performing calculation,each of them have more than 10GB hd left.

I am wondering if that is normal for Hadoop because the data is too large ? Or it might be
my programs problem ?

It is really not supposed to be since Hadoop was developed for processing large data sets.


Any idea is well appreciated











Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message