Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:from:to:date:subject:thread-topic:thread-index:
	message-id:in-reply-to:accept-language:content-language:
	x-ms-has-attach:x-ms-tnef-correlator:acceptlanguage:content-type:mime-version;
	b=ILQWg4dcPt6tH7OX8aLehhz+VbX+3dFIBkkEjzQOD2R/iHLWJZ//FSJwxgWgeq8s
From: Amogh Vasekar <amogh@yahoo-inc.com>
To: "common-user@hadoop.apache.org" <common-user@hadoop.apache.org>
Date: Mon, 19 Oct 2009 18:31:01 +0530
Subject: Re: Hadoop dfs can't allocate memory with enough hard disk space
 when data gets huge
Thread-Topic: Hadoop dfs can't allocate memory with enough hard disk space
 when data gets huge
Thread-Index: AcpQaB2aY6sUh2WtRTKjZS0KnB54BwAVBeUw
Message-ID: <C70260E5.265C%amogh@yahoo-inc.com>
In-Reply-To: <43980.31265.qm@web46215.mail.sp1.yahoo.com>
Accept-Language: en-US
Content-Language: en
acceptlanguage: en-US
Content-Type: multipart/alternative;
	boundary="_000_C70260E5265Camoghyahooinccom_"
MIME-Version: 1.0

--_000_C70260E5265Camoghyahooinccom_
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

Hi,
It would be more helpful if you provide the exact error here.
Also, hadoop uses the local FS to store intermediate data, along with HDFS =
for final output.
If your job is memory intensive, try limiting the number of tasks you are r=
unning in parallel on a machine.

Amogh


On 10/19/09 8:27 AM, "Kunsheng Chen" <keyek@yahoo.com> wrote:

I and running a hadoop program to perform MapReduce work on files inside a =
folder.

My program is basically doing Map and Reduce work, each line of any file is=
 a pair of string, and the result is a string associate with occurence insi=
de all files.

The program works fine until the number of files grow to about 80,000,then =
the 'cannot allocate memory' error occur for some reason.

Each of the file contains around 50 lines, but the total size of all files =
is no more than 1.5 GB. There are 3 datanodes performing calculation,each o=
f them have more than 10GB hd left.

I am wondering if that is normal for Hadoop because the data is too large ?=
 Or it might be my programs problem ?

It is really not supposed to be since Hadoop was developed for processing l=
arge data sets.


Any idea is well appreciated


--_000_C70260E5265Camoghyahooinccom_--