hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@yahoo-inc.com>
Subject Re: scaling issue, please help
Date Wed, 02 Jul 2008 04:35:45 GMT

On 7/1/08 3:20 PM, "Mori Bellamy" <mbellamy@apple.com> wrote:
> i've got a mapreduce task that works on small (~1G) input. when i try
> to run the same task on large (~100G) input, i get the following error
> around when the map tasks are almost done (~98%)

[error list deleted]

> I'm running the task on a cluster of 5 workers, one DFS master, and
> one task tracker. i'm chaining mapreduce tasks, so i'm using
> SequenceFileOutput and SequenceFileInput. this error happens before
> the first link in the chain sucessfully reduces.
> does anyone have any insight? thanks!

    Any chance your tasks are running out of memory?  I've seen similar
errors when we had our memory watchdog set too low and the tasks were killed
during the shuffle. Woops. :)

View raw message