Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.198.233 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=iFCK3yfhyRDXfmjE5X1IHy9UE3IfwE7mJ/qBBDpRun8EZcdde+3RcoBfJtuGtPzDcERxVIX7+xvbT7nOZASW7J7bG2WTgs7I0JwkOExXDuA6HUn0df4wyy+Gdl7qOucw4TKJM75UOpXJSpetx5/ADih49oa/Dc4CcFfj5Mh3kds=
Message-ID: <c7d45fc70806011013u6e1f0712ladc908530328fcf6@mail.gmail.com>
Date: Sun, 1 Jun 2008 10:13:30 -0700
From: "Ted Dunning" <ted.dunning@gmail.com>
To: core-user@hadoop.apache.org
Subject: Re: In memory Map Reduce
In-Reply-To: <F35E1D0F-CBAC-4629-BEE0-5566893E857E@gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_2653_25870455.1212340410644"
References: <C49CD854-FCE3-4A7F-A997-35628C332C67@gmail.com>
	 <c7d45fc70806010108k2e752308k9d1ea2c047fbb77e@mail.gmail.com>
	 <F35E1D0F-CBAC-4629-BEE0-5566893E857E@gmail.com>

------=_Part_2653_25870455.1212340410644
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Hadoop goes to some lengths to make sure that things can stay in memory as
much as possible.  There are still cases, however, where intermediate
results are  normally written to disk.  That means that implementors will
have those time scales in their head as they do things which will inevitably
make the trade-offs somewhat poor compared to a system that never envisions
intermediate data being written to disk.

But other than guessing like this, I couldn't actually say how it would turn
out except that for very short jobs, moving jar files around and other
startup costs can be the dominant cost.

On Sun, Jun 1, 2008 at 5:05 AM, Martin Jaggi <m.jaggi@gmail.com> wrote:

>
> So in the case that all intermediate pairs fit into the RAM of the cluster,
> does the InMemoryFileSystem already allow the intermediate phase to be done
> without much disk access? Or what would be the current bottleneck in Hadoop
> in this scenario (huge computational load, not so much data in/out)
> according to your opinion?
>
>
>

------=_Part_2653_25870455.1212340410644--