Mailing-List: contact yarn-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of acm@hortonworks.com
 designates 209.85.220.50 as permitted sender)
From: Arun C Murthy <acm@hortonworks.com>
Mime-Version: 1.0 (Apple Message framework v1084)
Content-Type: multipart/alternative; boundary=Apple-Mail-102--299357186
Subject: Re: In memory shuffle phase
Date: Tue, 11 Jun 2013 17:35:50 -0700
In-Reply-To: <1370911566.29644.YahooMailNeo@web122406.mail.ne1.yahoo.com>
To: yarn-dev@hadoop.apache.org
References: <1370911461.76918.YahooMailNeo@web122401.mail.ne1.yahoo.com>
 <1370911566.29644.YahooMailNeo@web122406.mail.ne1.yahoo.com>
Message-Id: <FE708349-8939-479D-A55B-B30EC06F0E6B@hortonworks.com>

--Apple-Mail-102--299357186
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii

Take a look at Apache Tez =
(http://incubator.apache.org/projects/tez.html).

I'll warn you that it's a little rough, but works.

Arun

On Jun 10, 2013, at 5:46 PM, Grandl Robert wrote:

> Hi,
>=20
>=20
> It is possible to run shuffle phase in memory ? I mean, the mapper to =
write the intermediate output in memory only and the reducer to read it =
from there. So no disk involved for intermediate output.=20
>=20
>=20
> If so, what are the configuration parameters I need to tune ?
>=20
> Thanks,
> Robert

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/


--Apple-Mail-102--299357186--