Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of harsh@cloudera.com designates
 209.85.223.177 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <869970D71E26D7498BDAC4E1CA92226B6589F561@MBX021-E3-NJ-2.exch021.domain.local>
References: 
 <869970D71E26D7498BDAC4E1CA92226B6589F561@MBX021-E3-NJ-2.exch021.domain.local>
From: Harsh J <harsh@cloudera.com>
Date: Fri, 24 May 2013 12:13:17 +0530
Message-ID: 
 <CAOcnVr2PR-iFsYvJFtKxz0t5F6o6c6KddHS+tdWi_tZ7FokhwQ@mail.gmail.com>
Subject: Re: HTTP file server, map output, and other files
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

YARN has a ShuffleHandler plugin used for MR purposes, but the APIs
used here aren't "general"/public so you'd have to build your own
utilities to do that. Its not too difficult to achieve but a general
API would certainly be nice.

Tez (Incubating) aims to solve some of this for users writing YARN
apps in a general way, but it isn't consumable yet. You can follow Tez
on the Apache Incubator at
http://incubator.apache.org/projects/tez.html.

P.s. As mentioned, YARN-based MR2 does not use HTTP (Jetty) anymore.
It uses Netty.

On Fri, May 24, 2013 at 3:14 AM, John Lilley <john.lilley@redpoint.net> wro=
te:
> Thanks to previous kind answers and more reading in the elephant book, I =
now
> understand that mapper tasks place partitioned results into local files t=
hat
> are served up to reducers via HTTP:
>
>
>
> =93The output file=92s partitions are made available to the reducers over=
 HTTP.
> The maximum number of worker threads used to serve the file partitions is
> controlled by the tasktracker.http.threads property; this setting is per
> tasktracker, not per map task slot. The default of 40 may need to be
> increased for large clusters running large jobs. In MapReduce 2, this
> property is not applicable because the maximum number of threads used is =
set
> automatically based on the number of processors on the machine. (MapReduc=
e 2
> uses Netty, which by default allows up to twice as many threads as there =
are
> processors.)=94
>
>
>
> My question is, for a custom (non-MR) application under YARN, how would I
> set up my application tasks=92 output data to be served over HTTP?  Is th=
ere
> an API to control this, or are there predefined local folders that will b=
e
> served up?  Once I am finished with the temporary data, how do I request
> that the files are removed?
>
>
>
> Thanks
>
> John
>
>


--
Harsh J