Mailing-List: contact crunch-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-user@incubator.apache.org
Received-SPF: pass (athena.apache.org: domain of gabriel.reid@gmail.com
 designates 209.85.212.180 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BLU0-SMTP1025B0E41417FA7A978C833AD070@phx.gbl>
References: <BLU0-SMTP1025B0E41417FA7A978C833AD070@phx.gbl>
Date: Wed, 6 Feb 2013 09:04:21 +0100
Message-ID: 
 <CAA5C_ptPXmDsTU0qp=okNGoEGbXbQhJsQRG45iguoCDnRPE37A@mail.gmail.com>
Subject: Re: Deal with CPU intensive tasks
From: Gabriel Reid <gabriel.reid@gmail.com>
To: crunch-user@incubator.apache.org
Content-Type: multipart/alternative; boundary=089e013d163410bd4004d509c62b

--089e013d163410bd4004d509c62b
Content-Type: text/plain; charset=ISO-8859-1

Hi Chao,

There's currently no way of marking a particular part of the pipeline as
being CPU intensive -- however, what you can do is force a slightly
different execution plan by calling "materialize.iterator()" on the
PCollection containing the results of the "FirstPass" parallelDo. This will
force Crunch to run the pipeline up to that point and serialize the
"FirstPass" data, and then use the serialized collection for future
processing instead of rebuilding it.

The plan for the future is to include functionality like this in the API
(which could also possibly run somewhat more efficiently by not immediately
running the pipeline at such a point), but for now the materialize hack is
the easiest way to achieve this.

- Gabriel


On Wed, Feb 6, 2013 at 5:57 AM, Chao Shi <stepinto@live.com> wrote:

> Hi crunch users,
>
> The execution plan of my pipeline is attached with this mail. The
> ParallelDo "FirstPass" (at the top of the graph) is highly CPU intensive,
> which needs to call parsers to build ASTs from source code. The best plan I
> can imagine for my case is to have a map-only job in the front and have the
> following 3 MRs read its output.
>
> I wonder if there's a way to mark my ParallelDo as CPU intensive, so that
> crunch only create a single instane  of it.
>
> Thanks,
> Chao
>

--089e013d163410bd4004d509c62b
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Hi Chao,<div><br></div><div>There&#39;s currently no way of marking a parti=
cular part of the pipeline as being CPU intensive -- however, what you can =
do is force a slightly different execution plan by calling &quot;materializ=
e.iterator()&quot; on the PCollection containing the results of the &quot;F=
irstPass&quot; parallelDo. This will force Crunch to run the pipeline up to=
 that point and serialize the &quot;FirstPass&quot; data, and then use the =
serialized collection for future processing instead of rebuilding it.</div>
<div><br></div><div>The plan for the future is to include functionality lik=
e this in the API (which could also possibly run somewhat more efficiently =
by not immediately running the pipeline at such a point), but for now the m=
aterialize hack is the easiest way to achieve this.</div>
<div><br></div><div>- Gabriel</div><div><br><br><div class=3D"gmail_quote">=
On Wed, Feb 6, 2013 at 5:57 AM, Chao Shi <span dir=3D"ltr">&lt;<a href=3D"m=
ailto:stepinto@live.com" target=3D"_blank">stepinto@live.com</a>&gt;</span>=
 wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi crunch users,<div><br></div><div>The exec=
ution plan of my pipeline is attached with this mail. The ParallelDo &quot;=
FirstPass&quot; (at the top of the graph) is highly CPU intensive, which ne=
eds to call parsers to build ASTs from source code. The best plan I can ima=
gine for my case is to have a map-only job in the front and have the follow=
ing 3 MRs read its output.</div>


<div><br></div><div>I wonder if there&#39;s a way to mark my ParallelDo as =
CPU intensive, so that crunch only create a single instane =A0of it.</div><=
div><br></div><div>Thanks,</div><div>Chao</div>
</blockquote></div><br></div>

--089e013d163410bd4004d509c62b--