Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of jwills@cloudera.com
 designates 209.85.216.171 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <BLU0-SMTP3549C2C1110091D51223CDDAD980@phx.gbl>
References: <BLU0-SMTP3549C2C1110091D51223CDDAD980@phx.gbl>
From: Josh Wills <jwills@cloudera.com>
Date: Thu, 6 Jun 2013 05:04:56 -0700
Message-ID: 
 <CAH29n6Muz3b59NKaSgJ4Gt=iU_1E3xNQHW2O5rb2e2njKY8jUQ@mail.gmail.com>
Subject: Re: Small files produced by a map-only job
To: user@crunch.apache.org
Content-Type: multipart/alternative; boundary=e89a8f6469abb023aa04de7b2086

--e89a8f6469abb023aa04de7b2086
Content-Type: text/plain; charset=ISO-8859-1

Hey Chao,

It had dropped off my radar, but I'm happy to throw together a patch to do
it this AM.

J


On Thu, Jun 6, 2013 at 4:06 AM, Chao Shi <stepinto@live.com> wrote:

> Hey guys,
>
> I'm writing MR jobs using crunch. Crunch optimizes some very simple
> pipeline into map-only jobs, e.g. sample or grep.
>
> As MR framework splits the input data by HDFS block, the map phase will
> produce plenty of small files on HDFS, which is unpleasant and makes the
> following data processing inefficient. When I write raw MR, I typically
> append this with an identity reducer, which simply merges outputs from map.
>
> I think CRUNCH-162 <https://issues.apache.org/jira/browse/CRUNCH-162> is
> related to this. Is there anyone still working on it?
>
> Thanks,
> Chao
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

--e89a8f6469abb023aa04de7b2086
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey Chao,<div><br></div><div style>It had dropped off my r=
adar, but I&#39;m happy to throw together a patch to do it this AM.</div><d=
iv style><br></div><div style>J</div><div style>=A0 =A0 =A0=A0</div></div><=
div class=3D"gmail_extra">

<br><br><div class=3D"gmail_quote">On Thu, Jun 6, 2013 at 4:06 AM, Chao Shi=
 <span dir=3D"ltr">&lt;<a href=3D"mailto:stepinto@live.com" target=3D"_blan=
k">stepinto@live.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_qu=
ote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex=
">

<div dir=3D"ltr">Hey guys,<div><br></div><div>I&#39;m writing MR jobs using=
 crunch. Crunch optimizes some very simple pipeline into map-only jobs, e.g=
. sample or grep.</div><div><br></div><div>As MR framework splits the input=
 data by HDFS block, the map phase will produce plenty of small files on HD=
FS, which is unpleasant and makes the following data processing inefficient=
. When I write raw MR, I typically append this with an identity reducer, wh=
ich simply merges outputs from map.</div>


<div><br></div><div>I think <a href=3D"https://issues.apache.org/jira/brows=
e/CRUNCH-162" target=3D"_blank">CRUNCH-162</a>=A0is related to this. Is the=
re anyone still working on it?</div><div><br></div><div>Thanks,<br>Chao</di=
v>


</div>
</blockquote></div><br><br clear=3D"all"><div><br></div>-- <br><div>Directo=
r of Data Science</div><div><a href=3D"http://www.cloudera.com" target=3D"_=
blank">Cloudera</a></div><div>Twitter: <a href=3D"http://twitter.com/josh_w=
ills" target=3D"_blank">@josh_wills</a></div>


</div>

--e89a8f6469abb023aa04de7b2086--