Mailing-List: contact user-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of stepinto@live.com designates
 65.55.116.39 as permitted sender)
Message-ID: <BLU0-SMTP3549C2C1110091D51223CDDAD980@phx.gbl>
MIME-Version: 1.0
From: Chao Shi <stepinto@live.com>
Date: Thu, 6 Jun 2013 19:06:24 +0800
Subject: Small files produced by a map-only job
To: crunch-user@apache.org
Content-Type: multipart/alternative; boundary="047d7ba972724516ff04de7a4fc3"

--047d7ba972724516ff04de7a4fc3
Content-Type: text/plain; charset="ISO-8859-1"

Hey guys,

I'm writing MR jobs using crunch. Crunch optimizes some very simple
pipeline into map-only jobs, e.g. sample or grep.

As MR framework splits the input data by HDFS block, the map phase will
produce plenty of small files on HDFS, which is unpleasant and makes the
following data processing inefficient. When I write raw MR, I typically
append this with an identity reducer, which simply merges outputs from map.

I think CRUNCH-162 <https://issues.apache.org/jira/browse/CRUNCH-162> is
related to this. Is there anyone still working on it?

Thanks,
Chao

--047d7ba972724516ff04de7a4fc3
Content-Type: text/html; charset="ISO-8859-1"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hey guys,<div><br></div><div style>I&#39;m writing MR jobs=
 using crunch. Crunch optimizes some very simple pipeline into map-only job=
s, e.g. sample or grep.</div><div style><br></div><div style>As MR framewor=
k splits the input data by HDFS block, the map phase will produce plenty of=
 small files on HDFS, which is unpleasant and makes the following data proc=
essing inefficient. When I write raw MR, I typically append this with an id=
entity reducer, which simply merges outputs from map.</div>

<div style><br></div><div style>I think <a href=3D"https://issues.apache.or=
g/jira/browse/CRUNCH-162">CRUNCH-162</a>=A0is related to this. Is there any=
one still working on it?</div><div style><br></div><div style>Thanks,<br>Ch=
ao</div>

</div>

--047d7ba972724516ff04de7a4fc3--