Mailing-List: contact common-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of dnquark@gmail.com designates
 209.85.214.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=pU1l8s3ezN9/Yc03ytDX4xH7VKz6btR2UdPWrFshMFmZTKt7tr9DsbHJq6aNVTncmu
         2ShPQNPfb3shUrQZp7EEEst5c5nbcD1gCt23cuyXiKNciupn5UhYvhd2j4mb+BYKXGsM
         ZJ4uteftEsp888qPhJT5mighKg1Yop0oKXYKU=
MIME-Version: 1.0
In-Reply-To: <AANLkTikpiCEM3DFYTwHoVsBtihLwuuuV20ysmG8WnF4n@mail.gmail.com>
References: <AANLkTilg6fFVyTKDwEjuNexmV_LNOyGqY7ue2cH_jzWN@mail.gmail.com>
	<AANLkTikpiCEM3DFYTwHoVsBtihLwuuuV20ysmG8WnF4n@mail.gmail.com>
Date: Thu, 22 Jul 2010 13:39:43 -0700
Message-ID: <AANLkTimt4BQDyOIg2ZIRJaZHxdOx1-3KK0NXuFW3Id_z@mail.gmail.com>
Subject: Re: Is it possible to use NullWritable in combiner? + general
	question about combining output from many small maps
From: Leo Alekseyev <dnquark@gmail.com>
To: common-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Thanks for everybody's responses.  I think I've got things sorted out
for the time being; some folks were asking me for clarification of my
problem, so let me elaborate, for the list archives if nothing else.
In brief, say I have 1000
mappers each outputting 20 MB chunks; my problem doesn't require a
reduce step, but I'm not happy with the file being partitioned in many
small chunks smaller than DFS block size.  But when I tried running
e.g. 4 reducers so that I get 4 5 GB files at the end, the reduce step
took quite a bit longer than the map step, most of it being network
traffic during the shuffle step.  It seemed wasteful to me, in light
of the fact that the reducers' only purpose was to "glue together" the
small files.

I am guessing that there's no way to
get around this -- when using reducers you'll have to be sending
chunks to machines that combine them.  It is possible, however, to
tweak the size of each map's output (and thus the number of mappers)
by adjusting min split input size; for some of my jobs it's proving to
be a good solution

--Leo

On Wed, Jul 21, 2010 at 2:57 AM, Himanshu Vashishtha
<vashishtha.h@gmail.com> wrote:
> Please see my comments in-line, as per my understanding of Hadoop & your
> problems. See if they are helpful.
>
> Cheers,
> Himanshu
>
> On Wed, Jul 21, 2010 at 2:59 AM, Leo Alekseyev <dnquark@gmail.com> wrote:
>
>> Hi All,
>> I have a job where all processing is done by the mappers, but each
>> mapper produces a small file, which I want to combine into 3-4 large
>> ones. =A0In addition, I only care about the values, not the keys, so
>> NullWritable key is in order. =A0I tried using the default reducer
>> (which according to the docs is identity) by setting
>> job.setReducerClass(org.apache.hadoop.mapreduce.Reducer.class) and
>> using a NullWritable key on the mapper output. =A0However, this seems to
>> concentrate the work on one reducer only.
>
>
> NullWritable is a singleton class. So, the entire map output related to i=
t
> will go to a single reduce node.
>
>
>> I then tried to output
>> LongWritable as the mapper key, and write a combiner to output
>> NullWritable (i.e. class GenerateLogLineProtoCombiner extends
>> Reducer<LongWritable, ProtobufLineMsgWritable, NullWritable,
>> ProtobufLineMsgWritable>); still using the default reducer. =A0This gave
>> me the following error thrown by the combiner:
>>
>> 10/07/21 01:21:38 INFO mapred.JobClient: Task Id :
>> attempt_201007122205_1058_m_000104_2, Status : FAILED
>> java.io.IOException: wrong key class: class
>> org.apache.hadoop.io.NullWritable is not class
>> org.apache.hadoop.io.LongWritable
>> =A0 =A0 =A0 =A0at org.apache.hadoop.mapred.IFile$Writer.append(IFile.jav=
a:164)
>> =A0 =A0 =A0 =A0 =A0.........
>>
>> A combiner goal is to lessen the =A0reducer's workload. Ideally, its out=
put
> key-value should be same as that of Mapper's output key-value. Therefore =
the
> error.
>
>> I was able to get things working by explicitly putting in an identity
>> reducer that takes (LongWritable key, value) and outputs
>> (NullWritable, value). =A0However, now most of my processing is in the
>> reduce phase, which seems like a waste -- it's copying and sorting
>> data, but all I really need is to "glue" together the small map
>> outputs.
>>
>> Thus, my questions are: I don't really understand why the combiner is
>> throwing an error here. =A0Does it simply not allow NullWritables on the
>> output?...
>> The second question is -- is there a standard strategy for quickly
>> combining the many small map outputs? =A0Is it worth, perhaps, to look
>> into adjusting the min split size for the mappers?.. (can this value
>> be adjusted dynamically based on the input file size?..)
>>
>> I don't know of any such strategy. How about defining a smaller number o=
f
> reducers. I am also not able to understand teh problem. It will be great =
if
> you are a bit more specific (in terms of map file input and output size, =
and
> reduce output size).
>
>
>> Thanks to anyone who can give me some pointers :)
>> --Leo
>>
>