Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of Jan.Lukavsky@firma.seznam.cz
 designates 77.75.74.246 as permitted sender)
Message-ID: <5228409A.5040801@firma.seznam.cz>
Date: Thu, 5 Sep 2013 10:28:10 +0200
From: =?ISO-8859-1?Q?Jan_Lukavsk=FD?= <jan.lukavsky@firma.seznam.cz>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130803 Thunderbird/17.0.8
MIME-Version: 1.0
To: <user@hadoop.apache.org>
Subject: Re: M/R API and Writable semantics in reducer
References: <522492C4.80000@firma.seznam.cz>
In-Reply-To: <522492C4.80000@firma.seznam.cz>
Content-Type: text/plain; charset="ISO-8859-1"; format=flowed
Content-Transfer-Encoding: 8bit

Hi,

is there anyone interested in this topic? Basically, what I'm trying to 
find out is, whether it is 'safe' to rely on the side-effect of updating 
key during iterating values. I believe that there must be someone who is 
also interested in this, the secondary sort pattern is very common (at 
least in our jobs). So far, we have been emulating the 
GroupingComparator by holding state in the Reducer class and therefore 
being able to keep track of 'groups' of keys among several calls to 
reduce() method. This method seems quite safe in the sense of API, but 
in the sense of code is not as pretty (and vulnerable to ugly bugs if 
you forget to reset the state correctly for instance).

On the other hand, if the way key gets updated while iterating the 
values is to be considered contract of the MapReduce API, I think it 
should be implemented in MRUnit (or you basically cannot use MRUnit to 
unittest your job) and if it isn't, than it is probably a bug. If this 
is internal behavior and might be subject to change anytime, than it 
clearly seems that keeping the state in Reducer is the only option.

Does anyone else have similar considerations? How do others implement 
the secondary sort?

Thanks,
  Jan

On 09/02/2013 03:29 PM, Jan Lukavsk� wrote:
> Hi all,
>
> some time ago, I wrote a note to this conference, that it would be 
> nice if it would be possible to get the *real* key emitted from mapper 
> to reducer, when using the GroupingComparator. I got the answer, that 
> it is possible, because of the Writable semantics and that currently 
> the following holds:
>
> @Override
> protected void reduce(Key key, Iterable<Value> values, Context context)
> {
>   for (Value v : values) {
>     // The key MIGHT change its value in this cycle, because 
> readFields() will be called on it.
>     // When using GroupingComparator that groups only by some part of 
> the key,
>     // many different keys might be considered single group, so the 
> *real* data matters.
>   }
> }
>
> When you use GroupingComparator the contents of the key can matter, 
> because if you cannot access it, you have to duplicate the data in 
> value (which means more network traffic in shuffle phase, and more I/O 
> generally).
>
> Now, the question is, how much is this a matter of API that is 
> reliable, or how much it is likely, that relying on this feature might 
> break in future versions. To me, it seems more like a side effect, 
> that is not guaranteed to be maintained in the future. There already 
> exists a suggestion, that this is probably very fragile, because 
> MRUnit seems not to update the key during the iteration.
>
> Does anyone have any suggested way around? Is the 'official' preferred 
> way of accessing the original key to call context.getCurrentKey()? 
> Isn't this the same case? Wouldn't it be nice, if the API itself had 
> some guaranties or suggestions how it works? I can imagine modified 
> reduce() metod, with a signature like
>
> protected void reduce(Key key, Iterable<Pair<Key, Value>> keyValues, 
> Context context);
>
> This seems easily transformable to the old call (which could be 
> default implementation of this method).
>
> Any opinion on this?
>
> Thanks,
>  Jan