hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Douglas <cdoug...@apache.org>
Subject Re: question about org.apache.hadoop.mapred.join
Date Tue, 10 Apr 2012 17:33:53 GMT
Your understanding is correct. The framework doesn't do anything to
align input splits across datasets. In the situation you describe-
where one can't seek among key groups in the input data- it often
makes sense to disable splitting of the individual files by setting
the min split size to Integer.MAX_VALUE.

The description probably shouldn't use "partitioned", since that
implies that the partitioner is sufficient. -C

On Tue, Apr 10, 2012 at 8:11 AM, Koert Kuipers <koert@tresata.com> wrote:
> I read about CompositeInputFormat and how it allows one to join two datasets
> together as long as those datasets were sorted and partitioned the same way.
> Ok i think i get it, but something bothers me. It is suggested that two
> datasets are "sorted and partitioned the same way" if they were both outputs
> from the mapreduce process with the same number of reducers with the same
> sorting & partitioning. However, something like CompositeInputFormat depends
> on the splits lining up, and two datasets going through the same reducer
> setup doesn't guarantee that at all. Splits after all are based on stuff
> like data size in MBs, and the reducers do not control that this will be the
> same. part-00007 for dataset 1 could be a different size (and have different
> number of splits) than part-00007 for dataset 2, even if they have the same
> keys and are sorted the same way. So now CompositeInputFormat would not
> work. Is this correct?

View raw message