Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-dev@lucene.apache.org
Received-SPF: neutral (herse.apache.org: local policy)
Mime-Version: 1.0 (Apple Message framework v752.2)
In-Reply-To: <b23e01b50704080148n7373527csccdd41dd17fdf37c@mail.gmail.com>
References: <C22FFA40.154B%michael@powerset.com>
 <460AB81D.7040708@yahoo-inc.com>
 <b23e01b50704080148n7373527csccdd41dd17fdf37c@mail.gmail.com>
Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed
Message-Id: <FC9DDDD0-ABC2-45BC-A240-29E5A4B37E14@yahoo-inc.com>
Content-Transfer-Encoding: 7bit
From: Owen O'Malley <owen@yahoo-inc.com>
Subject: Re: Java 1.5?
Date: Mon, 9 Apr 2007 11:11:58 -0700
To: hadoop-dev@lucene.apache.org


On Apr 8, 2007, at 1:48 AM, Tom White wrote:

> I think we can do a lot to improve the use of generics, particularly
> in MapReduce.
> <... use generics in interfaces ...>

I like it. I was thrown off at first because classes aren't  
specialized based on their template parameters, but specialization of  
the parent class is available.

> Reducer would be changed similarly, although I'm not sure how we could
> constrain the output types of the Mapper to be the input types of the
> Reducer. Perhaps via the JobConf?

That is easy, actually. In the JobClient, we'd just check to see if  
the types all play well together. Basically, you need:
K1,V1 -> map -> K2, V2
K2, V2 -> combiner -> K2, V2 (if used)
K2, V2 -> reduce -> K3, V3

It will be a tricky bit of specification to decide exactly what the  
right semantics are, since even with the generics, the application  
isn't required to define them. Therefore, we have 5 places where we  
could find a value for K2 (config, mapper output, combiner input,  
combiner output, or reduce input). Clearly all classes must be  
checked for consistency once Hadoop decides what the right values are  
for each type.

The other piece that this interacts with is the desire to use context  
objects in the parameter list. However, they appear to be orthogonal  
to each other.

-- Owen