hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <o...@yahoo-inc.com>
Subject Re: Creating splits/tasks at the client
Date Fri, 29 Sep 2006 15:49:23 GMT

On Sep 29, 2006, at 12:20 AM, Benjamin Reed wrote:

> I please correct me if I'm reading the code incorrectly, but it seems
> like submitJob puts the submitted job on the jobInitQueue which is
> immediately dequeued by the JobInitThread and then initTasks() will  
> get
> the file splits and create Tasks. Thus, it doesn't seem like there is
> any difference in memory foot print.

Agreed, it won't cost more memory. In fact, it will be less because  
we won't have the init task thread running and creating InputFormats  
and running user code. Of course, once we allow user-defined  
InputSplits we will be back in exactly the same boat of running user- 
code on the JobTracker, unless we also ship over the preferred hosts  
for each InputFormat too.

-- Owen

View raw message