hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <cutt...@apache.org>
Subject Re: InputFiles, Splits, Maps, Tasks Questions 1.3 Base
Date Thu, 18 Oct 2007 22:15:39 GMT
Lance Amundsen wrote:
> Example: let's say I have 10K one second jobs and I want the whole thing to
> run 2 seconds.  I currently see no way for Hadoop to achieve this,

That's right.  That has not been a design goal to date.  Tasks are 
typically expected to last at least several seconds.  To fix this we'd 
need to permit a single JVM to handle a sequence of tasks, which would 
be possible.  Currently tasks are always run in a separate JVM since 
they load user code and may fail in arbitrary ways.  So if someone 
wishes to implement this, it should probably be optional.


View raw message