Lance Amundsen wrote:
> Example: let's say I have 10K one second jobs and I want the whole thing to
> run 2 seconds. I currently see no way for Hadoop to achieve this,
That's right. That has not been a design goal to date. Tasks are
typically expected to last at least several seconds. To fix this we'd
need to permit a single JVM to handle a sequence of tasks, which would
be possible. Currently tasks are always run in a separate JVM since
they load user code and may fail in arbitrary ways. So if someone
wishes to implement this, it should probably be optional.
Doug
|