hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benjamin Reed <br...@yahoo-inc.com>
Subject Creating splits/tasks at the client
Date Thu, 28 Sep 2006 08:04:30 GMT
One of the things that bothers me about the JobTracker is that it is
running user code when it creates the FileSplits. In the long term this
puts the JobTracker JVM at risk due to errors in the user code.

The JobTracker uses the InputFormat to create a set of tasks that it
then schedules. The task creation does not need to happen at the
JobTracker. If we allowed the clients to create the set of tasks, the
JobTracker would not need to load and run any user generated code. It
would also remove some of the processing load from the JobTracker. On
the downside it does greatly increase the amount of information sent to
the JobTracker when a job is submitted.


View raw message