hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1220) Implement an in-cluster LocalJobRunner
Date Sat, 21 Nov 2009 07:31:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12780926#action_12780926

Aaron Kimball commented on MAPREDUCE-1220:


Thank you for explaining your use cases above more thoroughly. Given the nature of clusters
with submission nodes, etc., this does seem like it would be valuable functionality. As such,
it can/should be added. +1 on the direction.

Maybe I was jumping the gun on reading your description; you mentioned the LocalJobRunner,
so I thought you were off-the-bat proposing to build another *JobRunner. Since we already
have three job runners (the usual one, the local one, and the isolation runner), all of which
have their own quirks, idiosyncrasies and bugs, I would be nervous about yet another one of
these which will have its own slightly deviant semantics, and hope that we could reuse a lot
of the existing task deployment code. 

Some questions about the proposal itself:
* Will users be able to set a number of reduce tasks greater than one? Or are you limited
to at most one reduce task?
* How does this work with speculative execution, if at all?
* Are all InputSplits associated with the task delivered all-at-once from the JobTracker?
Or does this "super-task" for the job fetch them one-at-a-time? For that matter, will this
be represented in the JobTracker as multiple InputSplits, or just one?
** How is InputSplit locality information regarded? If we want to map over (for example) three
small files, and FileInputFormat enumerates three InputSplits, each of which have different
locality hints, how does the JobTracker pick where to schedule this job? 
* How do these sorts of jobs interact with the resource allocation algorithms used by the
Fair/Capacity Schedulers?

- Aaron

> Implement an in-cluster LocalJobRunner
> --------------------------------------
>                 Key: MAPREDUCE-1220
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1220
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: client, jobtracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.22.0
> Currently very small map-reduce jobs suffer from latency issues due to overheads in Hadoop
Map-Reduce such as scheduling, jvm startup etc. We've periodically tried to optimize all parts
of framework to achieve lower latencies.
> I'd like to turn the problem around a little bit. I propose we allow very small jobs
to run as a single task job with multiple maps and reduces i.e. similar to our current implementation
of the LocalJobRunner. Thus, under certain conditions (maybe user-set configuration, or if
input data is small i.e. less a DFS blocksize) we could launch a special task which will run
all maps in a serial manner, followed by the reduces. This would really help small jobs achieve
significantly smaller latencies, thanks to lesser scheduling overhead, jvm startup, lack of
shuffle over the network etc. 
> This would be a huge benefit, especially on large clusters, to small Hive/Pig queries.
> Thoughts?

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message