hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-279) Map-Reduce 2.0
Date Tue, 22 Feb 2011 06:37:38 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12997654#comment-12997654

Joydeep Sen Sarma commented on MAPREDUCE-279:

I have been working on maintaining/enhancing MR for FB's use case for last 6 months or so.
Here are a few priority areas for us that are relevant to this discussion:

# Latency.

  This is, imho, #1 priority wrt scheduling. As Scott has already remarked, the ping-response
model is broken. So is preemption as an after-thought. We need to get small/medium jobs scheduled
instantly. Period.

# Scalability.

  We have made a number of vital fixes to keep the JT working at our scale - but we have merely
bought some time.

# (wrt. ResourceManager) Open API

  By which i mean something like Thrift/PB/Avro. We are, of course, most comfortable with
Thrift and it would be nice (but not critical) if it were possible to build a Thrift wrapper
(even if one was not baked in from scratch). 

  One thing i have found is that writing Thrift services is a breeze because of inbuilt service
framework. Everything else on the serialization side being equal - this has been a big win
for me personally as a developer (and something to be considered as other distributed execution
frameworks try to use the RM).

# Ability to back-plug into older Hadoop versions

  This is related to #3. Unlike many other organizations - we cannot make big jumps in hadoop
revisions anymore. We have too many custom changes and we don't have a QA department. Unlike
in the past, where we could have depended on Yahoo's QA'ed releases - we don't have that luxury
anymore (because we are now both running software at similar versions - and we can't wait
until Yahoo has deployed/QA'ed new versions before deploying newer upgrades).

  If the RM api is open (and satisfactory from design perspective) - we can do the work in-house
to our older version of Hadoop to use it. This is critical for us (although i am not sure
it applies to other users).

I cannot emphasize the urgency around #1. Whether we continue to use Hadoop or not is predicated
on big improvements in latency and efficiency (the latter is a different topic).

I hope #3 and #4 contribute to the discussion around component architecture. At our scale
- i don't think we can build services using large software that is tightly integrated. We
need too much customization and we can't afford the long upgrade cycles of such tightly integrated
software. Of course, this is specific to our deployment and the requirements of most other
deployments is likely to be quite different.


As a developer - i have found the current JobTracker code totally unmaintainable - I hope
the new version (broken across RM/App-Master) is better. There are several design points that
have struck me as particularly evil:

# synchronous RPC based architecture: limits concurrency and forces bad implementation choices
(see #2)
# crazy locking: this is just bad implementation for the most part - but i hope the new design/implementation
clearly articulates some principles around the fundamental data structures and how transactional
changes to these data structures are meant to be accomplished.
# poor data structure maintenance: 99% of the data structures in the JT have a pattern of
  a. a primary collection (eg: list of all jobs in the system)
  b. several secondary indices/views (list of all runnable jobs from above, list of all completed
jobs etc)
  Instead of modeling updates to such collections and related views through a common entry
point - updates to primary and secondary data structures are at disjoint places throughout
the code and make maintenance of code a nightmare.

i can only hope that a big rewrite like this will try to address some of these issues (others
- like hard-wiring to specific (M/R) task types - are already addressed i presume in the new

my 2c.

> Map-Reduce 2.0
> --------------
>                 Key: MAPREDUCE-279
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-279
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: jobtracker, tasktracker
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>             Fix For: 0.23.0
> Re-factor MapReduce into a generic resource scheduler and a per-job, user-defined component
that manages the application execution. 

This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message