hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Patrick Wendell (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
Date Tue, 17 Apr 2012 07:48:51 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13255396#comment-13255396
] 

Patrick Wendell commented on MAPREDUCE-3451:
--------------------------------------------

Thanks for the feedback. About to attach a patch with the following changes:

- Updated the terminology from “pool” to “queue” to be consistent. 
- I added annotations to all classes.
- Fixed the debug → log statement
- I modified the web UI to give a more precise error message in the logs, but it still has
a generic 500 in the web page. MRv2 uses a custom built MVC framework based on Guice dependency
injection. I’m embarrassed to say that I spend 40 minutes trying to figure out how to make
it render a 404 and couldn’t. It seems like it’s not easy to trigger a 404 from within
a view, rather than at the URL router. 
- Extra license header removed
- Noted spelling errors fixed.
- There is extensive test coverage in the MRS FS I’d like to get there eventually but probably
not with this first commit. I tried to included tests of most of the core functionality. We
can add this as a JIRA, though.
- I consolidated the configuration constants as you suggested. For now I’m going to leave
the queue manifest file as its own thing unless this is a deal-breaker. There is a lot of
logic in parsing that file which I’d prefer to leave as-is since it’s very well tested.
While the term names have changed it’s essentially identical configuration to the MR1 FS,
so leaving things as-is should ease the transition for people running current clusters. It’s
also a fundamentally hierarchical type of configuration, and the apache conf is all flat key/value
space which will be much more clunky.

I haven’t done much testing other than on a single node. One reason I want to get this in
is that some are waiting on having something in trunk to start testing. The build is stable
and the unit tests pass, so I hope that is sufficient to get this in there with the caveat
that things may be brittle when run on a real cluster.
 
I also added an updated version of the documentation (even the MR1 version wasn’t totally
up-to-date with changes that have been made in the configuration options). This will need
some more work going forward but it should give people to the basic configuration/installation
info they need to get started testing this.
                
> Port Fair Scheduler to MR2
> --------------------------
>
>                 Key: MAPREDUCE-3451
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2, scheduler
>            Reporter: Patrick Wendell
>            Assignee: Patrick Wendell
>         Attachments: MAPREDUCE-3451.v1.patch.txt, MAPREDUCE-3451.v2.patch.txt
>
>
> The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to
MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include
design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message