hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlo Curino (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
Date Mon, 20 May 2013 20:27:16 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662330#comment-13662330
] 

Carlo Curino commented on MAPREDUCE-5176:
-----------------------------------------

Karthik, I think there are several other annotations we might want to think about. @Stateless
is one, another one is @PreserveKeyOrder which can express for example that maps are not messing
with sort order and would allow a smart runtime to pipeline maps and reduces (and skip shuffling)
when the input is sorted. I think this could be a powerful tool to expose opportunities for
runtime optimization which are not possible in the general case (unless you know something
about the code semantics). 

If none disagree, I like the idea to have this in a common place (maybe yarn-common as Karthik
suggested?). 

BTW I think we are in a great spot to carry along this conversation, since we have one very
specific example of these annotations: @Preemptable for which we have the entire end-to-end
usage scenario (all the preemption in mapreduce stuff tracked in MAPREDUCE-5189, MAPREDUCE-5192,
MAPREDUCE-5194, MAPREDUCE-5196, MAPREDUCE-5197 and few upcoming ones), and plenty more ideas
coming up from people. 

Ideally I would like to move forward with the @Preemptable one and see it through (so we can
evaluate it and learn from it), and in parallel we can initiate a broader (and rightfully
longer) conversation around annotations for runtime-optimization.

Sandy, what we envisioned for mapreduce is that and advanced user that have a stateful UDF
can mark it as @Preemptable and override the default save to checkpoint logic to include the
portion of state he/she cares about. 

                
> Preemptable annotations (to support preemption in MR)
> -----------------------------------------------------
>
>                 Key: MAPREDUCE-5176
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>            Reporter: Carlo Curino
>            Assignee: Carlo Curino
>         Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.patch
>
>
> Proposing a patch that introduces a new annotation @Preemptable that represents to the
framework property of user-supplied classes (e.g., Reducer, OutputCommiter). The intended
semantics is that a tagged class is safe to be preempted between invocations. 
> (this is in spirit similar to the Output Contracts of [Nephele/PACT | https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message