hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Subru Krishnan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14840) Tool to estimate resource requirements of an application pipeline based on prior executions
Date Tue, 05 Sep 2017 22:53:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16154517#comment-16154517

Subru Krishnan commented on HADOOP-14840:

[~vrushalic], thanks for your comments, please find my response inline.

bq. This sounds interesting. Will there be subtasks under this? 

We have developed this tool and are using it internally. So it has already been through rigorous
reviews etc so was just planning on a single uber patch to save time/efficiency. We can break
it up if really required.

bq. Similar estimations for MR jobs like reducer estimation & memory estimation are done
based on past runs using a tool called hRaven (https://github.com/twitter/hraven)

We are aware of hRaven. In fact, we have a similar work for Hive and Tez called [PerfOrator|http://dl.acm.org/citation.cfm?id=2987566].
With the ever growing frameworks (and optimizations/updates to existing ones), we decided
to follow a different approach here:
* Be framework agnostic, i.e. we model only resources over time and so are not limited to
map or reduce (or Tez or Reef). We are essentially emulating the YARN model natively.
* Work purely based on history (can be direct hook to hRaven). We have developed a Linear
Programming model to estimate the resources required for the job's future execution.
* Optionally reserve the resources using YARN's ReservationSystem.

 The gory details are available in the [Morpheus paper|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].

bq. I would think of this as a natural extension to Timeline Service v2. Do you think you
might want to make use of TSv2 in any way for this?

We can't agree more :). The initial version we have works on YARN RM logs only because that's
what is widely available (in prod deployments). We have designed it with the clear intent
to integrate with TSv2 (or even hRaven) in future, with the community help, of course. 

We will upload a design doc shortly.

> Tool to estimate resource requirements of an application pipeline based on prior executions
> -------------------------------------------------------------------------------------------
>                 Key: HADOOP-14840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14840
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools
>            Reporter: Subru Krishnan
>            Assignee: Rui Li
> We have been working on providing SLAs for job execution on Hadoop. At high level this
involves 2 parts: deriving the resource requirements of a job and guaranteeing the estimated
resources at runtime. The {{YARN ReservationSystem}} (YARN-1051/YARN-2572/YARN-5326) enable
the latter and in this JIRA, we propose to add a tool to Hadoop to predict the  resource requirements
of a job based on past executions of the job. The system (aka *Morpheus*) deep dive can be
found in our OSDI'16 paper [here|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message