hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-14840) Tool to estimate resource requirements of an application pipeline based on prior executions
Date Thu, 26 Oct 2017 01:23:03 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-14840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16219818#comment-16219818
] 

Hudson commented on HADOOP-14840:
---------------------------------

FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #13138 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13138/])
HADOOP-14840. Tool to estimate resource requirements of an application (subru: rev 625039ef20e6011ab360131d70582a6e4bf2ec1d)
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/solver/impl/TestSolver.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/validator/SkylineStoreValidator.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/estimator.sh
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/RecurrenceIdNotFoundException.java
* (add) hadoop-tools/hadoop-resourceestimator/README.md
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/rmLog.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/SkylineStoreException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/exceptions/DataFieldNotFoundException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/SolverException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/site/resources/css/site.css
* (edit) hadoop-tools/pom.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/exception/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/InvalidSolverException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/exceptions/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog8.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/EmptyResourceSkylineException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog3.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/exception/ResourceEstimatorException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/LogParser.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/RmSingleLineParser.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/stop-estimator.cmd
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/validator/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/SkylineStore.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/SingleLineParser.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/validator/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/api/Solver.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullRecurrenceIdException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/resourceestimator-config.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog5.txt
* (edit) hadoop-assemblies/src/main/resources/assemblies/hadoop-tools.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/ResourceSerDe.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ShutdownHook.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestResourceSkylineSerDe.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/validator/ParserValidator.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/api/TestResourceSkyline.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog6.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/service/TestResourceEstimatorService.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/NativeSingleLineParser.java
* (add) hadoop-tools/hadoop-resourceestimator/pom.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullRLESparseResourceAllocationException.java
* (edit) hadoop-tools/hadoop-tools-dist/pom.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/api/JobMetaData.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/start-estimator.sh
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/answer.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/log4j.properties
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/PredictionSkylineStore.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/preprocess/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestHistorySkylineSerDe.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/LogParserUtil.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/api/TestJobMetaData.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/package-info.java
* (add) hadoop-assemblies/src/main/resources/assemblies/hadoop-resourceestimator.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/tpch_q12.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/data/resourceEstimatorService.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/start-estimator.cmd
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/skylinestore/impl/TestInMemoryStore.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/impl/InMemoryStore.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/RecurrenceId.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/api/HistorySkylineStore.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullPipelineIdException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/resourceEstimatorService.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/DuplicateRecurrenceIdException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/serialization/RLESparseResourceAllocationSerDe.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/ResourceEstimatorConfiguration.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ResourceEstimatorServer.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/LpSolver.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog7.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/resources/webapps/ResourceEstimatorServer/.gitignore
* (edit) hadoop-project/src/site/site.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/impl/TestNativeParser.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/lp/tinySample.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/api/package-info.java
* (edit) LICENSE.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog2.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/NullResourceSkylineException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/stop-estimator.sh
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/nativeLog.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/translator/impl/TestRmParser.java
* (edit) hadoop-project/pom.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/ResourceSkyline.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog1.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/service/GuiceServletConfig.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/translator/impl/BaseLogParser.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog4.txt
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/config/ResourceEstimatorUtil.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/conf/resourceestimator-config.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/preprocess/SolverPreprocessor.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/skylinestore/impl/TestSkylineStore.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/common/api/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/exceptions/InvalidInputException.java
* (add) hadoop-tools/hadoop-resourceestimator/src/site/markdown/ResourceEstimator.md
* (add) hadoop-tools/hadoop-resourceestimator/src/main/bin/estimator.cmd
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/impl/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/common/serialization/TestResourceSerDe.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/skylinestore/exceptions/package-info.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/service/ResourceEstimatorService.java
* (add) hadoop-tools/hadoop-resourceestimator/src/main/java/org/apache/hadoop/resourceestimator/solver/impl/BaseSolver.java
* (add) hadoop-tools/hadoop-resourceestimator/src/test/java/org/apache/hadoop/resourceestimator/solver/impl/TestLpSolver.java
* (add) hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml
* (add) hadoop-tools/hadoop-resourceestimator/src/test/resources/trace/invalidLog9.txt


> Tool to estimate resource requirements of an application pipeline based on prior executions
> -------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-14840
>                 URL: https://issues.apache.org/jira/browse/HADOOP-14840
>             Project: Hadoop Common
>          Issue Type: New Feature
>          Components: tools
>            Reporter: Subru Krishnan
>            Assignee: Rui Li
>            Priority: Blocker
>             Fix For: 2.9.0, 3.0.0, 3.1.0
>
>         Attachments: HADOOP-14840-v1.patch, HADOOP-14840-v2.patch, HADOOP-14840-v3.patch,
HADOOP-14840-v4.patch, ResourceEstimator-design-v1.pdf
>
>
> We have been working on providing SLAs for job execution on Hadoop. At high level this
involves 2 parts: deriving the resource requirements of a job and guaranteeing the estimated
resources at runtime. The {{YARN ReservationSystem}} (YARN-1051/YARN-2572/YARN-5326) enable
the latter and in this JIRA, we propose to add a tool to Hadoop to predict the  resource requirements
of a job based on past executions of the job. The system (aka *Morpheus*) deep dive can be
found in our OSDI'16 paper [here|https://www.usenix.org/conference/osdi16/technical-sessions/presentation/jyothi].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message