Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Wed, 28 May 2014 23:54:03 +0000 (UTC)
From: "Nishkam Ravi (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12713424.1399679681589.35649.1401321243010@arcas>
In-Reply-To: <JIRA.12713424.1399679681589@arcas>
References: <JIRA.12713424.1399679681589@arcas>
Subject: [jira] [Commented] (YARN-2041) Hard to co-locate MR2 and Spark jobs
 on the same cluster in YARN
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/YARN-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011844#comment-14011844 ] 

Nishkam Ravi commented on YARN-2041:
------------------------------------

Unlike FIFO, whose performance deteriorates consistently across multiple benchmarks as value of yarn.nodemanager.resource.memory-mb is increased from 16GB to 40GB, Capacity scheduler performs well for all benchmarks except for TeraValidate. 

For TeraValidate in single-job mode:

Exec. time with Fair: 38 sec (yarn.nodemanager.resource.memory-mb = 16GB)
Exec. time with Fair: 38 sec (yarn.nodemanager.resource.memory-mb = 40GB)
Exec. time with Capacity: 51 sec (yarn.nodemanager.resource.memory-mb = 16GB)
Exec. time with Capacity: 100 sec (yarn.nodemanager.resource.memory-mb = 40GB)

Also, in multi-job mode, Capacity seems to be behaving like FIFO. Scheduling one job at a time for execution. 


> Hard to co-locate MR2 and Spark jobs on the same cluster in YARN
> ----------------------------------------------------------------
>
>                 Key: YARN-2041
>                 URL: https://issues.apache.org/jira/browse/YARN-2041
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.3.0
>            Reporter: Nishkam Ravi
>
> Performance of MR2 jobs falls drastically as YARN config parameter yarn.nodemanager.resource.memory-mb  is increased beyond a certain value. 
> Performance of Spark falls drastically as the value of yarn.nodemanager.resource.memory-mb is decreased beyond a certain value for a large data set.
> This makes it hard to co-locate MR2 and Spark jobs in YARN.
> The experiments are being conducted on a 6-node cluster. The following workloads are being run: TeraGen, TeraSort, TeraValidate, WordCount, ShuffleText and PageRank.
> Will add more details to this JIRA over time.


--
This message was sent by Atlassian JIRA
(v6.2#6252)