Mailing-List: contact dev-help@tajo.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tajo.incubator.apache.org
Date: Fri, 6 Dec 2013 00:54:36 +0000 (UTC)
From: "Keuntae Park (JIRA)" <jira@apache.org>
To: dev@tajo.incubator.apache.org
Message-ID: <JIRA.12680094.1384914671030.81783.1386291276085@arcas>
In-Reply-To: <JIRA.12680094.1384914671030@arcas>
References: <JIRA.12680094.1384914671030@arcas>
Subject: [jira] [Commented] (TAJO-317) Improve TajoResourceManager to
 support more elaborate resource management
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/TAJO-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13840755#comment-13840755 ] 

Keuntae Park commented on TAJO-317:
-----------------------------------

You are right, Jihoon.
I'm confused between default configuration and my working configuration :)
I agree with your fixed documentation, which is correct and much easier to understand.
Thank you for the review !!

> Improve TajoResourceManager to support more elaborate resource management
> -------------------------------------------------------------------------
>
>                 Key: TAJO-317
>                 URL: https://issues.apache.org/jira/browse/TAJO-317
>             Project: Tajo
>          Issue Type: Improvement
>          Components: resource manager
>            Reporter: Hyunsik Choi
>            Assignee: Keuntae Park
>             Fix For: 0.8-incubating
>
>         Attachments: TAJO-317.doc.patch, TAJO-317.doc_2.patch, TAJO-317.patch, TAJO-317_2.patch, TAJO-317_3.patch, TAJO-317_4.patch, TAJO-317_5.patch
>
>
> h3. Status of the current Tajo Resource Manager (RM)
>  * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management through memory allocations. 
>  * In addition, Tajo RM considers the memory resource as the fixed number of slots.
> h3. Problem
> In many cases, workloads can be categorized into I/O intensive job and CPU and memory consuming job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O intensive job. In general, Aggregation can be belong to CPU-memory consuming job. The current RM is not fit to support selectively I/O intensive job or CPU-memory consuming job because it provides only memory slots. We need more elaborate resource management mechanism.
> In addition, in most resource management systems, the remain resource less than required resource is not allocated in response to a resource request. It is not good to fully utilize the cluster resources. In order to mitigate this problem, we need to add resilience to allocation mechanism. For example, min-max request would be useful for it.
> h3. Proposal
>  * Tajo RM should provides resource management for disk and cpu-memory.
>  ** Tajo RM should provide allocation request call with min, max memory request, and min, max disk request.
>  *** min-max request will be useful to fully utilize remain cluster resources.
>  * Each resource request should have a priority. The priority can be disk or memory.
>   ** If the priority is disk
>   *** disk allocation will be limited depending on the remain disk resource
>   *** memory allocation will be not limited regardless of the remain memory resource, and just reduce the remain memory resource.
>   ** If the priority is memory
>    *** memory allocation will be limited depending on the remain memory resource
>    *** disk allocation will be not limited regardless of the remain disk resource, and just reduce the remain disk resource.
>  * disk resource in each worker is represented as a float value.
>  ** The initial disk resource will be the number of disks which participate in HDFS data directory.


--
This message was sent by Atlassian JIRA
(v6.1#6144)