Mailing-List: contact dev-help@tajo.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@tajo.incubator.apache.org
Date: Mon, 2 Dec 2013 11:26:36 +0000 (UTC)
From: "Hyunsik Choi (JIRA)" <jira@apache.org>
To: dev@tajo.incubator.apache.org
Message-ID: <JIRA.12680094.1384914671030.53521.1385983596754@arcas>
In-Reply-To: <JIRA.12680094.1384914671030@arcas>
References: <JIRA.12680094.1384914671030@arcas>
Subject: [jira] [Comment Edited] (TAJO-317) Improve TajoResourceManager to
 support more elaborate resource management
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/TAJO-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13836437#comment-13836437 ] 

Hyunsik Choi edited comment on TAJO-317 at 12/2/13 11:26 AM:
-------------------------------------------------------------

Could you please update the documentation by this patch?
http://tajo.incubator.apache.org/tajo-0.8.0-doc.html

Please just update the markdown file (tajo-project/src/site/markdown/tajo-0.8.0-doc.md) in order to reflect the up-to-date contents to tajo-0.8.0-doc.html file.


was (Author: hyunsik):
Could you please update the documentation?
http://tajo.incubator.apache.org/tajo-0.8.0-doc.html

Please just update the markdown file (tajo-project/src/site/markdown/tajo-0.8.0-doc.md) in order to reflect the up-to-date contents to tajo-0.8.0-doc.html file.


> Improve TajoResourceManager to support more elaborate resource management
> -------------------------------------------------------------------------
>
>                 Key: TAJO-317
>                 URL: https://issues.apache.org/jira/browse/TAJO-317
>             Project: Tajo
>          Issue Type: Improvement
>          Components: resource manager
>            Reporter: Hyunsik Choi
>            Assignee: Keuntae Park
>             Fix For: 0.8-incubating
>
>         Attachments: TAJO-317.patch, TAJO-317_2.patch, TAJO-317_3.patch, TAJO-317_4.patch, TAJO-317_5.patch
>
>
> h3. Status of the current Tajo Resource Manager (RM)
>  * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management through memory allocations. 
>  * In addition, Tajo RM considers the memory resource as the fixed number of slots.
> h3. Problem
> In many cases, workloads can be categorized into I/O intensive job and CPU and memory consuming job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O intensive job. In general, Aggregation can be belong to CPU-memory consuming job. The current RM is not fit to support selectively I/O intensive job or CPU-memory consuming job because it provides only memory slots. We need more elaborate resource management mechanism.
> In addition, in most resource management systems, the remain resource less than required resource is not allocated in response to a resource request. It is not good to fully utilize the cluster resources. In order to mitigate this problem, we need to add resilience to allocation mechanism. For example, min-max request would be useful for it.
> h3. Proposal
>  * Tajo RM should provides resource management for disk and cpu-memory.
>  ** Tajo RM should provide allocation request call with min, max memory request, and min, max disk request.
>  *** min-max request will be useful to fully utilize remain cluster resources.
>  * Each resource request should have a priority. The priority can be disk or memory.
>   ** If the priority is disk
>   *** disk allocation will be limited depending on the remain disk resource
>   *** memory allocation will be not limited regardless of the remain memory resource, and just reduce the remain memory resource.
>   ** If the priority is memory
>    *** memory allocation will be limited depending on the remain memory resource
>    *** disk allocation will be not limited regardless of the remain disk resource, and just reduce the remain disk resource.
>  * disk resource in each worker is represented as a float value.
>  ** The initial disk resource will be the number of disks which participate in HDFS data directory.


--
This message was sent by Atlassian JIRA
(v6.1#6144)