tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keuntae Park (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (TAJO-317) Improve TajoResourceManager to support more elaborate resource management
Date Wed, 20 Nov 2013 07:55:23 GMT

     [ https://issues.apache.org/jira/browse/TAJO-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Keuntae Park reassigned TAJO-317:
---------------------------------

    Assignee: Keuntae Park

Thank you, Hyunsik.
I'll start this issue.

> Improve TajoResourceManager to support more elaborate resource management
> -------------------------------------------------------------------------
>
>                 Key: TAJO-317
>                 URL: https://issues.apache.org/jira/browse/TAJO-317
>             Project: Tajo
>          Issue Type: Improvement
>          Components: resource manager
>            Reporter: Hyunsik Choi
>            Assignee: Keuntae Park
>             Fix For: 0.8-incubating
>
>
> h3. Status of the current Tajo Resource Manager (RM)
>  * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management
through memory allocations. 
>  * In addition, Tajo RM considers the memory resource as the fixed number of slots.
> h3. Problem
> In many cases, workloads can be categorized into I/O intensive job and CPU and memory
consuming job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O
intensive job. In general, Aggregation can be belong to CPU-memory consuming job. The current
RM is not fit to support selectively I/O intensive job or CPU-memory consuming job because
it provides only memory slots. We need more elaborate resource management mechanism.
> In addition, in most resource management systems, the remain resource less than required
resource is not allocated in response to a resource request. It is not good to fully utilize
the cluster resources. In order to mitigate this problem, we need to add resilience to allocation
mechanism. For example, min-max request would be useful for it.
> h3. Proposal
>  * Tajo RM should provides resource management for disk and cpu-memory.
>  ** Tajo RM should provide allocation request call with min, max memory request, and
min, max disk request.
>  *** min-max request will be useful to fully utilize remain cluster resources.
>  * Each resource request should have a priority. The priority can be disk or memory.
>   ** If the priority is disk
>   *** disk allocation will be limited depending on the remain disk resource
>   *** memory allocation will be not limited regardless of the remain memory resource,
and just reduce the remain memory resource.
>   ** If the priority is memory
>    *** memory allocation will be limited depending on the remain memory resource
>    *** disk allocation will be not limited regardless of the remain disk resource, and
just reduce the remain disk resource.
>  * disk resource in each worker is represented as a float value.
>  ** The initial disk resource will be the number of disks which participate in HDFS data
directory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message