tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jihoon Son (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-317) Improve TajoResourceManager to support more elaborate resource management
Date Thu, 28 Nov 2013 08:15:35 GMT

    [ https://issues.apache.org/jira/browse/TAJO-317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13834591#comment-13834591

Jihoon Son commented on TAJO-317:

I found that the implementation is contrary to your description.
That is, you said that workers' memory is tajo.worker.resource.memory-mb when the dedication
flag is set, but is actually tajo.worker.resource.dedicated-memory-ratio * max heap in the
Please fix this difference.

Also, there are some unused imports, commented out codes, and print codes for the debug.
Please remove them.

> Improve TajoResourceManager to support more elaborate resource management
> -------------------------------------------------------------------------
>                 Key: TAJO-317
>                 URL: https://issues.apache.org/jira/browse/TAJO-317
>             Project: Tajo
>          Issue Type: Improvement
>          Components: resource manager
>            Reporter: Hyunsik Choi
>            Assignee: Keuntae Park
>             Fix For: 0.8-incubating
>         Attachments: TAJO-317.patch, TAJO-317_2.patch
> h3. Status of the current Tajo Resource Manager (RM)
>  * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management
through memory allocations. 
>  * In addition, Tajo RM considers the memory resource as the fixed number of slots.
> h3. Problem
> In many cases, workloads can be categorized into I/O intensive job and CPU and memory
consuming job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O
intensive job. In general, Aggregation can be belong to CPU-memory consuming job. The current
RM is not fit to support selectively I/O intensive job or CPU-memory consuming job because
it provides only memory slots. We need more elaborate resource management mechanism.
> In addition, in most resource management systems, the remain resource less than required
resource is not allocated in response to a resource request. It is not good to fully utilize
the cluster resources. In order to mitigate this problem, we need to add resilience to allocation
mechanism. For example, min-max request would be useful for it.
> h3. Proposal
>  * Tajo RM should provides resource management for disk and cpu-memory.
>  ** Tajo RM should provide allocation request call with min, max memory request, and
min, max disk request.
>  *** min-max request will be useful to fully utilize remain cluster resources.
>  * Each resource request should have a priority. The priority can be disk or memory.
>   ** If the priority is disk
>   *** disk allocation will be limited depending on the remain disk resource
>   *** memory allocation will be not limited regardless of the remain memory resource,
and just reduce the remain memory resource.
>   ** If the priority is memory
>    *** memory allocation will be limited depending on the remain memory resource
>    *** disk allocation will be not limited regardless of the remain disk resource, and
just reduce the remain disk resource.
>  * disk resource in each worker is represented as a float value.
>  ** The initial disk resource will be the number of disks which participate in HDFS data

This message was sent by Atlassian JIRA

View raw message