tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyunsik Choi (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (TAJO-317) Improve TajoResourceManager to support more elaborate resource management
Date Wed, 20 Nov 2013 02:33:22 GMT

     [ https://issues.apache.org/jira/browse/TAJO-317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Hyunsik Choi updated TAJO-317:
------------------------------

    Description: 
h3. Status of the current Tajo Resource Manager (RM)
 * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management
through memory allocations. 
 * In addition, Tajo RM considers the memory resource as the fixed number of slots.

h3. Problem
In many cases, workloads can be categorized into I/O intensive job and CPU and memory consuming
job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O intensive
job. In general, Aggregation can be belong to CPU-memory consuming job. The current RM is
not fit to support selectively I/O intensive job or CPU-memory consuming job because it provides
only memory slots. We need more elaborate resource management mechanism.

In addition, in most resource management systems, the remain resource less than required resource
is not allocated in response to a resource request. It is not good to fully utilize the cluster
resources. In order to mitigate this problem, we need to add resilience to allocation mechanism.
For example, min-max request would be useful for it.

h3. Proposal
 * Tajo RM should provides resource management for disk and cpu-memory.
 ** Tajo RM should provide allocation request call with min, max memory request, and min,
max disk request.
 *** min-max request will be useful to fully utilize remain cluster resources.
 * Each resource request should have a priority. The priority can be disk or memory.
  ** If the priority is disk
  *** disk allocation will be limited depending on the remain disk resource
  *** memory allocation will be not limited regardless of the remain memory resource, and
just reduce the remain memory resource.
  ** If the priority is memory
   *** memory allocation will be limited depending on the remain memory resource
   *** disk allocation will be not limited regardless of the remain disk resource, and just
reduce the remain disk resource.
 * disk resource in each worker is represented as a float value.
 ** The initial disk resource will be the number of disks which participate in HDFS data directory.


  was:
h4. Status of the current Tajo Resource Manager (RM)
 * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management
through memory allocations. 
 * In addition, Tajo RM considers the memory resource as the fixed number of slots.

h4. Problem
In many cases, workloads can be categorized into I/O intensive job and CPU and memory consuming
job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O intensive
job. In general, Aggregation can be belong to CPU-memory consuming job. The current RM is
not fit to support selectively I/O intensive job or CPU-memory consuming job because it provides
only memory slots. We need more elaborate resource management mechanism.

In addition, in most resource management systems, the remain resource less than required resource
is not allocated in response to a resource request. It is not good to fully utilize the cluster
resources. In order to mitigate this problem, we need to add resilience to allocation mechanism.
For example, min-max request would be useful for it.

h4. Proposal
 * Tajo RM should provides resource management for disk and cpu-memory.
 ** Tajo RM should provide allocation request call with min, max memory request, and min,
max disk request.
 *** min-max request will be useful to fully utilize remain cluster resources.
 * Each resource request should have a priority. The priority can be disk or memory.
  ** If the priority is disk
  *** disk allocation will be limited depending on the remain disk resource
  *** memory allocation will be not limited regardless of the remain memory resource, and
just reduce the remain memory resource.
  ** If the priority is memory
   *** memory allocation will be limited depending on the remain memory resource
   *** disk allocation will be not limited regardless of the remain disk resource, and just
reduce the remain disk resource.
 * disk resource in each worker is represented as a float value.
 ** The initial disk resource will be the number of disks which participate in HDFS data directory.



> Improve TajoResourceManager to support more elaborate resource management
> -------------------------------------------------------------------------
>
>                 Key: TAJO-317
>                 URL: https://issues.apache.org/jira/browse/TAJO-317
>             Project: Tajo
>          Issue Type: Improvement
>          Components: resource manager
>            Reporter: Hyunsik Choi
>             Fix For: 0.8-incubating
>
>
> h3. Status of the current Tajo Resource Manager (RM)
>  * Tajo RM manages CPU, DISK resource incompletely, and it only provides resource management
through memory allocations. 
>  * In addition, Tajo RM considers the memory resource as the fixed number of slots.
> h3. Problem
> In many cases, workloads can be categorized into I/O intensive job and CPU and memory
consuming job. For example, scan and hash partition or INSERT OVERWRITE may be belong to I/O
intensive job. In general, Aggregation can be belong to CPU-memory consuming job. The current
RM is not fit to support selectively I/O intensive job or CPU-memory consuming job because
it provides only memory slots. We need more elaborate resource management mechanism.
> In addition, in most resource management systems, the remain resource less than required
resource is not allocated in response to a resource request. It is not good to fully utilize
the cluster resources. In order to mitigate this problem, we need to add resilience to allocation
mechanism. For example, min-max request would be useful for it.
> h3. Proposal
>  * Tajo RM should provides resource management for disk and cpu-memory.
>  ** Tajo RM should provide allocation request call with min, max memory request, and
min, max disk request.
>  *** min-max request will be useful to fully utilize remain cluster resources.
>  * Each resource request should have a priority. The priority can be disk or memory.
>   ** If the priority is disk
>   *** disk allocation will be limited depending on the remain disk resource
>   *** memory allocation will be not limited regardless of the remain memory resource,
and just reduce the remain memory resource.
>   ** If the priority is memory
>    *** memory allocation will be limited depending on the remain memory resource
>    *** disk allocation will be not limited regardless of the remain disk resource, and
just reduce the remain disk resource.
>  * disk resource in each worker is represented as a float value.
>  ** The initial disk resource will be the number of disks which participate in HDFS data
directory.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message