tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jihoon Son (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (TAJO-385) Refactoring TaskScheduler to assign multiple fragments
Date Sun, 22 Dec 2013 14:04:50 GMT

    [ https://issues.apache.org/jira/browse/TAJO-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855173#comment-13855173
] 

Jihoon Son edited comment on TAJO-385 at 12/22/13 2:04 PM:
-----------------------------------------------------------

I uploaded the fourth patch that reflects Hyunsik's suggestions.
The progress is normally increased as tasks are finished.

The task scheduler can be configured as follows.
{noformat}
  <property>
    <name>tajo.querymaster.task-scheduler</name>
    <value>task_scheduler_name</value>
  </property>
{noformat}
The default task scheduler is DefaultTaskScheduler which is the previous one.

Also, the task volume is adjusted according to the number of containers and the remaining
number of scheduled fragments.

Please review this patch.
Thanks.


was (Author: jihoonson):
I uploaded the fourth patch that reflects Hyunsik's suggestions.
The task scheduler can be configured as follows.
{noformat}
  <property>
    <name>tajo.querymaster.task-scheduler</name>
    <value>task_scheduler_name</value>
  </property>
{noformat}
The default task scheduler is DefaultTaskScheduler which is the previous one.

Also, the task volume is adjusted according to the number of containers and the remaining
number of scheduled fragments.

Please review this patch.
Thanks.

> Refactoring TaskScheduler to assign multiple fragments
> ------------------------------------------------------
>
>                 Key: TAJO-385
>                 URL: https://issues.apache.org/jira/browse/TAJO-385
>             Project: Tajo
>          Issue Type: Improvement
>          Components: query master
>    Affects Versions: 0.8-incubating
>            Reporter: Jihoon Son
>            Assignee: Jihoon Son
>         Attachments: TAJO-385.patch, TAJO-385_2.patch, TAJO-385_3.patch, TAJO-385_4.patch
>
>
> In the current implementation, each task processes only one fragment.
> However, processing multiple fragments in a task will increase the query processing performance
according to the storage layout and the user queries.
> In this issue, TaskScheduler is refactored to enable assigning multiple fragments to
each task.
> Followings should be contained.
> * Schedule Fragments instead of QueryUnits in TaskScheduler
> ** The QueryUnit creation is postponed until TaskScheduler receives task requests from
workers.
> ** When TaskScheduler receives task requests from workers, it dynamically creates an
QueryUnit and assigns one or more fragments.
> ** The fragment scheduling should take into account the disk load balancing.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message