tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyunsik Choi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-385) Refactoring TaskScheduler to assign multiple fragments
Date Mon, 16 Dec 2013 11:10:08 GMT

    [ https://issues.apache.org/jira/browse/TAJO-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13849041#comment-13849041

Hyunsik Choi commented on TAJO-385:

This is a just quick review. I'll review this patch in depth within few days. 

Occasionally, this patch shows the number of completed tasks in SubQuery is greater than the
total number of tasks. Also, the query progress becomes higher very fast in the early stage,
and the progress becomes very slow as the query proceeds. Probably, it is because the progress
of an execution is _(c / n) * 100_, where _c_ is the number of completed tasks and _n_ is
the total number of tasks, and the total number of tasks is being increased during query processing.

> Refactoring TaskScheduler to assign multiple fragments
> ------------------------------------------------------
>                 Key: TAJO-385
>                 URL: https://issues.apache.org/jira/browse/TAJO-385
>             Project: Tajo
>          Issue Type: Improvement
>          Components: query master
>    Affects Versions: 0.8-incubating
>            Reporter: Jihoon Son
>            Assignee: Jihoon Son
>         Attachments: TAJO-385.patch, TAJO-385_2.patch, TAJO-385_3.patch
> In the current implementation, each task processes only one fragment.
> However, processing multiple fragments in a task will increase the query processing performance
according to the storage layout and the user queries.
> In this issue, TaskScheduler is refactored to enable assigning multiple fragments to
each task.
> Followings should be contained.
> * Schedule Fragments instead of QueryUnits in TaskScheduler
> ** The QueryUnit creation is postponed until TaskScheduler receives task requests from
> ** When TaskScheduler receives task requests from workers, it dynamically creates an
QueryUnit and assigns one or more fragments.
> ** The fragment scheduling should take into account the disk load balancing.

This message was sent by Atlassian JIRA

View raw message