tajo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jihoon Son (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (TAJO-385) Refactoring TaskScheduler to assign multiple fragments
Date Mon, 09 Dec 2013 12:54:06 GMT

     [ https://issues.apache.org/jira/browse/TAJO-385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jihoon Son updated TAJO-385:
----------------------------

    Attachment: TAJO-385.patch

Thanks guys for comments.
I attached a patch for this issue.
As described above, Fragments are scheduled to TaskScheduler and QueryUnits are dynamically
created when TaskScheduler receives a TaskRequest from a worker.
When an QueryUnit is created, multiple fragments can be assigned to it.
Here, as Keuntae said, the number of fragments assigned to an QueryUnit is an important factor
for the performance, but it is hard to get the optimal value because the optimal value can
be different according to the data and user queries.
So, I added a configuration for users to specify the task size as follows.
{noformat}
<property>
  <name>tajo.task.size.default</name>
  <value>task_size_in_byte</value>
</property>
{noformat}

Also, I refactored TaskScheduler to make the scheduling algorithm pluggable.

I tested the patch by running TPC-H queries on my in-house cluster.
After applying this patch, the number of remote tasks is a little bit reduced.

> Refactoring TaskScheduler to assign multiple fragments
> ------------------------------------------------------
>
>                 Key: TAJO-385
>                 URL: https://issues.apache.org/jira/browse/TAJO-385
>             Project: Tajo
>          Issue Type: Improvement
>          Components: master
>    Affects Versions: 0.8-incubating
>            Reporter: Jihoon Son
>            Assignee: Jihoon Son
>         Attachments: TAJO-385.patch
>
>
> In the current implementation, each task processes only one fragment.
> However, processing multiple fragments in a task will increase the query processing performance
according to the storage layout and the user queries.
> In this issue, TaskScheduler is refactored to enable assigning multiple fragments to
each task.
> Followings should be contained.
> * Schedule Fragments instead of QueryUnits in TaskScheduler
> ** The QueryUnit creation is postponed until TaskScheduler receives task requests from
workers.
> ** When TaskScheduler receives task requests from workers, it dynamically creates an
QueryUnit and assigns one or more fragments.
> ** The fragment scheduling should take into account the disk load balancing.



--
This message was sent by Atlassian JIRA
(v6.1.4#6159)

Mime
View raw message