tajo-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hyunsik Choi (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TAJO-982) Improve Fetcher to get multiple shuffle outputs through a request
Date Sun, 03 Dec 2017 23:58:01 GMT

    [ https://issues.apache.org/jira/browse/TAJO-982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16276160#comment-16276160

Hyunsik Choi commented on TAJO-982:

I rescheduled it to 0.13.0 because it didn't start yet.

> Improve Fetcher to get multiple shuffle outputs through a request
> -----------------------------------------------------------------
>                 Key: TAJO-982
>                 URL: https://issues.apache.org/jira/browse/TAJO-982
>             Project: Tajo
>          Issue Type: Improvement
>          Components: Data Shuffle
>            Reporter: Hyunsik Choi
>            Assignee: Jihoon Son
>             Fix For: 0.13.0
> Currently, Fetcher only can request at most a fetch for one shuffle output at a time.
The implementation can cause performance degradation even though intermediate data is actually
> For example, If an input data set of the first stage is big and the intermediate data
is very small, QueryMaster will choose a few of nodes for next execution block. Since each
node keeps limited threads for fetch, it will take a long time for the nodes for next stage
to fetch all intermediate.
> If Fetcher can get multiple shuffle outputs through a request, it would solve the slowness
which occurs in some cases.

This message was sent by Atlassian JIRA

View raw message