tez-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "TezQA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEZ-2172) FetcherOrderedGrouped using List to store InputAttemptIdentifier can lead to some inefficiency during remove() operation
Date Tue, 28 Jul 2015 18:56:04 GMT

    [ https://issues.apache.org/jira/browse/TEZ-2172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644851#comment-14644851
] 

TezQA commented on TEZ-2172:
----------------------------

{color:green}+1 overall{color}.  Here are the results of testing the latest attachment
  http://issues.apache.org/jira/secure/attachment/12747590/TEZ-2172.patch
  against master revision cf40088.

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 1 new or modified
test files.

    {color:green}+1 javac{color}.  The applied patch does not increase the total number of
javac compiler warnings.

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
3.0.1) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: https://builds.apache.org/job/PreCommit-TEZ-Build/930//testReport/
Console output: https://builds.apache.org/job/PreCommit-TEZ-Build/930//console

This message is automatically generated.

> FetcherOrderedGrouped using List to store InputAttemptIdentifier can lead to some inefficiency
during remove() operation
> ------------------------------------------------------------------------------------------------------------------------
>
>                 Key: TEZ-2172
>                 URL: https://issues.apache.org/jira/browse/TEZ-2172
>             Project: Apache Tez
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Saikat
>         Attachments: TEZ-2172.patch
>
>
> As part of fixing TEZ-2001, FetcherOrderedGrouped stores InputAttemptIdentifier in List.
 This can lead to some inefficiency - since the size of this list can be ~30, and remove()
calls can be expensive. 
> Option 1:  by using the spillId in the hashCode - or a wrapping structure for just this.
However, SpillId can not be added to the hashCode as it would break ShuffleScheduler shuffleInfoEventsMap.

> Option 2: consider using Map with an identifier. 
> Need to consider other options as well. Creating this jira as a placeholder to fix this
issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message