hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-756) Timing issue and file merging algorithm in PartitioningRunner make job fail
Date Sat, 11 May 2013 12:53:16 GMT

    [ https://issues.apache.org/jira/browse/HAMA-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655244#comment-13655244

Edward J. Yoon commented on HAMA-756:

+1 Nice. 

Just FYI, you can move the 'partitionID % peerNum' into a method and add unit test like:

    int peerNum = 6;
    for(int partitionID = 0; partitionID < 8; partitionID++) {
      int assignedID = PartitioningRunner.computeProcessPeerID(partitionID, peerNum);
      assertTrue(assignedID < peerNum);
> Timing issue and file merging algorithm in PartitioningRunner make job fail
> ---------------------------------------------------------------------------
>                 Key: HAMA-756
>                 URL: https://issues.apache.org/jira/browse/HAMA-756
>             Project: Hama
>          Issue Type: Bug
>            Reporter: MaoYuan Xian
>            Assignee: MaoYuan Xian
>            Priority: Blocker
>             Fix For: 0.6.2
>         Attachments: HAMA-756.patch
> There are two major problems in bsp methor of PartitioningRunner may make the partitioning
> 1. The call to peer.getNumPeers() may trigger the timing issue. In the special situation
when some tasks complete the bsp call but some others just enter the "for (FileStatus statu
: status)" loop, these remaining task calling to peer.getNumPeers() will trigger the problem.
> 2. The algorithm of merging the sequence files has the problem: e.g. when desiredNum
is 8 and partitioning task number (peer.getNumPeers()) is 6, the part-7 directory can not
find the handler to merging it as a file.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message