hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-757) The partitioning job output should be un-splitable
Date Thu, 16 May 2013 04:57:16 GMT

    [ https://issues.apache.org/jira/browse/HAMA-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13659226#comment-13659226
] 

Hudson commented on HAMA-757:
-----------------------------

Integrated in Hama-Nightly #911 (See [https://builds.apache.org/job/Hama-Nightly/911/])
    HAMA-757: The partitioning job output should be un-splitable (MaoYuan Xian via edwardyoon)
(Revision 1482677)

     Result = SUCCESS
edwardyoon : 
Files : 
* /hama/trunk/CHANGES.txt
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/BSPJobClient.java
* /hama/trunk/core/src/main/java/org/apache/hama/bsp/NonSplitSequenceFileInputFormat.java

                
> The partitioning job output should be un-splitable
> --------------------------------------------------
>
>                 Key: HAMA-757
>                 URL: https://issues.apache.org/jira/browse/HAMA-757
>             Project: Hama
>          Issue Type: Bug
>          Components: bsp core
>    Affects Versions: 0.6.1
>            Reporter: MaoYuan Xian
>            Assignee: MaoYuan Xian
>             Fix For: 0.6.2
>
>         Attachments: HAMA-757.patch
>
>
> When the output sequence files from partitioning job are large(bigger than two hdfs file
block size), the second round of the job (using these sequence file as input) will start up
more tasks than client want. Some times, this uncertainty make the job exceed the cluster
slot capacity.
> In the real project, I implemented an new Inputformat which marked as un-splitable to
solve the problem. Is there any better way?

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message