incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ChiaHung Lin (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HAMA-445) Make configurable checkpointing
Date Mon, 05 Mar 2012 15:25:57 GMT

    [ https://issues.apache.org/jira/browse/HAMA-445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222397#comment-13222397
] 

ChiaHung Lin commented on HAMA-445:
-----------------------------------

Can we refactor checkpoint() function to be executed in another thread? 

If I remember correctly, original checkpoint() execution makes use of main thread to save
message bundle to hdfs. So if message bundle size is too large, this might delay the whole
process. Even message bundle size is not huge, during sync() the process still need to wait
after message bundle is saved to hdfs. Then 

it.remove();
messenger.transfer(addr, bundle);

can happen.    

                
> Make configurable checkpointing
> -------------------------------
>
>                 Key: HAMA-445
>                 URL: https://issues.apache.org/jira/browse/HAMA-445
>             Project: Hama
>          Issue Type: Sub-task
>          Components: bsp
>    Affects Versions: 0.4.0
>            Reporter: Edward J. Yoon
>            Assignee: Suraj Menon
>              Labels: newbie
>             Fix For: 0.5.0
>
>         Attachments: HAMA-498-445.patch
>
>
> We should extend the BSPJob to let the user set the checkpoint intervals.
> bq.  job.setCheckpointInterval(5);
> This method should put the parameter into the configuration of the job with a meaningful
key, e.G. "bsp.checkpoint.interval".
> In the BSPPeerImpl we should check if this interval has been reached and make the checkpointing
accordingly.
> Checkpointing gets called in BSPPeerImpl#sync(), there is already a condition which checks
if checkpointing is enabled.
> Plus points:
> If you can provide an additional method in BSPJob that let's the user enable or disable
checkpointing. Hint: Configuration key is: "bsp.checkpoint.enabled".

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message