incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kurtt.Lin (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (S4-137) Add HDFS StateStorage for more reliable checkpoint saving
Date Sun, 08 Sep 2013 08:39:52 GMT

    [ https://issues.apache.org/jira/browse/S4-137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13761229#comment-13761229
] 

Kurtt.Lin commented on S4-137:
------------------------------

@Matthieu, thanks for your comments. Indeed HDFSStateStorage is not very suited to large-sized,
frequent checkpointing, as you point out.

I've almost get the code done, but cannot decide the Hadoop core and confs should be included
in the JUnit tests or not(I mean, they are kind of "heavy"). In addition, should pick Hadoop
client 1.x or 2.x lib to go with the dev code?
                
> Add HDFS StateStorage for more reliable checkpoint saving
> ---------------------------------------------------------
>
>                 Key: S4-137
>                 URL: https://issues.apache.org/jira/browse/S4-137
>             Project: Apache S4
>          Issue Type: New Feature
>    Affects Versions: 0.6
>            Reporter: Kurtt.Lin
>            Priority: Minor
>              Labels: checkpoint
>             Fix For: 0.7
>
>   Original Estimate: 120h
>  Remaining Estimate: 120h
>
> Currently DefaultFileSystemStateStorage is the only implementation for StateStorage.
Should have a storage like HDFS to improve the reliability of the checkpoints' states. Some
apps would benefit from this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message