incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mariano Valles (JIRA)" <>
Subject [jira] [Commented] (S4-137) Add HDFS StateStorage for more reliable checkpoint saving
Date Wed, 04 Sep 2013 09:44:52 GMT


Mariano Valles commented on S4-137:

I worked on something related to this back on S4 0.4.
At that moment one of the alternatives was to use Redis as a checkpoint storage mechanism.

I worked on creating an append-only file similar to the DefaultFileSystemStateStorage but
using one file instead of many. But this was more a proof of concept implementation.

While doing this we also thought on using a Distributed File System such as NFS or similar
and just using the DefaultFileSystemStateStorage on this path. This would improve the availability
of the checkpoints.
> Add HDFS StateStorage for more reliable checkpoint saving
> ---------------------------------------------------------
>                 Key: S4-137
>                 URL:
>             Project: Apache S4
>          Issue Type: New Feature
>    Affects Versions: 0.6
>            Reporter: Kurtt.Lin
>            Priority: Minor
>              Labels: checkpoint
>             Fix For: 0.7
>   Original Estimate: 120h
>  Remaining Estimate: 120h
> Currently DefaultFileSystemStateStorage is the only implementation for StateStorage.
Should have a storage like HDFS to improve the reliability of the checkpoints' states. Some
apps would benefit from this.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message