gearpump-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Manu Zhang (JIRA)" <>
Subject [jira] [Commented] (GEARPUMP-63) Gearpump Storage framework
Date Thu, 21 Apr 2016 07:44:25 GMT


Manu Zhang commented on GEARPUMP-63:

Comments history,

Perhaps the different types of storage could share a common API or aspects of collections
so AppStorage doesn't need to change every time a new type of storage is added. Also at least
with JarStore see [#1204|] where lack of streaming
causes Master to exit.

Should AppStorage contain write, read methods?


Hi @kkasravi, I agree that sharing common API is one goal. This is why the checkpoint store
and offset store in current codebase are unified.

The read/write methods are for KVStore instead of AppStorage. AppStorage is a quite general
concept. It includes everything an application needs for storage (except connector).

+1 for the ideas. The difficulty will be how to make the storage implementation plugable.
So that user can choose the storage system they want, like casandra, hdfs, or hbase...

>From [netcomm|],
maybe i lose something, why not just base on akka persistence, it now support hbase/casandra/...

The rest conversations are about using akka persistence as a general storage solution and
I'll create a new jira for it.

> Gearpump Storage framework
> --------------------------
>                 Key: GEARPUMP-63
>                 URL:
>             Project: Apache Gearpump
>          Issue Type: New Feature
>            Reporter: Manu Zhang
>            Assignee: Weihua Jiang
> imported from on behalf of [~whjiang].
 His original proposal,
> In general, a Gearpump application requires following storage support:
> # Jar-file storage to store the application jar file(s).
> # application log. Currently we store logs in each node which makes application log analysis
> # application metrics.
> # application configuration.
> # data source offset store (for at-least once semantics of streaming application)
> # application state checkpoint store (for transaction semantics)
> The general idea is:
> # Provide a storage system satisfied the above requirements.
> # Assume this storage is highly available. That means, it is user's duty to provide such
kind of a storage. For test purpose, user can use some non-HA storage system. But, in product
use, it shall be HAed.
> # Isolate usage from implementation. That is, Gearpump doesn't rely on Hadoop-common
or HDFS or one specific implementation to provide such storage. User is free to implement
its own storage.
> # This is a daemon provided functionality and can be used by every Gearpump application.
> # This storage shall provide data retentation functionality and access control.
> # This storage provides a set of API to meet the above requirements instead of one low-level
> # User can override the system setting to provide dedicated implementation for certain
sub-storage system, e.g. chekcpoint store.
> # Akka replication shall store minimal info for an application and leave the majority
to this storage system. I.e. akka replication is more like a seed to this storage system.
> # In release, each storage implementation (e.g. storage-hdfs) is a standalone module/artifact.
> The draft of this storage looks like (quite initial, tentative to change):
> {code}
> trait Storage {
>     def createAppStorage(AppName, AppId) : AppStorage
>     def getAppStorage(AppId) : Option[AppStorage]
> }
> trait AppStorage {
>     def open
>     def close
>     def getJarStore: JarStore
>     def getMetricsStore: AppMetricsStore
>     def getKVStore: KVStore
>     def getLogAppender: LogAppender
>     def getConfiguration(ProcessorId): UserConfig
>     def setConfiguration(ProcessorId, UserConfig)
> }
> trait JarStore {
>     def copyFromLocal(localPath, remotePath)
>     def copyToLocal(remotePath, localPath)
> }
> ///assume K is sortable
> trait KVStore[K,V] {
>     def append(key, value)
>     def read(key): Try[Option[V]]
> }
> {code}

This message was sent by Atlassian JIRA

View raw message