apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-283) Operator checkpointing in distributed in-memory store
Date Thu, 17 Dec 2015 20:16:47 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062713#comment-15062713
] 

ASF GitHub Bot commented on APEXCORE-283:
-----------------------------------------

Github user gauravgopi123 commented on a diff in the pull request:

    https://github.com/apache/incubator-apex-core/pull/184#discussion_r47955345
  
    --- Diff: engine/src/main/java/com/datatorrent/stram/StramClient.java ---
    @@ -455,6 +476,12 @@ public void startApplication() throws YarnException, IOException
           }
     
           dag.getAttributes().put(LogicalPlan.APPLICATION_PATH, appPath.toString());
    +      StorageAgent agent = dag.getAttributes().get(OperatorContext.STORAGE_AGENT);
    +      if (agent != null && agent instanceof StorageAgent.ApplicationAwareStorageAgent)
{
    +        StorageAgent.ApplicationAwareStorageAgent ap = (StorageAgent.ApplicationAwareStorageAgent)dag.getAttributes().get(OperatorContext.STORAGE_AGENT);
    --- End diff --
    
    This call is not needed


> Operator checkpointing in distributed in-memory store
> -----------------------------------------------------
>
>                 Key: APEXCORE-283
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-283
>             Project: Apache Apex Core
>          Issue Type: New Feature
>    Affects Versions: 3.2.0
>            Reporter: Ashish Tadose
>            Assignee: Ashish Tadose
>            Priority: Minor
>             Fix For: 3.3.0
>
>
> Currently Apex engine provides operator checkpointing in Hdfs ( with Hdfs backed StorageAgents
i.e. FSStorageAgent & AsyncFSStorageAgent )
> As operator check-pointing is critical functionality of Apex streaming platform to ensure
fault tolerant behavior, platform should also provide alternate StorageAgents which will work
seamlessly with large applications that requires Exactly once semantics.
> HDFS read/write latency is limited and doesn't improve beyond certain point because of
disk io & staging writes. Having alternate strategy to this check-pointing in fault tolerant
distributed in-memory grid would ensure application stability and performance is not impacted
by checkpointing
> *This feature will add below functionalities to core*
> - A KeyValue store interface which is used by In-memory checkpointing storage agent.
> - Abstract implementation of KeyValue storage agent which can be configured with concrete
implementation of KeyValue store for checkpointing.
> - Concrete implementation of In memory storage agent for Apache Geode
> - Interface for storage agent to provide application id



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message