Mailing-List: contact dev-help@apex.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@apex.apache.org
Date: Tue, 27 Sep 2016 18:10:20 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: dev@apex.incubator.apache.org
Message-ID: <JIRA.12985430.1467243008000.676973.1474999820599@Atlassian.JIRA>
In-Reply-To: <JIRA.12985430.1467243008000@Atlassian.JIRA>
References: <JIRA.12985430.1467243008000@Atlassian.JIRA> <JIRA.12985430.1467243008752@arcas>
Subject: [jira] [Commented] (APEXMALHAR-2130) Scalable windowed storage
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Tue, 27 Sep 2016 18:10:25 -0000


    [ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15526982#comment-15526982 ] 

ASF GitHub Bot commented on APEXMALHAR-2130:
--------------------------------------------

Github user asfgit closed the pull request at:

    https://github.com/apache/apex-malhar/pull/424


> Scalable windowed storage
> -------------------------
>
>                 Key: APEXMALHAR-2130
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: bright chen
>            Assignee: David Yan
>              Labels: roadmap
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the checkpointing window id.  This should be done incrementally (ManagedState) to avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that snapshot
> 4. When a window is committed, all windows with a lower ID should be purged from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage interfaces, and because of 2 and 3, we may want to add methods to the WindowedStorage interface so that the implementation of WindowedOperator can notify the storage of checkpointing, recovering and committing of a window.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)