apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXMALHAR-2130) implement scalable windowed storage
Date Thu, 08 Sep 2016 21:06:22 GMT

    [ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475007#comment-15475007
] 

ASF GitHub Bot commented on APEXMALHAR-2130:
--------------------------------------------

GitHub user davidyan74 opened a pull request:

    https://github.com/apache/apex-malhar/pull/405

    APEXMALHAR-2130 #resolve Added a spillable map that takes two keys with support of iterating
through all entries with a given first key

    @siyuanh @tweise @ilooner Please review. This is mostly for the scalable implementation
of a WindowedKeyedStorage.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/davidyan74/apex-malhar APEXMALHAR-2130

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/apex-malhar/pull/405.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #405
    
----
commit 183c9ff01e3f197c9088cdcf460827023d414ce1
Author: David Yan <david@datatorrent.com>
Date:   2016-09-08T20:59:50Z

    APEXMALHAR-2130 #resolve Added a spillable map that takes two keys with support of iterating
through all entries with a given first key

----


> implement scalable windowed storage
> -----------------------------------
>
>                 Key: APEXMALHAR-2130
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: bright chen
>            Assignee: David Yan
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the checkpointing
window id.  This should be done incrementally (ManagedState) to avoid wasting space with unchanged
data
> 3. When recovering, it takes the recovery window id and restores to that snapshot
> 4. When a window is committed, all windows with a lower ID should be purged from the
store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage interfaces, and because
of 2 and 3, we may want to add methods to the WindowedStorage interface so that the implementation
of WindowedOperator can notify the storage of checkpointing, recovering and committing of
a window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message