spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arush Kharbanda <ar...@sigmoidanalytics.com>
Subject Re: On app upgrade, restore sliding window data.
Date Tue, 24 Feb 2015 09:49:14 GMT
I think this could be of some help to you.

https://issues.apache.org/jira/browse/SPARK-3660



On Tue, Feb 24, 2015 at 2:18 AM, Matus Faro <matus.faro@kik.com> wrote:

> Hi,
>
> Our application is being designed to operate at all times on a large
> sliding window (day+) of data. The operations performed on the window
> of data will change fairly frequently and I need a way to save and
> restore the sliding window after an app upgrade without having to wait
> the duration of the sliding window to "warm up". Because it's an app
> upgrade, checkpointing will not work unfortunately.
>
> I can potentially dump the window to an outside storage periodically
> or on app shutdown, but I don't have an ideal way of restoring it.
>
> I thought about two non-ideal solutions:
> 1. Load the previous data all at once into the sliding window on app
> startup. The problem is, at one point I will have double the data in
> the sliding window until the initial batch of data goes out of scope.
> 2. Broadcast the previous state of the window separately from the
> window. Perform the operations on both sets of data until it comes out
> of scope. The problem is, the data will not fit into memory.
>
> Solutions that would solve my problem:
> 1. Ability to pre-populate sliding window.
> 2. Have control over batch slicing. It would be nice for a Receiver to
> dictate the current batch timestamp in order to slow down or fast
> forward time.
>
> Any feedback would be greatly appreciated!
>
> Thank you,
> Matus
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>


-- 

[image: Sigmoid Analytics] <http://htmlsig.com/www.sigmoidanalytics.com>

*Arush Kharbanda* || Technical Teamlead

arush@sigmoidanalytics.com || www.sigmoidanalytics.com

Mime
View raw message