spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <>
Subject Re: Valid spark streaming use case?
Date Thu, 17 Apr 2014 23:31:29 GMT
This is a good usecase for using DStream.updateStateByKey! This allows you
to maintain arbitrary per-key state. Checkout this example.

Also take a look at the documentation for more information:

This example just keeps the an integer (the running count) as the state,
but in practice it can be anything. In your case, you can keep that
per-group (i.e. per-key) threshold as the state. In the update function,
for each record of a group, you can check the timestamp against the
threshold and push stuff to DB and maybe update the threshold.


On Thu, Apr 17, 2014 at 1:24 PM, xargsgrep <> wrote:

> Hi, I'm completely new to Spark streaming (and Spark) and have been reading
> up on it and trying out various examples the past few days. I have a
> particular use case which I think it would work well for, but I wanted to
> put it out there and get some feedback on whether or not it actually would.
> The use case is:
> We have web tracking data continuously coming in from a pool of web
> servers.
> For simplicity, let's just say the data is text lines with a known set of
> fields, eg: "timestamp userId domain ...". What I want to do is:
> 1. group this continuous stream of data by "userId:domain", and
> 2. when the latest timestamp in each group is older than a certain
> threshold, persist the results to a DB
> #1 is straightforward and there are plenty of examples showing how to do
> it.
> However, I'm not sure how I would go about doing #2, or if that's something
> I can even do with spark because as far as I can tell it operates on
> sliding
> windows. I really just want to continue to accumulate these groups of
> "userId:domain" for all time (without specifying a window) and then roll
> them up and flush them once no new data has come in for a group after a
> certain amount of time. Would the updateStateByKey function allow me to do
> this somehow?
> Any help would be appreciated.
> --
> View this message in context:
> Sent from the Apache Spark User List mailing list archive at

View raw message