spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aniket Bhatnagar <aniket.bhatna...@gmail.com>
Subject Re: Example of updateStateByKey with initial RDD?
Date Thu, 08 Oct 2015 11:04:24 GMT
Here is an example:

val interval = 60 * 1000
val counts = eventsStream.map(event => {
  (event.timestamp - event.timestamp % interval, event)
}).updateStateByKey[Long](updateFunc = (events: Seq[Event], prevStateOpt:
Option[Long]) => {
  val prevCount = prevStateOpt.getOrElse(0L)
  val newCount = prevCount + events.size
  Some(newCount)
})
counts.print()

Hope it helps!

Thanks,
Aniket

On Thu, Oct 8, 2015 at 4:29 PM Bryan <bryan.jeffrey@gmail.com> wrote:

> Hello,
>
> Can anyone point me to a good example of updateStateByKey with an initial
> RDD? I am seeing a compile time error when following the API.
>
> Regards,
>
> Bryan Jeffrey
>

Mime
View raw message