Mailing-List: contact user-help@spark.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
References: <56164c72.4fda8c0a.63682.4615@mx.google.com>
In-Reply-To: <56164c72.4fda8c0a.63682.4615@mx.google.com>
From: Aniket Bhatnagar <aniket.bhatnagar@gmail.com>
Date: Thu, 08 Oct 2015 11:04:24 +0000
Message-ID: 
 <CAJOb8bvj_E0-FBJ6ra2fB07qESaBok1FmN_PDoBbRrCePPCYzA@mail.gmail.com>
Subject: Re: Example of updateStateByKey with initial RDD?
To: Bryan <bryan.jeffrey@gmail.com>, user <user@spark.apache.org>
Content-Type: multipart/alternative; boundary=001a11413996fc9ae0052195d31e

--001a11413996fc9ae0052195d31e
Content-Type: text/plain; charset=UTF-8

Here is an example:

val interval = 60 * 1000
val counts = eventsStream.map(event => {
  (event.timestamp - event.timestamp % interval, event)
}).updateStateByKey[Long](updateFunc = (events: Seq[Event], prevStateOpt:
Option[Long]) => {
  val prevCount = prevStateOpt.getOrElse(0L)
  val newCount = prevCount + events.size
  Some(newCount)
})
counts.print()

Hope it helps!

Thanks,
Aniket

On Thu, Oct 8, 2015 at 4:29 PM Bryan <bryan.jeffrey@gmail.com> wrote:

> Hello,
>
> Can anyone point me to a good example of updateStateByKey with an initial
> RDD? I am seeing a compile time error when following the API.
>
> Regards,
>
> Bryan Jeffrey
>

--001a11413996fc9ae0052195d31e
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Here is an example:<div><br></div><div><div>val interval =
=3D 60 * 1000</div><div>val counts =3D eventsStream.map(event =3D&gt; {</di=
v><div>=C2=A0 (event.timestamp - event.timestamp % interval, event)</div><d=
iv>}).updateStateByKey[Long](updateFunc =3D (events: Seq[Event], prevStateO=
pt: Option[Long]) =3D&gt; {</div><div>=C2=A0 val prevCount =3D prevStateOpt=
.getOrElse(0L)</div><div>=C2=A0 val newCount =3D prevCount + events.size</d=
iv><div>=C2=A0 Some(newCount)</div><div>})</div><div>counts.print()</div></=
div><div><br></div><div>Hope it helps!</div><div><br></div><div>Thanks,</di=
v><div>Aniket</div></div><br><div class=3D"gmail_quote"><div dir=3D"ltr">On=
 Thu, Oct 8, 2015 at 4:29 PM Bryan &lt;<a href=3D"mailto:bryan.jeffrey@gmai=
l.com">bryan.jeffrey@gmail.com</a>&gt; wrote:<br></div><blockquote class=3D=
"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding=
-left:1ex"><div><div><div style=3D"font-family:Calibri,sans-serif;font-size=
:11pt">Hello,<br><br>Can anyone point me to a good example of updateStateBy=
Key with an initial RDD? I am seeing a compile time error when following th=
e API.<br><br>Regards,<br><br>Bryan Jeffrey</div></div></div></blockquote><=
/div>

--001a11413996fc9ae0052195d31e--