spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gerard Maas <gerard.m...@gmail.com>
Subject Re: Batch timestamp in spark streaming
Date Fri, 19 Dec 2014 12:43:25 GMT
You could use dstream.foreachRDD( (rdd, timestamp) => ???)  to get access
to the time of each batch.

On Fri, Dec 19, 2014 at 12:59 PM, nelson <nelson.verdier@ysance.com> wrote:
>
> Hi all,
>
> I know the topic have been discussed before, but i couldn't find an answer
> that might suits me.
>
> How do you retrieve the current batch timestamp in spark streaming? Maybe
> via BatchInfo but it does not seem to be linked to streaming context or
> else... I currently have 1 minutes micro-batch and i'd like to link every
> logs i process with the start of the batch a log belongs to.
>
> I also thought about broadcasting a new timestamp at the end of every batch
> thanks to a StreamingListener but i couldn't manage to overwrite the first
> broadcasted value.
>
> Do you guys have any ideas?
> Thanks a lot,
>
> Nelson.
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Batch-timestamp-in-spark-streaming-tp20786.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message