spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tathagata Das <t...@databricks.com>
Subject Re: Calculating tuple count /input rate with time
Date Tue, 23 Jun 2015 20:23:31 GMT
This should give accurate count for each batch, though for getting the rate
you have to make sure that you streaming app is stable, that is, batches
are processed as fast as they are received (scheduling delay in the spark
streaming UI is approx 0).

TD

On Tue, Jun 23, 2015 at 2:49 AM, anshu shukla <anshushukla0@gmail.com>
wrote:

> I am calculating input rate using the following logic.
>
> And i think this foreachRDD is always running on driver (println are seen on driver)
>
> 1- Is there any other way to do that in less cost .
>
> 2- Will this give me the correct count for rate  .
>
>
> //code -
>
> inputStream.foreachRDD(new Function<JavaRDD<String>, Void>() {
>     @Override
>     public Void call(JavaRDD<String> stringJavaRDD) throws Exception {
>         System.out.println(System.currentTimeMillis()+",spoutstringJavaRDD," + stringJavaRDD.count()
);
>         return null;
>     }
> });
>
>
>
> --
> Thanks & Regards,
> Anshu Shukla
>

Mime
View raw message