flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: How to get latency info from benchmark
Date Wed, 24 Aug 2016 12:12:16 GMT
Hi Eric,

Max is right, the tool has been used for a different benchmark [1]. The
throughput logger that should produce the right output is this one [2].
Very recently, I've opened a pull request for adding metric-measuring
support into the engine [3]. Maybe that's helpful for your experiments.


[1]
http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/
[2]
https://github.com/dataArtisans/performance/blob/master/flink-jobs/src/main/java/com/github/projectflink/streaming/Throughput.java#L203
[3] https://github.com/apache/flink/pull/2386



On Wed, Aug 24, 2016 at 2:04 PM, Maximilian Michels <mxm@apache.org> wrote:

> I believe the AnaylzeTool is for processing logs of a different benchmark.
>
> CC Jamie and Robert who worked on the benchmark.
>
> On Wed, Aug 24, 2016 at 3:25 AM, Eric Fukuda <e.s.fukuda@gmail.com> wrote:
> > Hi,
> >
> > I'm trying to benchmark Flink without Kafka as mentioned in this post
> > (http://data-artisans.com/extending-the-yahoo-streaming-benchmark/).
> After
> > running flink.benchmark.state.AdvertisingTopologyFlinkState with
> > user.local.event.generator in localConf.yaml set to 1, I ran
> > flink.benchmark.utils.AnalyzeTool giving
> > flink-1.0.1/log/flink-[username]-jobmanager-0-[servername].log as a
> > command-line argument. I got the following output and it does not have
> the
> > information about the latency.
> >
> >
> > ================= Latency (0 reports ) =====================
> > ================= Throughput (1 reports ) =====================
> > ====== null (entries: 10150)=======
> > Mean throughput 639078.5018497099
> > Exception in thread "main" java.lang.IndexOutOfBoundsException: toIndex
> = 2
> >         at java.util.ArrayList.subListRangeCheck(ArrayList.java:962)
> >         at java.util.ArrayList.subList(ArrayList.java:954)
> >         at flink.benchmark.utils.AnalyzeTool.main(AnalyzeTool.java:133)
> >
> >
> > Reading the code in AnalyzeTool.java, I found that it's looking for lines
> > that include "Latency" in the log file, but apparently it's not finding
> any.
> > I tried grepping the log file, and couldn't find any either. I have one
> > server that runs both JobManager and Task Manager and another server that
> > runs Redis, and they are connected through a network with each other.
> >
> > I think I have to do something to read the data stored in Redis before
> > running AnalyzeTool, but can't figure out what. Does anyone know how to
> get
> > the latency information?
> >
> > Thanks,
> > Eric
>

Mime
View raw message