flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Fukuda <e.s.fuk...@gmail.com>
Subject Re: How to get latency info from benchmark
Date Wed, 24 Aug 2016 20:32:50 GMT
Hi Max, Robert,

Thanks for the advice. I'm trying to build the "performance" project, but
failing with the following error. Is there a solution for this?

[ERROR] Failed to execute goal on project streaming-state-demo: Could not
resolve dependencies for project com.dataartisans.flink:
streaming-state-demo:jar:1.0-SNAPSHOT: Failure to find
org.apache.flink:flink-connector-kafka-083:jar:0.10-SNAPSHOT in
https://repository.apache.org/content/repositories/snapshots/ was cached in
the local repository, resolution will not be reattempted until the update
interval of apache.snapshots has elapsed or updates are forced -> [Help 1]




On Wed, Aug 24, 2016 at 8:12 AM, Robert Metzger <rmetzger@apache.org> wrote:

> Hi Eric,
>
> Max is right, the tool has been used for a different benchmark [1]. The
> throughput logger that should produce the right output is this one [2].
> Very recently, I've opened a pull request for adding metric-measuring
> support into the engine [3]. Maybe that's helpful for your experiments.
>
>
> [1] http://data-artisans.com/high-throughput-low-latency-and
> -exactly-once-stream-processing-with-apache-flink/
> [2] https://github.com/dataArtisans/performance/blob/master/
> flink-jobs/src/main/java/com/github/projectflink/streaming/
> Throughput.java#L203
> [3] https://github.com/apache/flink/pull/2386
>
>
>
> On Wed, Aug 24, 2016 at 2:04 PM, Maximilian Michels <mxm@apache.org>
> wrote:
>
>> I believe the AnaylzeTool is for processing logs of a different benchmark.
>>
>> CC Jamie and Robert who worked on the benchmark.
>>
>> On Wed, Aug 24, 2016 at 3:25 AM, Eric Fukuda <e.s.fukuda@gmail.com>
>> wrote:
>> > Hi,
>> >
>> > I'm trying to benchmark Flink without Kafka as mentioned in this post
>> > (http://data-artisans.com/extending-the-yahoo-streaming-benchmark/).
>> After
>> > running flink.benchmark.state.AdvertisingTopologyFlinkState with
>> > user.local.event.generator in localConf.yaml set to 1, I ran
>> > flink.benchmark.utils.AnalyzeTool giving
>> > flink-1.0.1/log/flink-[username]-jobmanager-0-[servername].log as a
>> > command-line argument. I got the following output and it does not have
>> the
>> > information about the latency.
>> >
>> >
>> > ================= Latency (0 reports ) =====================
>> > ================= Throughput (1 reports ) =====================
>> > ====== null (entries: 10150)=======
>> > Mean throughput 639078.5018497099
>> > Exception in thread "main" java.lang.IndexOutOfBoundsException:
>> toIndex = 2
>> >         at java.util.ArrayList.subListRangeCheck(ArrayList.java:962)
>> >         at java.util.ArrayList.subList(ArrayList.java:954)
>> >         at flink.benchmark.utils.AnalyzeTool.main(AnalyzeTool.java:133)
>> >
>> >
>> > Reading the code in AnalyzeTool.java, I found that it's looking for
>> lines
>> > that include "Latency" in the log file, but apparently it's not finding
>> any.
>> > I tried grepping the log file, and couldn't find any either. I have one
>> > server that runs both JobManager and Task Manager and another server
>> that
>> > runs Redis, and they are connected through a network with each other.
>> >
>> > I think I have to do something to read the data stored in Redis before
>> > running AnalyzeTool, but can't figure out what. Does anyone know how to
>> get
>> > the latency information?
>> >
>> > Thanks,
>> > Eric
>>
>
>

Mime
View raw message