flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tobias Wiens <tobwi...@gmail.com>
Subject Flink Benchmark: comparing flink's performance in different environments
Date Mon, 04 May 2015 12:05:22 GMT
Hi,

I am writing my master thesis about how to orchestrate scalable tasks and
services with docker. The idea is to swap a task's environment very fast, I
plan to use Flink as the environment. For that I package Apache Flink into
container and start one master and X worker nodes plus one task on them.
Like a short term Flink PaaS.

I want to measure the performance of one Flink cluster and compare it to a
Flink cluster inside containers.
That benchmark will be part of a series of benchmarks in which I test disk,
network, memory and CPU. So the Flink benchmark can be an applied use-case
which mixes all sorts of demands.

I saw the TPC-H data generator example (
https://github.com/rmetzger/scratch/blob/distributed-tpch-generator/src/main/java/flink/generators/programs/TPCHGeneratorExample.java)
and I read about the sorting benchmarks on (http://sortbenchmark.org/).
TPC-H: I am not sure how meaningful it is to join those two tables and if
it is commonly used in publications. I thought the minute sort might be
interesting because it sounds small.

*Which benchmarks would you recommend?* The benchmarks should be easy to
implement.

I don't have any infrastructure, I want to rent bare metal servers at
softlayer (
https://www.softlayer.com/Store/orderHourlyBareMetalInstance/37276/64)
maybe 5 servers (4 cores, 8GB ram).* Does that make sense? Will I even see
differences in such a "small" cluster? *

Any help or pointing for papers which do benchmark of data processing
platforms very welcome. Thank you!

Best regards,
Tobias

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message