flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: flink performance
Date Mon, 08 Sep 2014 14:13:17 GMT
There is probably a little typo in Aljoscha's answer. The
taskmanager.numberOfTaskSlots should be 8 (there are 8 cores per machine)
The parallelization.degree.default is correct.

On Mon, Sep 8, 2014 at 4:09 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Hi Norman,
> I saw you were running our Scala Examples. Unfortunately those do not
> run as well as our Java examples right now. The Scala API was a bit of
> a prototype that has some issues with efficiency. For now, you could
> maybe try running our Java examples.
>
> For your cluster, good configuration values would be numberOfTaskSlots
> = 4 (number of CPU cores) and parallelization.degree.default = 32
> (number of nodes X number of CPU cores).
>
> The Scala API is being rewritten for our next release, so if you
> really want to check out Scala examples I could point you to my
> personal branch on github where development of the new Scala API is
> taking place.
>
> Cheers,
> Aljoscha
>
> On Mon, Sep 8, 2014 at 2:48 PM, Norman Spangenberg
> <wir12kqe@studserv.uni-leipzig.de> wrote:
> > Hello,
> > I'm a bit confused about the performance of Flink.
> > My cluster consists of 4 nodes, each with 8 cores and 16gb memory (1.5 gb
> > reserved for OS). using flink-0.6 in standalone-cluster mode.
> > i played a little bit with the config-settings but without much impact on
> > execution time.
> > flink-conf.yaml:
> > jobmanager.rpc.port: 6123
> > jobmanager.heap.mb: 1024
> > taskmanager.heap.mb: 14336
> > taskmanager.memory.size: -1
> > taskmanager.numberOfTaskSlots: 4
> > parallelization.degree.default: 16
> > taskmanager.network.numberOfBuffers: 4096
> > fs.hdfs.hadoopconf: /opt/yarn/hadoop-2.4.0/etc/hadoop/
> >
> > I tried two applications: wordcount and k-Means scala example code
> > wordcount needs 5 minutes for 25gb, and 13 minutes for 50gb.
> > kmeans (10 iterations) needs for 56mb input 86 seconds, but with 1.1gb
> input
> > it needs 33minutes with 2.2gb nearly 90 minutes!
> >
> > the monitoring tool ganglia says, that cpu has low cpu utilization and a
> lot
> > of waiting time. in wordcount cpu utilizes with nearly 100 percent.
> > Is this a ordinary dimension of execution time in spark? or are
> > optimizations in my config necessary? or maybe a bottleneck in the
> cluster?
> >
> > i hope somebody could help me :)
> > greets Norman
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message