flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michele Bertoni <michele1.bert...@mail.polimi.it>
Subject Yarn configuration
Date Fri, 24 Jul 2015 13:58:13 GMT
Hi everybody, i need a help on how to configure a yarn cluster
I tried a lot of conf but none of them was correct

We have a cluster on amazon emr let's say 1manager+5worker all of them are m3.2xlarge then
8 core each and 30 GB of RAM each

What is a good configuration for such cluster?

I would like to run 5 nodes with 8 slots each, is it correct?

Now the problems: by now i run all tests mistakenly using 40 task managers each with 2048MB
and 1 slot (at least it was working)

Today i found the error and i tried run 5 task manager and setting a default slot in conf-yaml
of 8, giving a task manager memory of 23040 (-tm 23040) that is the limit allowed by yarn
but i am getting errors: one TM is not running because there is no available memory. it seems
like the jm is not using memory from the master but from the nodes (in fact yarn says TM number
5 is missing 2048 that is the memory for the jm)

Then i reduced memories, everything started but i get a runtime error of missing buffer

Can someone help me syep-by-step in a good configuration for such cluster? I think the documentation
is really missing details

Thanks a lot

View raw message