Hi Fabian and till,
Below is what I've observed today.
Hope it provides a strong evidence to figure out the problem.
I attach another log file, jmlog2.txt, after observing the different behavior of a per-job cluster with more memory given to YARN nodemanagers (compared to jmlog.txt).
- jmlog.txt : Each of 7 NodeManagers has 96GB. Only a single TM (50GB) can be scheduled on a NM and I ended up with having only 7 NodeManagers. There's no room for extra unnecessary TaskManagers.
- jmlog2.txt : Each of 7 NMs has 128GB. After scheduling a TM on each NM, RM can schedule additional 7 TMs as each NM has remaining 78 GB.
What I see from both log files is that,
- ExecutionGraph creates 100 tasks as I specified.
- Initially 7 necessary containers (for 7 TMs each with 16 slots) are requested to YARN, which is quite desired behavior.
- However, extra unnecessary 93 requests are made after the very first TaskManager is registered to SlotManager with the following messages:
+ jmlog.txt : Register TaskManager 640b098f3a132b452a74673631a0bf7f at the SlotManager.
+ jmlog2.txt : Registering TaskManager container_1525676778566_0001_01_000005 under 35b9ceed32bd87fa23ddca4282f5abac at the SlotManager.
(Please note that the info messages are different in jmlog.txt and jmlog2.txt; it is due to a recent hotfix "Add resourceId to TaskManager registration messages")
The 93 containers should not be asked as JobMaster is going to have enough slots on the 6 TaskManagers which will be soon registered to SlotManager.
This causes a deadlock situation if YARN does not have resources to allocate such 93 containers as in jmlog.txt.
Unlike in jmlog.txt, jmlog2.txt shows
- Extra TMs are scheduled on newly scheduled containers.
- Extra TMs are not given any tasks for while.
- Extra TMs are shut down with the below message.
"Closing TaskExecutor connection container_1525676778566_0001_01_000015 because: TaskExecutor exceeded the idle timeout."
- At the end, there are no pending container requests in jmlog2.txt at the end.
p.s. I just found that SlotManager is only for flip-6. Nevertheless, I write this email to user@ as I originally start this thread on user@. Sorry for the inconvenience.