flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 徐涛 <happydexu...@gmail.com>
Subject Task manager count goes the expand then converge process when running flink on YARN
Date Wed, 24 Oct 2018 15:26:58 GMT
Hi experts
	I am running flink job on YARN in job cluster mode, the job is divided into 2 tasks, the
following are some configs of the job:
	parallelism.default => 16
	taskmanager.numberOfTaskSlots => 8
	-yn => 2

	when the program starts, I found that the count of task managers is not set immediately,
but first expand then converge, I record the number during the process:
		Task Managers         Task Slots         Available Task Slots
	1.	14				 104                    88
        2.	15                             120                    104
        3.	16                             128                    112
       	4.	 6                               48                      32
       	5.	 3                               24                      8
       	6.	 2                               16                      0

	The final state is correct. There are 2 tasks, 32 subtask in total, due to slot sharing,
only 16 slots are enough, the number of task slots per TM are 8, so 2 TMs are needed.
	I have the following question:
	Because I specify yn=2, why does not directly allocate 2 TMs, but goes the expand then converge
process?  Why does it apply 16 task managers at most? If it is not a must, how to avoid it?

	Thanks a lot!

Best
Henry
	
Mime
View raw message