flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yuta Morisawa <yu-moris...@kddi-research.jp>
Subject Re: CPU Cores of JobManager
Date Wed, 06 Dec 2017 00:20:58 GMT
Hi Timo

I execute streaming job without checkpointing and I don't configure any 
state backend, so it may be "MemoryStateBackend".

Actually, my streaming app just reads data from kafka and writes it to 
an external DB. Its not so complicated.

Regards,
Yuta

On 2017/12/05 19:55, Timo Walther wrote:
> I had some profiling tool like jvisualvm in mind. Are you executing 
> streaming or batch jobs? If streaming, is checkpointing enabled and 
> which type of statebackend?
> 
> @Chesnay do you have experience with slow behavior of the Web UI?
> 
> Regards,
> Timo
> 
> 
> Am 12/5/17 um 10:37 AM schrieb Yuta Morisawa:
>> Hi Timo
>>
>> Thank you for your early reply.
>>
>> These are commands which I run my apps.
>> ./bin/yarn-session.sh -n 20 -jm 6000 -tm 24000 -s 10
>> ./bin/flink run -p 100 <jar1> <jar1 parameters>
>> ./bin/flink run -p 100 <jar2> <jar2 parameters>
>>
>> So, JobManager Heap Memory = 6000 MB and it manages 2 jobs.
>>
>> > Maybe you can use a profiler and find out which component consumes so
>> > much CPU resources?
>> You mean Java Flight Recorder or JITWatch?
>> Or, Flink has original profiler?
>> https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html

>>
>>
>> Regards,
>> Yuta
>>
>> On 2017/12/05 18:02, Timo Walther wrote:
>>> Hi Yuta,
>>>
>>> as far as I know you cannot assign more cores to a JobManager.
>>>
>>> Can you tell us a bit more about your environment? How many jobs does 
>>> the JobManager has to manage? How much heap memory is assigned to the 
>>> JobManager?
>>>
>>> Maybe you can use a profiler and find out which component consumes so 
>>> much CPU resources?
>>>
>>> Regards,
>>> Timo
>>>
>>>
>>> Am 12/5/17 um 5:13 AM schrieb Yuta Morisawa:
>>>> Hi
>>>>
>>>> Now I am looking for the way to increase the number of allocated CPU 
>>>> cores because my JobManagaer WEBUI is very heavy and sometimes freeze.
>>>>
>>>> I think this is caused by the resource shortage of JobManager.
>>>> How can I increase the number of CPU for JobManager in YARN mode?
>>>>
>>>>
>>>>
>>>> Thanks
>>>> Yuta
>>>
>>>
>>>
> 
> 


Mime
View raw message