flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yun Tang <myas...@live.com>
Subject Re: Savepoint memory overhead
Date Thu, 30 Apr 2020 06:55:46 GMT
Hi Lasse

Would you please give more details?

  1.  What is the memory configuration of your task manager? e.g the memory size of process,
managed memory. And how large the memory would increase to once you meet problem.
  2.  Did you use managed memory to control RocksDB? [1]
  3.  Why you give the assumption that memory problem has relationship with savepoint?
  4.  What is the topology of you streaming job: how many states and window you use, how many
slots per task manager?

[1] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_tuning.html#rocksdb-state-backend

Best
Yun Tang
________________________________
From: Lasse Nedergaard <lassenedergaardflink@gmail.com>
Sent: Thursday, April 30, 2020 12:39
To: Yun Tang <myasuka@live.com>
Cc: user <user@flink.apache.org>
Subject: Re: Savepoint memory overhead

We using Flink 1.10 running on Mesos.

Med venlig hilsen / Best regards
Lasse Nedergaard


Den 30. apr. 2020 kl. 04.53 skrev Yun Tang <myasuka@live.com>:


Hi Lasse

Which version of Flink did you use? Before Flink-1.10, there might exist memory problem when
RocksDB executes savepoint with write batch[1].


[1] https://issues.apache.org/jira/browse/FLINK-12785

Best
Yun Tang
________________________________
From: Lasse Nedergaard <lassenedergaardflink@gmail.com>
Sent: Wednesday, April 29, 2020 21:17
To: user <user@flink.apache.org>
Subject: Savepoint memory overhead

Hi.

I would like to know if there are any guidelines/recommendations for the memory overhead we
need to calculate for when doing savepoint to s3. We use RockDb state backend.

We run our job on relative small task managers and we can see we get memory problems if the
state size for each task manager get "big" (we haven't found the rule of thumbs yet) and we
can remove the problem if we reduce the state size, or increase parallelism and jobs with
none or small state don't have any problems.
So I see a relation between between allocated memory to a task manager and the state it can
handle.

So do anyone have any recommendations/ base practices for this and can someone explain why
savepoint requires memory.

Thanks

In advance

Lasse Nedergaard

Mime
View raw message