flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shashwat Rastogi <shashwat.rast...@reflektion.com>
Subject Re: Memory Leak - Flink / RocksDB ?
Date Thu, 27 Jul 2017 08:49:45 GMT
Hi Kien,

Sorry it took me sometime to fetch the logs. I am attaching logs of the machine which died
due to lack of free memory. 

I have set only
`taskmanager.heap.mb: 35840`
taskmanager.numberOfTaskSlots: 8
And the rest are just default properties in the flink-conf.yaml

Thank you in advance.


> On 26-Jul-2017, at 12:10 PM, Kien Truong <duckientruong@gmail.com> wrote:
> Hi,
> What're your task manager memory configuration ? Can you post the TaskManager's log ?
> Regards,
> Kien
> On 7/25/2017 8:41 PM, Shashwat Rastogi wrote:
>> Hi,
>> We have several Flink jobs, all of which reads data from Kafka do some aggregations
(over sliding windows of (1d, 1h)) and writes data to Cassandra. Something like :
>> ```
>> DataStream<String> lines = env.addSource(new FlinkKafkaConsumer010( … ));
>> DataStream<Event> events = lines.map(line -> parse(line));
>> DataStream<Statistics> stats = stream
>> 	.keyBy(“id”)
>> 	.timeWindow(1d, 1h)
>> 	.sum(new MyAggregateFunction());
>> writeToCassandra(stats);
>> ```
>> We recently made a switch to RocksDbStateBackend, for it’s suitability for large
states/long windows. However, after making the switch a memory issues has come up, the memory
utilisation on TaskManager gradually increases from 50 GB to ~63GB until the container is
killed. We are unable to figure out what is causing this behaviour, is there some memory leak
on the RocksDB ?
>> How much memory should we allocate to the Flink TaskManager? Since, RocksDB is a
native application and it does not use the JVM how much of the memory should we allocate/leave
for RocksDB (out of 64GB of total memory).
>> Is there a way to set the maximum amount of memory that will be used by RocksDB so
that it doesn’t overwhelms the system? Are there some recommended optimal settings for RocksDB
for larger states (for 1 day window average state size is 3GB).
>> Any help would be greatly appreciated. I am using Flink v1.2.1.
>> Thanks in advance.
>> Best,
>> Shashwat

View raw message