flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinay Patil <vinay18.pa...@gmail.com>
Subject Re: Checkpointing with RocksDB as statebackend
Date Wed, 28 Jun 2017 15:26:23 GMT
Hi Aljoscha,

I am using event Time based tumbling window wherein the allowedLateness is
kept to Long.MAX_VALUE and I have custom trigger which is similar to 1.0.3
where Flink was not discarding late elements (we have discussed this
scenario before).

The watermark is working correctly because I have validated the records
earlier.

I was doubtful that the RocksDB statebackend is not set , but in the logs I
can clearly see that RocksDB is initialized successfully, so that should
not be an issue.

Even I have not changed any major  code from the last performance test I
had done.

The snapshot I had attached is of Off-heap memory, I have only assigned
12GB heap memory per TM


Regards,
Vinay Patil

On Wed, Jun 28, 2017 at 8:43 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Hi,
>
> Just a quick question, because I’m not sure whether this came up in the
> discussion so far: what kind of windows are you using? Processing
> time/event time? Sliding Windows/Tumbling Windows? Allowed lateness? How is
> the watermark behaving?
>
> Also, the latest memory usage graph you sent, is that heap memory or
> off-heap memory or both?
>
> Best,
> Aljoscha
>
> > On 27. Jun 2017, at 11:45, vinay patil <vinay18.patil@gmail.com> wrote:
> >
> > Hi Stephan,
> >
> > I am observing similar issue with Flink 1.2.1
> >
> > The memory is continuously increasing and data is not getting flushed to
> > disk.
> >
> > I have attached the snapshot for reference.
> >
> > Also the data processed till now is only 17GB and above 120GB memory is
> > getting used.
> >
> > Is there any change wrt RocksDB configurations
> >
> > <http://apache-flink-user-mailing-list-archive.2336050.n4.
> nabble.com/file/n14013/TM_Memory_Usage.png>
> >
> > Regards,
> > Vinay Patil
> >
> >
> >
> > --
> > View this message in context: http://apache-flink-user-maili
> ng-list-archive.2336050.n4.nabble.com/Re-Checkpointing-
> with-RocksDB-as-statebackend-tp11752p14013.html
> > Sent from the Apache Flink User Mailing List archive. mailing list
> archive at Nabble.com.
>
>

Mime
View raw message