flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyula.f...@gmail.com>
Subject Re: Task manager processes crashing one after the other
Date Thu, 25 Aug 2016 17:08:00 GMT
Yes seems like that, I remember the fix in Flink. I apparently made a
mistake somewhere in our code :)

Thanks,
Gyula

On Thu, Aug 25, 2016, 18:59 Stephan Ewen <sewen@apache.org> wrote:

> We saw some crashes in earlier versions when native handles in RocksDB
> (even for config option objects) were manually and too eagerly released.
>
> Maybe you have a similar issue here?
>
> On Thu, Aug 25, 2016 at 6:27 PM, Gyula Fóra <gyula.fora@gmail.com> wrote:
>
> > Hi,
> > This seems to be a sneaky concurrency issue in our custom statebackend
> > implementation.
> >
> > I made some changes, will keep you posted.
> >
> > Cheers,
> > Gyula
> >
> > On Thu, Aug 25, 2016, 10:54 Gyula Fóra <gyula.fora@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > Sure I am sending the TM logs in priv.
> > >
> > > Currently what I did was to bump the Rocks version to 4.9.0 let's see
> if
> > > that helps.
> > >
> > > Cheers,
> > > Gyula
> > >
> > > Till Rohrmann <trohrmann@apache.org> ezt írta (időpont: 2016. aug.
> 25.,
> > > Cs, 10:35):
> > >
> > >> Hi Gyula,
> > >>
> > >> I haven't seen this problem before. Do you have the logs of the failed
> > TMs
> > >> so that we have some more context what was going on?
> > >>
> > >> Cheers,
> > >> Till
> > >>
> > >> On Thu, Aug 25, 2016 at 9:40 AM, Gyula Fóra <gyfora@apache.org>
> wrote:
> > >>
> > >> > Hi guys,
> > >> >
> > >> > For quite some time now we fairly frequently experience a task
> manager
> > >> > crashes around the time new streaming jobs are deployed. We use
> > RocksDB
> > >> > backend so this might be related.
> > >> >
> > >> > We tried changing the GC from G1 to CMS that didnt help.
> > >> >
> > >> > Yesterday for instance 6 task managers crashed one ofter the other
> > with
> > >> > similar errors:
> > >> >
> > >> > *** Error in `java': double free or corruption (!prev):
> > >> 0x00007fac0414d760
> > >> > ***
> > >> > *** Error in `java': free(): invalid pointer: 0x00007f8dcc0026c0 ***
> > >> > *** Error in `java': double free or corruption (!prev):
> > >> 0x00007f15247f9a90
> > >> > ***
> > >> > ...
> > >> >
> > >> > Does anyone have any clue what might cause this or how to debug?
> > >> > This is very a critical issue :(
> > >> >
> > >> > Cheers,
> > >> > Gyula
> > >> >
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message