lucene-pylucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marcus <qwe...@gmail.com>
Subject Re: finding exceptions the crash pylucene
Date Fri, 15 Apr 2011 18:01:20 GMT
64bit linux(ubuntu), so I don't think there's any practical memory limit on
a python process in that case
I've had python processes up to 8GB personally
Marcus

On Fri, Apr 15, 2011 at 10:54 AM, Bill Janssen <janssen@parc.com> wrote:

> Marcus <qwelby@gmail.com> wrote:
>
> > Bill:
> > I'm not sure I follow.
> > why would raising the JVM memory to 4GB ever cause a crash in python?
> > Our server has 48GB.
>
> I don't know the specifics of your deployment, but you may not be able
> to use that much.  32-bit Python, for instance, won't be able to use it.
> Even with 64-bit Python, the OS may place limits on how much memory can
> be used by single process.  If the Java VM uses too much, the Python VM
> will be choked.
>
> Bill
>
> >
> > thanks
> > Marcus
> >
> > On Fri, Apr 15, 2011 at 7:33 AM, Bill Janssen <janssen@parc.com> wrote:
> >
> > > Marcus <qwelby@gmail.com> wrote:
> > >
> > > > --bcaec53043296dfbfd04a0ece1ac
> > > > Content-Type: text/plain; charset=ISO-8859-1
> > > >
> > > > we're currently using 4GB max heap.
> > > > We recently moved from 2GB to 4GB when we discovered it prevented a
> crash
> > > > with a certain set of docs.
> > > > Marcus
> > >
> > > I've tried the same workaround with the heap in the past, and I found
> it
> > > caused NoMemory crashes in the Python side of the house, because the
> > > Python VM couldn't get enough memory to operate.  So, be careful.
> > >
> > > > On Thu, Apr 14, 2011 at 5:01 PM, Andi Vajda <vajda@apache.org>
> wrote:
> > > >
> > > > >
> > > > > On Thu, 14 Apr 2011, Marcus wrote:
> > > > >
> > > > >  thanks.
> > > > >>
> > > > >> I have documents that will consistently cause this upon writing
> them
> > > to
> > > > >> the
> > > > >> index. let me see if I can reduce them down to the crux of the
> crash.
> > > > >> granted, these are docs are very large, unruly "bad" data, that
> should
> > > > >> have
> > > > >> never gotten this stage in our pipeline, but I was hoping for
a
> java
> > > or
> > > > >> lucene exception.
> > > > >>
> > > > >> I also get "Java GC overhead" exceptions passed into my code
from
> time
> > > to
> > > > >> time, but those manageable, and not crashes.
> > > > >>
> > > > >> Are there known memory constraint scenarios that force a c++
> > > exception,
> > > > >> whereas in a normal Java environment,  you would get a memory
> error?
> > > > >>
> > > > >
> > > > > Not sure.
> > > > >
> > > > >
> > > > >  and just confirming, do "java.lang.OutOfMemoryError" errors pass
> into
> > > > >> python, or force a crash?
> > > > >>
> > > > >
> > > > > Not sure, I've never seen these as I make sure I've got enough
> memory.
> > > > > initVM() is the place where you can configure the memory for your
> JVM.
> > > > >
> > > > > Andi..
> > > > >
> > > > >
> > > > >
> > > > >> thanks again
> > > > >> Marcus
> > > > >>
> > > > >> On Thu, Apr 14, 2011 at 2:07 PM, Andi Vajda <vajda@apache.org>
> wrote:
> > > > >>
> > > > >>
> > > > >>> On Thu, 14 Apr 2011, Marcus wrote:
> > > > >>>
> > > > >>>  in certain cases when a java/pylucene exception occurs,
 it gets
> > > passed
> > > > >>> up
> > > > >>>
> > > > >>>> in my code, and I'm able to analyze the situation.
> > > > >>>> sometimes though,  the python process just crashes, and
if I
> happen
> > > to
> > > > >>>> be
> > > > >>>> in
> > > > >>>> top (linux top that is), I see a JCC exception flash
up in the
> top
> > > > >>>> console.
> > > > >>>> where can I go to look for this exception, or is it just
lost?
> > > > >>>> I looked in the locations where a java crash would be
located,
> but
> > > > >>>> didn't
> > > > >>>> find anything.
> > > > >>>>
> > > > >>>>
> > > > >>> If you're hitting a crash because of an unhandled C++ exception,
> > > running
> > > > >>> a
> > > > >>> debug build with symbols under gdb will help greatly in tracking
> it
> > > down.
> > > > >>>
> > > > >>> An unhandled C++ exception would be a PyLucene/JCC bug. If
you
> have a
> > > > >>> simple way to reproduce this failure, send it to this list.
> > > > >>>
> > > > >>> Andi..
> > > > >>>
> > > > >>>
> > > > >>
> > > >
> > > > --bcaec53043296dfbfd04a0ece1ac--
> > >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message