lucene-pylucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Janssen <jans...@parc.com>
Subject Re: finding exceptions the crash pylucene
Date Fri, 15 Apr 2011 10:54:20 GMT
Marcus <qwelby@gmail.com> wrote:

> Bill:
> I'm not sure I follow.
> why would raising the JVM memory to 4GB ever cause a crash in python?
> Our server has 48GB.

I don't know the specifics of your deployment, but you may not be able
to use that much.  32-bit Python, for instance, won't be able to use it.
Even with 64-bit Python, the OS may place limits on how much memory can
be used by single process.  If the Java VM uses too much, the Python VM
will be choked.

Bill

> 
> thanks
> Marcus
> 
> On Fri, Apr 15, 2011 at 7:33 AM, Bill Janssen <janssen@parc.com> wrote:
> 
> > Marcus <qwelby@gmail.com> wrote:
> >
> > > --bcaec53043296dfbfd04a0ece1ac
> > > Content-Type: text/plain; charset=ISO-8859-1
> > >
> > > we're currently using 4GB max heap.
> > > We recently moved from 2GB to 4GB when we discovered it prevented a crash
> > > with a certain set of docs.
> > > Marcus
> >
> > I've tried the same workaround with the heap in the past, and I found it
> > caused NoMemory crashes in the Python side of the house, because the
> > Python VM couldn't get enough memory to operate.  So, be careful.
> >
> > > On Thu, Apr 14, 2011 at 5:01 PM, Andi Vajda <vajda@apache.org> wrote:
> > >
> > > >
> > > > On Thu, 14 Apr 2011, Marcus wrote:
> > > >
> > > >  thanks.
> > > >>
> > > >> I have documents that will consistently cause this upon writing them
> > to
> > > >> the
> > > >> index. let me see if I can reduce them down to the crux of the crash.
> > > >> granted, these are docs are very large, unruly "bad" data, that should
> > > >> have
> > > >> never gotten this stage in our pipeline, but I was hoping for a java
> > or
> > > >> lucene exception.
> > > >>
> > > >> I also get "Java GC overhead" exceptions passed into my code from
time
> > to
> > > >> time, but those manageable, and not crashes.
> > > >>
> > > >> Are there known memory constraint scenarios that force a c++
> > exception,
> > > >> whereas in a normal Java environment,  you would get a memory error?
> > > >>
> > > >
> > > > Not sure.
> > > >
> > > >
> > > >  and just confirming, do "java.lang.OutOfMemoryError" errors pass into
> > > >> python, or force a crash?
> > > >>
> > > >
> > > > Not sure, I've never seen these as I make sure I've got enough memory.
> > > > initVM() is the place where you can configure the memory for your JVM.
> > > >
> > > > Andi..
> > > >
> > > >
> > > >
> > > >> thanks again
> > > >> Marcus
> > > >>
> > > >> On Thu, Apr 14, 2011 at 2:07 PM, Andi Vajda <vajda@apache.org>
wrote:
> > > >>
> > > >>
> > > >>> On Thu, 14 Apr 2011, Marcus wrote:
> > > >>>
> > > >>>  in certain cases when a java/pylucene exception occurs,  it gets
> > passed
> > > >>> up
> > > >>>
> > > >>>> in my code, and I'm able to analyze the situation.
> > > >>>> sometimes though,  the python process just crashes, and if
I happen
> > to
> > > >>>> be
> > > >>>> in
> > > >>>> top (linux top that is), I see a JCC exception flash up in
the top
> > > >>>> console.
> > > >>>> where can I go to look for this exception, or is it just lost?
> > > >>>> I looked in the locations where a java crash would be located,
but
> > > >>>> didn't
> > > >>>> find anything.
> > > >>>>
> > > >>>>
> > > >>> If you're hitting a crash because of an unhandled C++ exception,
> > running
> > > >>> a
> > > >>> debug build with symbols under gdb will help greatly in tracking
it
> > down.
> > > >>>
> > > >>> An unhandled C++ exception would be a PyLucene/JCC bug. If you
have a
> > > >>> simple way to reproduce this failure, send it to this list.
> > > >>>
> > > >>> Andi..
> > > >>>
> > > >>>
> > > >>
> > >
> > > --bcaec53043296dfbfd04a0ece1ac--
> >

Mime
View raw message