Mailing-List: contact dev-help@harmony.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@harmony.apache.org
Received-SPF: pass (herse.apache.org: domain of ivan.volosyuk@gmail.com
 designates 66.249.92.169 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=Oq2WoXuGm/agkqWJekRHbv6qW86+E2deX8X7N579oAqLtHJnWil7OexVQOxW+m52/20O+T806tCnr55x0mf6xSJEo0sfJlfzcl1KuX1fMRW2/4f66CPBQKaX8eTledW+PBguCqfgwcaJVqHCdO5W3suOL/FEsNusgPZOpjpG08Y=
Message-ID: <12385bbd0702041724k4d3715a5j4f7471161f3df18d@mail.gmail.com>
Date: Mon, 5 Feb 2007 04:24:48 +0300
From: "Ivan Volosyuk" <ivan.volosyuk@gmail.com>
To: dev@harmony.apache.org
Subject: Re: [VM]How to trigue GC while free native memory is low.
In-Reply-To: <12385bbd0702041713w34add2eas4998ff1e7cd881fc@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <e66844de0702012308r619847e4wc66f692d7ac67a8b@mail.gmail.com>
	 <e66844de0702020201o1f5a0645ifbc64b7ede8d052c@mail.gmail.com>
	 <9623c9a50702020244p4bba7163m801286d46851212c@mail.gmail.com>
	 <5c8e69f0702021902q1119434aqa8dd7b8b4bd0f9ff@mail.gmail.com>
	 <12385bbd0702030500g54a20475ke2cdefaa063337ab@mail.gmail.com>
	 <eq35kr$m88$1@sea.gmane.org>
	 <e66844de0702032021l4e1c9313o6e7c712b207f017d@mail.gmail.com>
	 <eq4nqq$bfs$1@sea.gmane.org>
	 <C4A3015C-AB78-4779-A78B-FA1C5E9FDE87@pobox.com>
	 <12385bbd0702041713w34add2eas4998ff1e7cd881fc@mail.gmail.com>

On 2/5/07, Ivan Volosyuk <ivan.volosyuk@gmail.com> wrote:
> On 2/4/07, Geir Magnusson Jr. <geir@pobox.com> wrote:
> >
> > On Feb 4, 2007, at 8:41 AM, Gregory Shimansky wrote:
> >
> > > Leo Li wrote:
> > >> On 2/4/07, Gregory Shimansky <gshimansky@gmail.com> wrote:
> > >>>
> > >>> I see no difference in the approach I've suggested already. If we
> > >>> have
> > >>> to take care about all the native resource allocation inside of
> > >>> classlib
> > >>> APIs, then there is no difference which code calls GC, be it
> > >>> gc_native_malloc, or the API native code like this:
> > >>>
> > >>> void *p = malloc(size);
> > >>> if (NULL == p)
> > >>> {
> > >>>   gc_force_gc();
> > >>>   p = malloc(size);
> > >>>   if (NULL == p)
> > >>>   {
> > >>>       jni_env->ThrowNew("java/lang/OutOfMemoryError");
> > >>>       return;
> > >>>   }
> > >>> }
> > >>
> > >>
> > >> But I am worried whether it is too late to force gc when memory
> > >> allocation
> > >> fails. It will take some time to complete the release of
> > >> resources, for
> > >> example, in finalizer() and this period of time is not
> > >> predictable. Thus
> > >> the
> > >> malloc will still fails...:(
> > >> Futhermore, if we wait till memory allocation fails then to force
> > >> gc, it
> > >> might be possible that more than one thread will encounter such a
> > >> problem
> > >> and force gc. I am not sure whether it will lead to problem in vm.
> > >> So, I think it may be wiser to trigger a gc when free memory ratio is
> > >> low and before the failure in malloc actually occurs.
> > >
> > > Yes, I shouldn't have called the above code a "solution". I just
> > > wanted
> > > to point to Ivan that there is no difference with taking care about
> > > memory allocation in GC code or in classlib code.
> > >
> > > You've found another problem in such approach, just running GC is not
> > > enough in this case because finalization may not be done in time to
> > > free
> > > needed native resources.
> > >
> > > Speaking in DRLVM terms, the above code could also contain call to
> > > vm_run_pending_finalizers, but it is still not a solution for
> > > multithreaded case.
> >
> > I'm getting confused on where people think that this problem needs to
> > be managed.  Am I misunderstanding that there should be some
> > awareness of this in the classlib or VM natives?  Seems like it can
> > be localized.
> >
> > I think that if whoever calls malloc() finds that the system is out
> > of memory, then it's too late.  I always thought that VMs manage
> > their conceptual native "heap" (all native resources consumed from
> > OS, including as Robin mentioned, filehandles) in a proactive manner,
> > like they do their java heap.  it's clear now that they don't.
> >
> > So when a caller to malloc() gets a null, it should simply throw an
> > OOM, assuming that the runtime environment did all it could to
> > prevent that.
> >
> > Can this be done through portlib, letting the VMs implementation of
> > it manage to whatever degree of sophistication is warranted?
>
> Actually, the portlib is the lowest layer we rely on. If we can have
> here a callback to VM's GC we can place the code in the portlib. I
> would also distinguish memory allocations which associated with java
> objects and that which doesn't. No need to account resources which
> will not be freed after garbage collection.
>
> As was mentioned we have different types of critical system resources.
> It makes sense to make a list of such resources:
>    malloc'ed memory
>    mmap'ed memory
>    shared memory (?)
>    file descriptors (files, sockets)
>    XWin / GDI resources (?)
>    more?
>
> What would be the accounting strategy for the resources? We could
> allocate them via some functions in this facility or we could only
> account them here and call GC using some buildin criteria.

More thoughts.

It makes sense that allocation or accounting routine will block for
some time until finalizers path completes as Robin suggested. To
prevent deadlocks we should avoid blocking finalizer threads, though
it can cause unavoidable massive-allocation-in-finalizer attack. Do we
have any more possible deadlocks here?

-- 
Ivan
Intel Enterprise Solutions Software Division