harmony-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Volosyuk" <ivan.volos...@gmail.com>
Subject Re: [VM]How to trigue GC while free native memory is low.
Date Sat, 03 Feb 2007 00:33:06 GMT
Proactive freeing such native resources may lead to instability. We
don't have enough control of all references to that native resources.

On 2/3/07, Rana Dasgupta <rdasgupt@gmail.com> wrote:
> I don't know much about DirectByteBuffer implementation, but it would be
> nice to keep the memory management of managed and unmanaged heaps seperate
> from each other since they are independent resources. Housekeeping the
> active java proxy reference and freeing up native memory proactively/lazily
> when there are none would be great, but both would need some kind of
> refcounting on these instances independent of gc. Some jit support would be
> needed to update the refcount when these proxies( instances of this class
> ) are getting copied around, like a write barrier call.
>
> Depending on hooks to monitor the unmanaged heap has gaps. There could be
> other running processes on the machine with unpredictable resource requests.
> It may be easier to empirically call System.GC() periodically based on
> an estimation of unmanaged heap allocation and tune it to be realistic.
>
> Thanks,
> Rana
>
>
>
> On 2/2/07, Tim Ellison <t.p.ellison@gmail.com> wrote:
> >
> > You don't want to invoke a GC though, as you said there is no allocation
> > failure in the Java heap.
> >
> > If you are eager you just want to know when there are no more references
> > to the 'byteBuffer' instance, so you can free the native resources it
> > references.  And if you are lazy you would wait for a resource
> > allocation failure when creating a new direct byte buffer then be
> > prepared to ask if there are any collectable 'byteBuffer' instances.
> >
> > IIRC the NIO buffer code tries to behave eagerly, and uses a
> > PhantomReference to the object holding onto the resource.  I'm assuming
> > that you see the out of malloc failure because the phantom is only
> > enqueued by invoking System.gc(), right?
> >
> > Regards,
> > Tim
> >
> > Leo Li wrote:
> > > Hi, Xiao-Feng:
> > >     Excuse me if I am confusing you.
> > >     The direct byte buffer holds a block of native memory space used in
> > nio
> > > operation. The native byte buffer will be freed through the byte buffer
> > is
> > > gc collected and added to ReferenceQueue monitored by MemorySpy series
> > > classes.
> > >     But the gc will not be triggered if the java heap is still empty.
> > Here
> > > is an example: I allocate a large block of native memory for a byte
> > buffer
> > > while the byte buffer itself is quite small. Thus quickly native heap is
> > > depleted while VM still does not think gc is needed since java heap has
> > a
> > > lot of free memory.
> > >
> > >
> > > import java.nio.*;
> > > public class Test {
> > >    public static void main(String[] args) throws Exception {
> > >        for(int i = 0;i<1000;i++)
> > >        {
> > >            ByteBuffer byteBuffer = ByteBuffer.allocateDirect(10240000);
> > >            //System.gc();
> > >        }
> > >   }
> > > }
> > > RI runs well but Harmony will soon throw OutOfMemory exception.
> > > But if remove the comment before System.gc(), Harmony will become ok.
> > > So my point is that we need a mechanism to notify VM to start gc if we
> > have
> > > no more native memory because sometimes problem will be solved if gc is
> > > fired. Of cause we cannot avoid a  user always malloc space but never
> > free
> > > them.:)
> > >
> > > Good luck!
> > >
> > > On 2/2/07, Xiao-Feng Li <xiaofeng.li@gmail.com> wrote:
> > >>
> > >> Leo, I can't fully understand your problem and proposal. I am working
> > >> in DRLVM GC component, I want to understand how GC can tackle the
> > >> issue you meet.
> > >>
> > >> You said the native memory is allocated (probably with malloc) for
> > >> byte buffer. I wonder how the process goes. I assume it's in native
> > >> code. Then you use malloc to allocate a block of memory, and put some
> > >> Java object (byte buffer) into it? Is this what you do? I wonder how
> > >> do you manipulate Java object placement in native code; and if the
> > >> native memory block is out of Java heap, do you expect GC to manage
> > >> it?
> > >>
> > >> You said when GC reclaims those byte buffer object, the native memory
> > >> block can be freed in native code. How do you know if an object is
> > >> reclaimed by GC or not?
> > >>
> > >> Thanks,
> > >> xiaofeng
> > >>
> > >> On 2/2/07, Leo Li < liyilei1979@gmail.com> wrote:
> > >> > Hi, all:
> > >> >     After applying patch3073, I added the support for direct byte
> > >> buffer
> > >> in
> > >> > nio module as spec requires. But now here exists a problem: The
> > native
> > >> > memory allocated to the byte buffer can be released when gc releases
> > >> the
> > >> > byte buffer object, however, if the native heap is full while java
> > heap
> > >> > still has space, gc will not be triggered.
> > >> >     It seems that RI will start gc before native memory heap is
> > >> depleted
> > >> and
> > >> > thus prevents out-of-memory error.
> > >> >
> > >> >    Then our work focuses on:
> > >> >    1. When gc is required.
> > >> >    2. Trigger a gc.
> > >> >    The first one requires that we get support from operating system,
> > >> since
> > >> > the memory allocated in the native code, for example by malloc, is
> > out
> > >> of
> > >> > the control of java VM.( I have ever thought of counting the used
> > >> memory
> > >> in
> > >> > hymemory_allocate, but the plan fails since hymemory_free will not
> > >> report
> > >> > how much space is released.)
> > >> >    The second one needs the help from VM. System.gc() is not so
> > >> reliable
> > >> as
> > >> > spec says, so it is necessary to have a internal channel to notify
> > >> VM to
> > >> > start gc.
> > >> >
> > >> >     One solution, I think, is to let a monitor thread in VM to check
> > >> whether
> > >> > OS physical memory is low. For example, the
> > >> QueryMemoryResourceNotification
> > >> > of win32 API is a candidate. Although the interrupt model is more
> > >> effective:
> > >> > win32 SDK provides a CreateMemoryResourceNotification to get a
> > handler
> > >> on
> > >> > which the monitor thread can wait, maybe on other platforms, OS does
> > >> not
> > >> > supply such a convenience. So the monitor thread in the VM might
> > >> have to
> > >>
> > >> > check the OS resource once for a while and if necessary the monitor
> > >> thread
> > >> > will call a GC.
> > >> >
> > >> >    My suggestion is first to add some function to monitor memory in
> > >> portlib,
> > >> > since it is highly related to platforms and in portlib there has been
> > >> > some useful tools.(Thanks Mark to point out that.) On the other
> > >> hand, we
> > >> can
> > >> > negotiate a channel to trigger gc in VM. Actually I am not an expert
> > on
> > >> VM
> > >> > since I am not sure whether there has been some monitor thread and
> > the
> > >> load
> > >> > on performance if such a monitor thread is added to the VM...
> > >> >
> > >> >
> > >> > --
> > >> > Leo Li
> > >> > China Software Development Lab, IBM
> > >> >
> > >> >
> > >>
> > >
> > >
> > >
> >
>
>


-- 
Ivan
Intel Enterprise Solutions Software Division

Mime
View raw message